Logseq-Git-Sync-101

Description

This repo aims to help Logseq users to sync their data via Git and GitHub.

Credits

Credits to @danieltomasz¹, @Sawhney², @Caps³, @Abulafia⁴ @qwxlea⁵ for the workflow build up. And credits to @danzu⁶ for the incredible diagram!

FYI, those IDs are IDs in Logseq Discord Server.

Why use Git + GitHub to sync Logseq graph?

Self Managed Sync Diagram, by @danzu⁶

From the above diagram, it's pretty obvious that Git is the most robust way to sync your graph. iCloud is slow and problematic, and Syncthing is not available on iOS/iPadOS.

However, Git is quite scary for non-programmers, so this doc is here to help!

Prerequisite

GitHub account.
Git, for computer users.
Working Copy, for iOS/iPadOS users.
Termux, for Android users.

What is Git?

Git is a free and open source distributed version control system created by Linus Torvalds in 2005.

What is GitHub?

GitHub is a for-profit company that offers a cloud-based Git repository hosting service. Essentially, it makes it a lot easier for individuals and teams to use Git for version control and collaboration.

How does Git works?

⚠️ Limitations of GitHub

Recommend repositories remain small, ideally less than 1 GB, and less than 5 GB is strongly recommended.
GitHub blocks pushes that exceed 100 MB, and you will receive a warning form Git when you attempt to add or update a file larger than 50 MB.
Use Git Large File Storage (Git LFS), if you really need to track those big files with Git.

⚠️ Things you should avoid

Open multiple instances of Logseq at a time, e.g. open Logseq on your computer and on your phone at the same time. This will cause conflict in your repo.
Put large files in assets folder, like images, videos, etc.

💥 Rejected push & Conflicts handling & Trailing whitespace

What is Rejected push?

Rejected push happens when the remote (eg, GitHub) contains modifications that you have yet to download. This can happen if you forgot to do a pull in your local version before commitng new changes to it.

What is Git Conflict?

Git conflict happens when you have two commits, one local and the other remote, which modify the same file in the same lines.

What is Trailing whitespace?

Trailing whitespace happens due to two reasons:

CRLF : Windows uses CRLF(carriage return line feed: \r\n) and Unix uses LF(line feed \n).
Trailing whitespace: File contains lines ending with space.

Why do I need to know how to solve rejected push & Git conflict?

Rejected push and Git conflict are something every Git users will meet eventually. It's important to know how to solve them.

Case study

For example, You type "I'm faithful to Logseq." in your journal on your pc, but you also type "Na, I also use other note-taking tools." in your journal on your phone at the same time. GitHub will accept the first commit you push to it. But when you push the second commit, Git will say something like:

error: failed to push some refs to 'github.com:{your-username}/{your-reponame}.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

which basically means: "WTF Bruh, I can't believe you just lied to me. How can I trust you again?"

So you type git pull in your Git Bash/iTerms/Termux to gain GitHub's trust again, like what Git suggested in hint:. And that's how to resolve most of the rejected push.

However, sometimes that's not enough. After using git pull to update your local repo, Git will say:

Auto-merging {the conflict file} #e.g. journals/2022_04_23.md
CONFLICT (content): Merge conflict in {the conflict file}
Automatic merge failed; fix conflicts and then commit the result.

which means you have to manually delete which part you don't want.

So you open {the conflict file} with whatever text editor you like and fix it.

In the text editor:

<<<<<<< HEAD
- I’m faithful to Logseq. # Ok, I choose to keep this part.
=======
- Na, I also use other note-taking tools.
>>>>>>> 744f5cf94a46da43f5b318dab74c0f672bae31e2

After deletion:

- I’m faithful to Logseq.

Open Logseq and that let Logseq do the rest (Logseq will commit and push due to git hooks setting.) for you, and the Git conflict should be resolved. Now you should only see the chosen part remain in your GitHub repo. And you are again a happy Logseq user now! 😍

※ Noted: For Android users, you have to manually commit and push the changes since your workflow does not contain git hooks I provided.

※ Noted: Sometimes, it's the logseq/metadata.edn or logseq/pages-metadata.edn having Git conflicts. This is trickier because you cannot tell easily which part is the one you need to keep. In this case I would suggest simply remove logseq/metadata.edn or logseq/pages-metadata.edn and do a git pull again, that will restore it from the one coming from GitHub. After pulling logseq/metadata.edn or logseq/pages-metadata.edn from GitHub, Re-index and Refresh Logseq are advised.

Trailing whitespace handling

In your Git Bash/iTerms/Termux, turn off crlf check in your Git config.

git config --global core.autocrlf false
git config --global core.safecrlf false

and try to push again.

If that's not enough, add below code in your .git/hooks/pre-commit to remove trailing whitespace.

if git rev-parse --verify HEAD >/dev/null 2>&1
then
    against=HEAD
else
    # Initial commit: diff against an empty tree object
    against=4b825dc642cb6eb9a060e54bf8d69288fbee4904 # <the first commit id of your repo>
fi

# Find files with trailing whitespace
for FILE in `exec git diff-index --check $against -- | sed '/^[+-]/d' | (sed -r 's/:[0-9]+:.*//' > /dev/null 2>&1 || sed -E 's/:[0-9]+:.*//') | uniq` ; do
    # Fix them!
    (sed -i 's/[[:space:]]*$//' "$FILE" > /dev/null 2>&1 || sed -i '' -E 's/[[:space:]]*$//' "$FILE")
    git add "$FILE"
    echo "NOTE: removed trailing whitespace from $FILE"
done

and try to push again.

🪜 Workflow

Set up Git

This is like typing in the account/password, so make sure you follow ALL the steps in set-up-git.

※ Noted: SSH-keys is preferred to use in this workflow, PLEASE set it properly, PLEASE. Connecting over SSH Section Link

Create a private repo

This is like creating a folder in GitHub, follow create-a-repo to create a private repo.

※ Noted: If you create a public repo, that means everyone on GitHub can see what you put in there. You don't want that, do you?

Add .gitignore in your repo (Optional)

In your repo page > Add file > Create new file > Name your file: .gitignore. Type those:

logseq/bak/
logseq/.recycle

Download this repo, for computer users

Click the green Code button of this repo and Download ZIP, then unzip it for later use.

For Windows users

After Git is set and a private repo is created:

Go to your local drive (for example: D:\) and right-click on it.
You should see Git Bash Here, click it.
Type git clone git@github.com:{your-username}/{your-reponame}.git and hit enter.
If it's your first time git clone something from GitHub, it will probably ask if you agree to authorized the connection, just type Yes and hit enter.
After it's done, you should see a new folder with your repo name.
Open the folder, there should have a hidden folder named .git.
Copy&paste the post-commit and pre-commit in .git/hooks. (Note: this folder could be hidden)
Open Logseq and add the folder with .git as your new graph.
Open Logseq > Settings > Version control > toggle on "Enable Git auto commit".
1. If you don't want auto-commit to commit in the middle of writing, you can use the git-plugin by haydenull to commit manually by yourself.
Type something and wait few minutes to see if what you typed has also appear in GitHub.
If nothiong goes wrong, you are a happy Logseq user!

For MacOS users

After Git is set and a private repo is created:

Go to your local drive and right-click on it.
You should see New Terminal at folder, click it. If you don't see this option, check this link to enable it.
Type git clone git@github.com:{your-username}/{your-reponame}.git and hit enter.
If it's your first time git clone something from GitHub, it will probably ask if you agree to authorized the connection, just type Yes and hit enter.
After it's done, you should see a new folder with your repo name.
Open the folder, there should have a hidden folder named .git.
Copy&paste the post-commit and pre-commit in .git/hooks.
Right click in .git/hooks and click New Terminal at folder, type
```
chmod +x ./pre-commit && chmod +x ./post-commit
```
to make those files executable by MacOS.
Open Logseq and add the folder with .git as your new graph.
Open Logseq > Settings > Version control > toggle on "Enable Git auto commit".
1. If you don't want auto-commit to commit in the middle of writing, you can use the git-plugin by haydenull to commit manually by yourself.
Type something and wait few minutes to see if what you typed has also appear in GitHub.
If nothing goes wrong, you are a happy Logseq user!

For iOS/iPadOS users

※ Noted: For people need more graphical instructions, check this post by martyna.

Pay for the Pro version of the App Working Copy, it is cheap since you will never bother by iCloud crashing your Logseq anymore.
Open Working Copy > Settings(⚙️) > Hosting Providers > Click Test > Sign-in to your GitHub account.
Open Working Copy > click + > Clone repository > Choose your repo.
When download is done, long-press on the repo > Share > Link Repository to Folder > My iPhone/iPad (I would suggest don't choose iCloud) > Logseq-icon folder > create a new sub-folder under the Logseq-icon one (I would suggest don't name the folder with the same name as your repo.) > click Done
Open Logseq and add new graph to that folder.
Open ShortCuts > Automation > Create Personal Automation:
- On App Open:
  - App > choose Logseq > click Next
  - Search actions > Pull Repository - Repo: your Logseq repo
  - Turn off ask before running (or leave it on up to you)
- On App Close:
  - App > choose Logseq > click Next
  - Search actions >
    - Stage for Commit - Path: *, Repo: your Logseq repo
    - Commit Repository - Repo: your Logseq repo, Message: Auto-commit from iOS/iPadOS
    - Noted: In Commit Repository, you can toggle off Fail when nothing to Commit if you find the failure notification annoying. Although I prefer to keep it toggle on to prevent empty commit and empty push.
    - Push Repository - Repo: your Logseq repo
  - Turn off ask before running (or leave it on up to you)
Type something, exit Logseq, and wait few minutes to see if what you typed has also appear in GitHub.
If nothing goes wrong, you are a happy Logseq user!

※ Noted: If you are a student, you can sign up github student pack and apply for free Working Copy Pro.

For Android users, by @Abulafia⁴

Initial steps to install git and link it with logseq

Install Termux in Android, using the F-droid app. Follow instructions here.
Open Termux and type pkg install git to install Git.
Configure Git username and email:
```
git config --global user.name "John Doe"
git config --global user.email johndoe@example.com`
```
※ Noted: This is the name and email which will go into commits, it can be anything, not necessary related to your GitHub account.
In the Termux terminal, create a folder for your graph, for example cd documents; mkdir MyGraph and clone your repo there with the command git clone git@github.com:{your-username}/{your-reponame}.git ~/documents/MyGraph
Open Logseq and add a graph on that folder. Check that all looks ok.

Install shortcuts to synchronize the repository

Install Termux:Widgets from F-droid.
In Termux, type cd ~/.shortcuts and edit a file named, for example, pull-graph containing:
```
#!/usr/bin/bash
source bin/source-ssh-agent
cd {your repo location}  # eg: cd ~/documents/MyGraph
git pull
```
※ Noted: If you use password-free private keys for Git you don't need the ssh-agent line. You can include it to avoid typing the private key passphrase each time.

In the same ~/.shortcuts folder create a second one named, for example, pull-graph containing:

#!/usr/bin/bash
source bin/source-ssh-agent
cd {your repo location}
git add -A
git commit -m "sync from android"
git push

Add a widget for Termux Widget app to the Android screen, which will show the scripts available in ~/.shortcuts (in our example, it will show pull-graph and push-graph). Taping on them, they will be executed by Termux.

Android-Termux-Workflow

Tap pull-graph to download the most recent version of your graph from GitHub.
Open Logseq and admire the latest version of your notes.
Edit/add notes, work on them.
Exit Logseq and tap push-graph to upload your changes to GitHub.

For Linux users

Linux users click here

If you are a Linux user, why are you still watching this repo? Shooo. 😎

Or take the steps in MacOS's section as reference.

References

Working Copy on iOS, by @danieltomasz¹.
Steps to auto sync iOS + Windows, by @Caps³.
Gitea - self hosted git server, by @Sawhney².
Self Managed Sync + Android setup, by @danzu⁵ and @Abulafia⁴.
git hooks settings
What is github?
How does Git work?
Repository size limits for GitHub.com
About large files on GitHub
Differences between CRLF and LF, in Chinese.

Footnotes

danieltomasz#5748 ↩ ↩²
Sawhney#7320 ↩ ↩²
Caps#0219 ↩ ↩²
Abulafia#3734 ↩ ↩² ↩³
qwxlea#3490 ↩ ↩²
danzu#6567 ↩ ↩²

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Logseq-Git-Sync-101

Description

Credits

Why use Git + GitHub to sync Logseq graph?

Self Managed Sync Diagram, by @danzu⁶

Prerequisite

⚠️ Limitations of GitHub

⚠️ Things you should avoid

💥 Rejected push & Conflicts handling & Trailing whitespace

What is Rejected push?

What is Git Conflict?

What is Trailing whitespace?

Why do I need to know how to solve rejected push & Git conflict?

Case study

Trailing whitespace handling

🪜 Workflow

Set up Git

Create a private repo

Add .gitignore in your repo (Optional)

Download this repo, for computer users

For Windows users

For MacOS users

For iOS/iPadOS users

For Android users, by @Abulafia⁴

Initial steps to install git and link it with logseq

Install shortcuts to synchronize the repository

Android-Termux-Workflow

For Linux users

References

Files

README.md

Latest commit

History

README.md

File metadata and controls

Logseq-Git-Sync-101

Description

Credits

Why use Git + GitHub to sync Logseq graph?

Self Managed Sync Diagram, by @danzu6

Prerequisite

⚠️ Limitations of GitHub

⚠️ Things you should avoid

💥 Rejected push & Conflicts handling & Trailing whitespace

What is Rejected push?

What is Git Conflict?

What is Trailing whitespace?

Why do I need to know how to solve rejected push & Git conflict?

Case study

Trailing whitespace handling

🪜 Workflow

Set up Git

Create a private repo

Add .gitignore in your repo (Optional)

Download this repo, for computer users

For Windows users

For MacOS users

For iOS/iPadOS users

For Android users, by @Abulafia4

Initial steps to install git and link it with logseq

Install shortcuts to synchronize the repository

Android-Termux-Workflow

For Linux users

References

Footnotes

Self Managed Sync Diagram, by @danzu⁶

For Android users, by @Abulafia⁴