Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Customisation Sync timestamp changes lead to excessive updates #475

Open
zedtools opened this issue Jul 28, 2024 · 9 comments
Open

Customisation Sync timestamp changes lead to excessive updates #475

zedtools opened this issue Jul 28, 2024 · 9 comments

Comments

@zedtools
Copy link

zedtools commented Jul 28, 2024

Abstract

I recently switched from syncing hidden files to using customisation sync.

When customisation sync updates a file, it seems to update the timestamp of the file to the time the local file was modified. This is different to the way LiveSync updates .md files in the vault.

On top of that, customisation sync seems to both notify and update files when the timestamp of a file has changed. This leads to excessive updates of config files.

The only way around this is to "Select All Shiny" and manually apply individual changes where the contents have changed. This is something which should really be automatic, as it is a tedious process to do this each time.

Expected behaviour

  • If a configuration file's contents has not changed, do not notify if another device has a more recently modified file.
  • Alternatively, when updating a configuration file, set the file's timestamp to the time it was modified on the other device.

Actually happened

  • If Device 1 make a change to a configuration file, Device 2 will notify a file has changed and allow the changed to be applied.
  • After Device 2 applies the change, it saves a file with a newer timestamp.
  • Device 1 detects the same configuration file with a newer timestamp, and notifies the configuration has changed.
  • Clicking "Select All Shiny" followed by "Apply All Selected" will update the timestamps to a later time again.

With two devices, this is kind of manageable, but I sync four devices (desktop, laptop, iPhone, iPad), so this issue is magnified because one small configuration changes requires me to go back and forth between the four devices multiple times until the changes have been applied.

Reproducing procedure

  1. I have the following options enabled under customisation sync:
    • Per-file-saved customisation sync
    • Enable customisation sync
    • Notify customised
  2. Open the dialog and set all files to "Selective".
  3. On Device 1, make a change to a plugin's configuration (e.g. change a setting in the Dataview plugin), and wait for it to sync.
  4. Open Obsidian on Device 2, and wait for the notification "Some configuration has been arrived...".
  5. Click "Select All Shiny". The changed file comes up, so apply the changes by clicking "Apply All Selected".
  6. Go back to Device 1, and wait for the notification "Some configuration has been arrived...".
  7. Click "Select All Shiny". The changed file comes up again, so click "Apply All Selected".
  8. Go back to step 4 and repeat.

Report materials

Report from the LiveSync

Report from hatch
---- Obsidian info ----
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.4.16 Chrome/114.0.5735.289 Electron/25.8.1 Safari/537.36
---- remote config ----
cors:
  credentials: "true"
  headers: accept, authorization, content-type, origin, referer
  max_age: "3600"
  methods: GET, PUT, POST, HEAD, DELETE
  origins: app://obsidian.md,capacitor://localhost,http://localhost
chttpd:
  bind_address: any
  max_http_request_size: "4294967296"
  port: "5984"
  require_valid_user: "true"
admins: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
vendor:
  name: The Apache Software Foundation
feature_flags:
  partitioned||*: "true"
chttpd_auth:
  authentication_redirect: /𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷/_utils/session.html
  hash_algorithms: sha256, sha
  require_valid_user: "true"
indexers:
  couch_mrview: "true"
prometheus:
  additional_port: "false"
  bind_address: 127.0.0.1
  port: "17986"
httpd:
  WWW-Authenticate: Basic realm="couchdb"
  bind_address: 127.0.0.1
  enable_cors: "true"
  port: "5986"
smoosh:
  state_dir: ./data
couch_httpd_auth:
  authentication_db: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  secret: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  authentication_redirect: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchdb_engines:
  couch: couch_bt_engine
couchdb:
  database_dir: ./data
  max_document_size: "50000000"
  single_node: "true"
  uuid: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  view_index_dir: ./data

---- Plug-in config ---
version:0.23.18
remoteType: ""
useCustomRequestHandler: false
couchDB_URI: self-hosted(HTTPS)
couchDB_USER: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchDB_PASSWORD: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchDB_DBNAME: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
liveSync: true
syncOnSave: false
syncOnStart: false
savingDelay: 200
lessInformationInLog: false
gcDelay: 0
versionUpFlash: ""
minimumChunkSize: 20
longLineThreshold: 250
showVerboseLog: false
suspendFileWatching: false
trashInsteadDelete: true
periodicReplication: false
periodicReplicationInterval: 60
syncOnFileOpen: false
encrypt: false
passphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
usePathObfuscation: false
doNotDeleteFolder: false
resolveConflictsByNewerFile: false
batchSave: false
batchSaveMinimumDelay: 5
batchSaveMaximumDelay: 60
deviceAndVaultName: macbook
usePluginSettings: false
showOwnPlugins: false
showStatusOnEditor: true
showStatusOnStatusbar: true
showOnlyIconsOnEditor: false
usePluginSync: true
autoSweepPlugins: false
autoSweepPluginsPeriodic: false
notifyPluginOrSettingUpdated: true
checkIntegrityOnSave: false
batch_size: 25
batches_limit: 25
useHistory: true
disableRequestURI: true
skipOlderFilesOnSync: true
checkConflictOnlyOnOpen: false
showMergeDialogOnlyOnActive: false
syncInternalFiles: false
syncInternalFilesBeforeReplication: false
syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/, \/workspace(-mobile)?.json
syncInternalFilesInterval: 60
additionalSuffixOfDatabaseName: 32e6163397f8cc82
ignoreVersionCheck: false
lastReadUpdates: 23
deleteMetadataOfDeletedFiles: false
syncIgnoreRegEx: ""
syncOnlyRegEx: ""
customChunkSize: 50
readChunksOnline: false
watchInternalFileChanges: true
automaticallyDeleteMetadataOfDeletedFiles: 0
disableMarkdownAutoMerge: false
writeDocumentsIfConflicted: false
useDynamicIterationCount: false
syncAfterMerge: false
configPassphraseStore: ""
encryptedPassphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
encryptedCouchDBConnection: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
permitEmptyPassphrase: false
useIndexedDBAdapter: true
useTimeouts: false
writeLogToTheFile: false
doNotPaceReplication: false
hashCacheMaxCount: 300
hashCacheMaxAmount: 50
concurrencyOfReadChunksOnline: 30
minimumIntervalOfReadChunksOnline: 25
hashAlg: xxhash64
suspendParseReplicationResult: false
doNotSuspendOnFetching: false
useIgnoreFiles: false
ignoreFiles: .gitignore
syncOnEditorSave: false
pluginSyncExtendedSetting: {}
syncMaxSizeInMB: 50
settingSyncFile: ""
writeCredentialsForSettingSync: false
notifyAllSettingSyncFile: false
isConfigured: true
settingVersion: 0
enableCompression: false
accessKey: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
bucket: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷(0 letters)
endpoint: Not configured or AWS
region: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷(4 letters)
secretKey: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
useEden: false
maxChunksInEden: 10
maxTotalLengthInEden: 1024
maxAgeInEden: 10
disableCheckingConfigMismatch: false
displayLanguage: ""
enableChunkSplitterV2: false
disableWorkerForGeneratingChunks: false
processSmallFilesInUIThread: false
notifyThresholdOfRemoteStorageSize: 800
usePluginSyncV2: true
usePluginEtc: false
configPassphrase: ""
preset: ""
syncMode: LIVESYNC
dummy: 0

Plug-in log

Plug-in log
28/07/2024, 13:32:50->Initialized, NOW TRACKING!
28/07/2024, 13:32:50->Scanning customizations...
28/07/2024, 13:32:50->Scanning customizing files.
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/app.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/appearance.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/bookmarks.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/community-plugins.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/core-plugins-migration.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/core-plugins.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/global-search.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/graph.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/hotkeys.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/types.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/workspace-mobile.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/workspace.json.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/dataview.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/homepage.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/metadata-menu.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-linter.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-livesync.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-mind-map.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-minimal-settings.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-outliner.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-plugin-update-tracker.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-tasks-plugin.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-zoom.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/omnisearch.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/quickadd.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/templater-obsidian.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/todoist-sync-plugin.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/actions-uri.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/buttons.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/dataview.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/homepage.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/metadata-menu.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-importer.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-linter.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-livesync.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-mind-map.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-outliner.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-plugin-update-tracker.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-style-settings.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-tasks-plugin.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-zoom.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/omnisearch.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/quickadd.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/templater-obsidian.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/todoist-sync-plugin.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/SNIPPET/S - Checkboxes.css.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/SNIPPET/metadata-menu.css.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/SNIPPET/outliner.css.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/THEME/ITS Theme.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/THEME/Minimal.md: (config) already deleted (Not found on database)
28/07/2024, 13:32:51->Chunks saved: doc: macbook/CONFIG/workspace.json%workspace.json ,chunks: 1 (new:0, recycled:1, cached:0)
28/07/2024, 13:32:51->STORAGE --> DB:ix:macbook/CONFIG/workspace.json%workspace.json: (config) Done
28/07/2024, 13:32:51->Scanning customizations : done
28/07/2024, 13:32:51->Cache initialized 300 / 50000000
28/07/2024, 13:32:51->Modifying callback of the save command
28/07/2024, 13:32:51->Additional safety scan..
28/07/2024, 13:32:51->Checking storage sizes
28/07/2024, 13:32:51->Before LiveSync, start OneShot once...
28/07/2024, 13:32:51->OneShot Sync begin... (pullOnly)
28/07/2024, 13:32:51->Remote storage size: 198.37MB
28/07/2024, 13:32:52->Replication paused
28/07/2024, 13:32:52->Replication completed
28/07/2024, 13:32:52->LiveSync begin...
28/07/2024, 13:32:53->Replication activated
28/07/2024, 13:32:53->There are no conflicted files
28/07/2024, 13:32:53->Additional safety scan done
28/07/2024, 13:32:54->Replication paused
28/07/2024, 13:32:55->Replication closed
28/07/2024, 13:33:00->Before LiveSync, start OneShot once...
28/07/2024, 13:33:00->OneShot Sync begin... (pullOnly)
28/07/2024, 13:33:00->Replication paused
28/07/2024, 13:33:00->Replication completed
28/07/2024, 13:33:00->LiveSync begin...
28/07/2024, 13:33:00->Replication paused
28/07/2024, 13:33:00->Replication paused
28/07/2024, 13:33:02->STORAGE --> DB:ix:macbook/PLUGIN_DATA/dataview%data.json: (config) Done
28/07/2024, 13:33:02->Replication activated
28/07/2024, 13:33:02->Replication paused
28/07/2024, 13:33:06->Replication closed
28/07/2024, 13:33:14->Before LiveSync, start OneShot once...
28/07/2024, 13:33:14->OneShot Sync begin... (pullOnly)
28/07/2024, 13:33:15->Replication activated
28/07/2024, 13:33:15->Replication paused
28/07/2024, 13:33:15->Replication completed
28/07/2024, 13:33:15->LiveSync begin...
28/07/2024, 13:33:15->Replication activated
28/07/2024, 13:33:15->Replication paused
28/07/2024, 13:33:15->Replication paused

Screenshots

Here is an example showing files where the configuration has not changed, but the timestamps are different:

image

Other information, insights and intuition.

Launching Obsidian also seems to update the last modified timestamps for several files, such as app.json, appearance.json, etc.

This adds to the excessive update notifications on other devices. Since these timestamp changes are beyond the control of this plugin, perhaps ignoring timestamp changes and just looking at the file contents is the most robust solution.

@vrtmrz
Copy link
Owner

vrtmrz commented Jul 31, 2024

Thank you for your very detailed report! This report quite helps me!

As you mentioned, I missed the following two logics for skipping non-changed files:

  • Checking the actual content (As you mentioned), And keeping the pair of mtime for two changes.
  • If the storage file and stored one have different mtimes, check they are already paired.

And one more thing, the differences in time are also shown correctly.

This has been fixed at v0.23.20. Would you mind if I ask you to check the fixes, please?

@zedtools
Copy link
Author

Thanks for the quick update.

Yes, the mtime looked a bit odd, but I did not pay much attention to that. Now the files are showing a much more reasonable change in mtime such as several days.

image

As for the syncing of updates, I played around with it a little, and it looks like after updating the plugin I also had to click Select All Shiny and then Apply All Selected at least once, after which I no longer get the excessive notifications.

It looks like the behaviour now is:

  1. Select All Shiny shows all files where the mtime or contents have changed.
  2. Apply All Selected only applies changes where the contents have changed.
  3. After applying changes, Select All Shiny still shows files with a more recent from another device mtime.

@zedtools
Copy link
Author

zedtools commented Jul 31, 2024

On a side note, from a usability perspective it would help a lot to only display files whose contents have changed. I have 21 plugins, and it takes a while to scroll down to see what files may or may not have changed.

The ideal behaviour for me would be:

  1. When opening the dialog, by default show only files whose contents have changed.
  2. If I want to see more details, I can click a button or a checkbox to show all files, including mtime changes.

This way, if I get a notification that new configuration is available, I can open the dialog and quickly see what the relevant changes were. The more plugins installed, the more useful this behaviour would be.

Is this something you would consider?

@zedtools
Copy link
Author

I also have a question about the four options for each file:

  • Selective
  • Automatic
  • Ignore
  • Flagged Selective

The Automatic option does not seem to do anything. If I enable that for a file, I sometimes get a notification that a hidden file was updated, but the file contents are not changed. Any settings I changed do not propagate to a device with Automatic selected.

If I change it back to Selective, I can manually update it fine.

How is Automatic supposed to work?

@zedtools
Copy link
Author

zedtools commented Aug 3, 2024

Something is still not correct with the differences in mtime.

I just updated the Tasks plugin on my Mac and iPhone, but not yet on my iPad. Customisation sync is showing the iPad as having a newer version, which is incorrect.

I used Textastic to check the file modification times and grabbed the following screenshots. Note that all dates are dd/mm/yyyy.

Screenshots

These screenshots show the contents of .obsidian/plugins/obsidian-tasks-plugin on each platform.

File modified time

Mac

image

iPhone

image

iPad

Note that these are almost one month older:

iPad-files

Customisation sync

Why are my Mac and iPhone showing that my iPad has a newer version of MAIN?

Mac

image

iPhone

image

@0zd0
Copy link

0zd0 commented Sep 29, 2024

maybe I configured something wrong (half of the settings are described unclearly), but when opening obsidian makes its settings seem new, but in fact the new settings from the database do not have time to arrive and the plugin considers them obsolete (as I understood from the logs). as a result, new changes are rolled back when opening another device, as happens with iconize

@vrtmrz
Copy link
Owner

vrtmrz commented Sep 30, 2024

Sorry for being late!
@zedtools

How is Automatic supposed to work?

This means Leave it to the hidden file sync. We have to enable Hidden file sync. But it is certainly more complicated since it was first created. Indeed I think you should put one over the other, but there are pros and cons to both, and turning this to automatic when both are enabled can be very problematic.

And, some mtime problems have been fixed in recent versions. I am sorry to trouble you, but could you check if the same problem occurs again?

P.S. I found that handling mtime is necessary in some cases. So, the item can be selected even if there is only a difference in mtime.

@zedtools
Copy link
Author

And, some mtime problems have been fixed in recent versions. I am sorry to trouble you, but could you check if the same problem occurs again?

I think I saw it again about a week ago, specifically:

  1. I updated a plugin on Windows desktop.
  2. On Windows, customisation sync's Select All Shiny button selected another device (I think it was iPad).
  3. Any other device I updated did the same.
  4. Once I updated all devices to the latest plugin version, Select All Shiny no longer showed changes.

I can certainly keep an eye out for it and capture some more screenshots and data from the CouchDB database when it happens again.

@zedtools
Copy link
Author

zedtools commented Oct 3, 2024

I just had this happen again.

Since it seems to be a separate issue from this one, I have logged a separate issue for under #506.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants