Can I update dictionary files?

[soshial]

What happens if I close the app, replace the dictionary file with the updated version, and open the app again?

• will it crash?
• will my history be deleted?
• will my favourites be deleted?

@itkach

[itkach]

It won't crash, history and favorites won't be deleted and will continue to work. Taking a closer look at the code to answer your question I see one thing that will break is forgetting a dictionary - exactly what you reported in #186 :)

Each .slob dictionary file contains an id (a uuid). History and bookmark items include that id as well as value of slob's uri tag. uri tag is sort of a "logical" name for a dictionary, a namespace for lookup keys - slob files with the same value of uri tag are considered to be part of the same logical dictionary. If slob with the exact uuid is not found, then a slob with the same uri is used instead (if present). This is what allows history and bookmarks (favorites) to continue to work when dictionary is updated.

When dictionary file is replaced with a different file of the same name, loading dictionaries still works just fine: dictionary descriptors (each in a separate file) are simply slurped from a directory and the only thing that's used from the descriptor is file uri. "Forget" gets broken though, because "forget" deletes dictionary descriptor, which is a file with slob id for a name, and slob id in the newly loaded file is different. I guess #186 remained undetected until now because typically dictionary file name includes some sort of a version, and even if a user downloads a file with the same name browsers usually rename them (e.g. append number to the original file name), so having a dictionary with the same file name but different id is common only in development.

[soshial]

I saw you made a fix, but I've tried to replace the dictionary file with the newer version and my bookmarks got corrupted (see first 2 bookmarked words in the top of the screenshot). I guess this is the bug some users mentioned here. App version is v0.58

[itkach]

As I explained in my previous comment, when a dictionary is replaced, same file name or not, bookmark migration is based on uri tag. For bookmarks and history to continue to work both old and new dictionary file must have the uri tag set to the same value. As far as I can tell, pyglossary does not set uri tag at all - hence "broken" bookmarks.

[soshial]

I opened a dictionary that had been converted to Slob and search log printed this: 2024-08-06 22:02:01.997 GET /slob/c52bfba2-c166-46bf-8005-0ae51dcf5c1e/doinformowa%C4%87%20si%C4%99?blob=9-128 200.

Is c52bfba2-c166-46bf-8005-0ae51dcf5c1e the URI you're talking about? Is there a method in https://github.com/itkach/slob to set this URI (UUID?) during Slob file creation?

[itkach]

No, c52bfba2-c166-46bf-8005-0ae51dcf5c1e is a unique dictionary identifier, this is part of the fixed header defined by the format. Slob format also defines "tags" - text key-value pairs that may provide additional information about the dictionary. Such optional information is not needed for lookup functionality, but may be desired or necessary for applications - for example, aard2-android and aard2-web look for label tag and if present use it in UI as dictionary's display name and aard2-android uses uri tag to determine if two distinct dictionaries should be treated as versions or parts of the same logical data set. The file format does not prescribe any particular tags, tools producing dictionaries and apps using dictionaries are free to make their own conventions.

Is there a method in https://github.com/itkach/slob to set this URI (UUID?) during Slob file creation?

Yes. See for example here:

https://github.com/itkach/mw2slob/blob/69eb1c90d2cce75cb91601763bcf8fad8ff8366d/mw2slob/core.py#L176-L196

You can use slob cli to check basic info including tags, like so:

❯ slob info simplewiki-NS0-20240401-ENTERPRISE-HTML.slob


simplewiki-NS0-20240401-ENTERPRISE-HTML.slob
--------------------------------------------
         id: ca7be0b95b0644429da690e186a23c97
   encoding: utf-8
compression: lzma2
 blob count: 1088
  ref count: 1367


CONTENT TYPES
-------------
text/html;charset=utf-8
text/css
image/svg+xml
application/javascript


TAGS
----
version.python: 3.12.4 (main, Jun  6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)]
 version.pyicu: 2.12
   version.icu: 74.2
    created.at: 2024-06-28T18:48:11.144952+00:00
  license.name: Creative Commons Attribution-Share Alike 4.0
   license.url: https://creativecommons.org/licenses/by-sa/4.0/deed.simple
    created.by:
     copyright:
        source: http://simple.wikipedia.org
           uri: http://simple.wikipedia.org
         label: Wikipedia (en)

Tags must be added at dictionary creation, but value can be changed afterwards, for example using slob cli:

❯ slob tag -n created.by -v me simplewiki-NS0-20240401-ENTERPRISE-HTML.slob
❯ slob tag -n created.by simplewiki-NS0-20240401-ENTERPRISE-HTML.slob
me

[itkach]

Pyglossary already sets label tag:

https://github.com/ilius/pyglossary/blob/01b56060c06a58b51b696ca330a738bce6b6f810/pyglossary/plugins/aard2_slob.py#L303

Also setting uri tag should help with bookmark migration, it would also be good to set source (showed as a link in dictionary info in aard2-android), license.name and license.url (also shown in dictionary info) if possible.

[ilius]

I am very concerned about these:

version.python: 3.12.4 (main, Jun  6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)]
 version.pyicu: 2.12
   version.icu: 74.2

As a user, I would not expect any tool to automatically add any software info (OS name or version, versions of different software, home directory path, etc) to a file that I may want to publicly publish!
It has security and privacy implications!

[ilius]

That's why I added version_info argument to slog.Writer, and a write-option to pyglossary.
@soshial You could just set this option instead of adding slobWriter.tag(t_python, sys.version).

[itkach]

I am very concerned about these:

version.python: 3.12.4 (main, Jun  6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)]
 version.pyicu: 2.12
   version.icu: 74.2

As a user, I would not expect any tool to automatically add any software info (OS name or version, versions of different software, home directory path, etc)

But no such information is added. The only information here is versions of Python, PyICU and ICU which is relevant because they potentially affect the collation order of keys and may be helpful in debugging lookup issues. How exactly does this have security or privacy implications?

[soshial]

@ilius: I agree with @itkach that there's no harm in Python and ICU version, since those can help troubleshoot problems.