New storage format

[bobek]

Structure of the notes' backup (and storage I guess) has changed, at least as of firmware build 2021-03-31_19-14_3.1_0631e26. Backups are also password protected, successfully went through the #9 (thanks @timbertson).

The NewShapeModel doesn't have points anymore, but has a new column boundingRect (and also orientation, rotationPointXCoordinate and rotationPointYCoordinate which may play a role in the rendering). The backup archive contains a directory points with hierarchy of document-page-revision point files.

.
├── 4cff11cd-35df-469a-8a4a-6f590d14a809.db
├── fda759a8-8ae4-4787-9c15-9352e6821de3.db
├── point
│   ├── 4cff11cd-35df-469a-8a4a-6f590d14a809
│   │   └── b5e9ad82-fa04-4035-ba3b-268363a0d7b2
│   │       └── b5e9ad82-fa04-4035-ba3b-268363a0d7b2#db178691-532f-40f4-a824-736d0091a19a#points
│   ├── 6db1c121-096a-433b-9954-d230bdb1573b
│   └── fda759a8-8ae4-4787-9c15-9352e6821de3
│       ├── 8cc7215c-3bbd-412b-b000-3cefdcf944dc
│       │   └── 8cc7215c-3bbd-412b-b000-3cefdcf944dc#9542f3f9-32a8-4ef9-8c7e-b99438cc76bc#points
│       └── 9679ffb0-6d44-429d-b7b5-8bdbca0a158c
│           ├── 9679ffb0-6d44-429d-b7b5-8bdbca0a158c#3276f03e-d759-469f-a999-092627849bdc#points
│           ├── 9679ffb0-6d44-429d-b7b5-8bdbca0a158c#9273b738-a5cd-42c5-bb8a-e875c485eb4f#points
│           ├── 9679ffb0-6d44-429d-b7b5-8bdbca0a158c#94c2cdf3-c266-4100-8b3f-9adf61379559#points
│           └── 9679ffb0-6d44-429d-b7b5-8bdbca0a158c#ada9f053-0fcc-4b8c-a99d-308da8f9ff12#points
└── ShapeDatabase.db

I have not reverse-engineered the point file content yet. It seems to have a metadata at the beginning and the end. And point data, which may be similar to the original blog content.

Above listed backup with password stripped -- onyx_notes_backup.zip

[mleithner]

I wish I had more hours to spend on understanding the old and new format, in the meantime, here's a starting point for further research if anyone can look at it:
Check out com/onyx/android/sdk/data/point/v1/PointDocumentLoaderV1, it implements the loader for these documents. void parse(com/onyx/android/sdk/data/point/PointDocument, int, int) is the initial entry point, the first integer seems to be 0 at all times, the second is 0x7fffffff.

There's some sanity check on the size of the document, then it loads a List<ShapeRepo> (which are most likely the parts at the end of the file); each ShapeRepo is serialized as an entry of 36 bytes, the first 28 of which are a UTF-8 string (passed to setShapeUniqueId(String)), the next 4 bytes are an integer describing the offset (setOffset(int)) and the next are the length (setLength(int)). [NOTE: Please check if the length of the identifier is actually 28; the integers are definitely correct, but I might've screwed up a hex2dec in my brain…]

It then uses this list of metadata to find the actual points in each ShapeRepo, like this:
It positions itself at the offset of the ShapeRepo, allocates a buffer with the length, and reads into this buffer from the file. Using a DataInputStream over a ByteArrayInputStream over the bytearray created from this buffer, it first reads two shorts and passes them to ShapeRepo.setAttrA(short) and ShapeRepo.setAttrB(short).
The rest of the data consists of a list of TinyPoints that are serialized as 5 primitive values (for a total of 16 bytes) in the following order: Float, float, short, short, int. Please note that the order of shorts is switched when they're passed to the constructor of TinyPoint; the first short is passed as the second and vice versa. Even more confusingly, the constructor of TinyPoint itself switches them back.
At the end, the constructed list is passed to ShapeRepo.setTinyPointList().

As for TinyPoint, it has the following fields: Pressure, size, time, X, and Y.
The constructor of TinyPoint is TinyPoint(float x, float y, short pressure, short size, int time), BUT please keep in mind that the serialized format swaps the two shorts.

That's all for today, I hope it helps.

[DIGIHOME-MK]

Hi @bobek, @mleithner.
Is it possible to read data from new backup format? (devices on firmware 3.2, I've got Note Air 2)