Support for Typed text based notebooks

[OGALI]

Maybe add to roadmap: Add support for typed notes. Currently, handwritten notes are synced to obsidian, but NOT notes that are typed with keyboard.

When syncing into Obsidian, typed notes into remarkable appear blank in obsidian. A blank PDF is generated with the same number of pages, but pages are empty.

Expected to show the text :)

[Azeirah]

Yep, good point that I should add it to the roadmap.

I recall having worked on this a while back, but haven't looked at it in a while. I believe I worked together with ricklupton from https://github.com/ricklupton/rmscene to make it possible to read the text correctly. As far as I remember, the feature was nearly finished too.

It just so happened that my own ReMarkable broke :(

[OGALI]

@Azeirah Can we add @ricklupton to this issue :)
I can maybe make a pull request if you point me in the right direction

[Azeirah]

Rick has done in rmscene what was necessary: rmscene parses the data structures correctly and provides them for consumers of the library. I prefer not to bother him with things that are outside of the scope of his project, he is doing amazing work, but he is doing all of it for free.

See here for the discussion with Rick: ricklupton/rmscene#22

It's about rendering the text in remarks now:

https://github.com/Azeirah/remarks/tree/main/remarks

[Azeirah]

For what it's worth, I had it running almost perfectly. i believe all that needed to be done was get the spacing and the coordinates correct. I'll take a look if I have a local branch somewhere, it might have been deleted since last October.

Otherwise, I will get to work on that really soon anyway. I really want to make progress with remarks. The notebook rendering is by far the most important functionality of Scrybble.

[Azeirah]

Azeirah/remarks@1de4a9c

From what I can see this was the code as I had it. From the top of my head:

1. I was working on rendering the text in remarks before I encountered the issue I discussed with rick here: A styling span (bold/italic) can cover more than one paragraph, causing an unbalanced stack ricklupton/rmscene#22
2. The code you can see here: Azeirah/remarks@1de4a9c is the code as I left it before Rick's updates.

The newest data structure is a lot simpler, so it should be easier to get the text rendered with the code that I already have.

I will work on this the coming weekend.

What I could really use from you is the following

1. Make a notebook on your tablet
2. Write something with the folio
3. Export this file as PDF with the built-in ReMarkable export functionality. Attach the pdf to this issue (or send it to me via mail, whatever you prefer)
4. Sync this file with Scrybble too. When it is synced please press the "share with developer" button in the sync status page. See the image below.
5. I will attach these files to this Github issue then, so I can use them for testing.

Normally I do these steps myself, but like I said, my ReMarkable is non-functional 😅

[OGALI]

@Azeirah I don't have my remarkable right now, because I am waiting for the paper pro XD

However, I have done all the steps through remarkable connect windows app, which I believe would be equivalent.

Here is the PDF: github.pdf

[Azeirah]

github.zip

Here are the files!

[Azeirah]

Making good progress!

Couple of things for during development:

1. ReMarkable just stores the text as strings without positional information. Meaning that whoever takes responsibility for exporting the text as paragraphs needs to emulate their text layout engine. This is of course not easy whatsoever. For now, I've settled on an extremely simple algorithm that just breaks the text up by words, and creates a newline if adding the next word would exceed the line width. Perhaps it is a better approach to use SVG instead of PyMuPDF as a render target, and let the browser figure out text layout as well as exporting to PDF. This is what @ricklupton does in his remarkable export tool.
2. They support far more than just plain text. I believe they have checkboxes, bullet points and probably a few more things. These are not implemented whatsoever at this stage.
3. Page size and their coordinate system remains a mystery. This code cannot be released, even in the current state because I have been messing with the coordinate-system again, and all output is mangled all over the place. ReMarkable is doing some magic with their page dimensions.

Perhaps it might have to do with what we can see in SceneTreeNode.anchorMode?

[OGALI]

Hey! Awesome :)

Just a thought, but if you don't have the positional information, it might be messy when syncing a page that has both Handwritten and typed notes. Handwritten and typed notes will probably overlap.

Another thought, since you can get the strings right now and we're syncing to obsidian, maybe it would make sense to give the option to output .md files (if there is only text). Haven't thought whether it should be 1 or multiple .md files.

Looking at the file in acrobat, the pdf page size seems to be 6.18 x 8.25 in, unless there is extra content which will extend the page as long as you keep adding content.

[Azeirah]

Another thought, since you can get the strings right now and we're syncing to obsidian, maybe it would make sense to give the option to output .md files (if there is only text). Haven't thought whether it should be 1 or multiple .md files.

This is a good idea. Scrybble already generates markdown files with the texts of the highlights. I can easily add the written text to Obsidian. I will do so tomorrow.

[ricklupton]

I haven't looked at it for a little while, but I think there are improvements in this PR -- does that help? https://github.com/ricklupton/rmc/pull/5/files#diff-538deb7981b2fb5957c3bf53371d1c860bddac89e8c99ff99914e7670083ab1e

The drawings are anchored relative to the position of specific characters in the text, which means I think it's not possible to draw just the handwritten parts without also working out where the text is positioned.

[Azeirah]

I haven't looked at it for a little while, but I think there are improvements in this PR -- does that help? https://github.com/ricklupton/rmc/pull/5/files#diff-538deb7981b2fb5957c3bf53371d1c860bddac89e8c99ff99914e7670083ab1e

The drawings are anchored relative to the position of specific characters in the text, which means I think it's not possible to draw just the handwritten parts without also working out where the text is positioned.

I found that the debugging mode that was found in xochitl will make a huge difference in figuring out how the positioning and coordinate system works. It's always been the most difficult to understand part of rmdocs imo.