Problems with coordinates

[Azeirah]

• Describe the issue
• Add clear photo explanation of the issue
• Why is it hard?
• Link related discussions
• Pool money for a solution?
• Get the right people together to fix it

What's the problem?

Highlights and annotations on ebooks and PDFs are not positioned correctly on output documents in Scrybble :(

This is by far the worst issue that Scrybble has. And despite trying my best two times, I failed to solve it two times. It's definitely not an insurmountable problem, but it's clearly difficult.

What does the problem look like?

Why is it hard?

ReMarkable's binary format for annotations is simply put, a mess. The biggest issue is that it is hard to understand.

For a start:

1. There are different versions. Remarks strives to support all versions of ReMarkable, whether it be notebooks made on the RM1 or on the recently released ReMarkable Paper Pro
2. The coordinate system isn't at all straightforward

• The easiest case: a notebook
• A more difficult case: A pdf gets scaled onto a remarkable

Who have expertise in this area?

• Me @Azeirah. I know remarks quite well, and I have ideas about how to approach automated testing for this.
• @ricklupton He's one of the best programmers in the ReMarkable scene. He's often busy unfortunately.
• @ddvk Without a doubt the best programmer and reverse engineer in the ReMarkable scene.

Pooling money

I'm not sure how to pool money, but I'm thinking of making Scrybble work on a sort of bug-bounty system. If you make an important contribution, you get paid with the money that I gather from Scrybble income.

I hereby pledge 250$ towards this issue.

Who are affected by this issue

In the past week alone, I have received two e-mails by Scrybble users who have run into this issue. Especially this e-mail really hit me hard. It has continued to be an issue for a long, long time.

I would firstly like to say thank you for creating Scrybble to help me streamline my Remarkable and my Obsidian more easily. I would like to raise an issue that happened with my most recent PDF. The attached PDF is which I had synced to Obsidian, and the screenshot attached below shows what the output PDF came out as on Obsidian. All the writings and drawings have been bunched up and clustered at the top of every page, making it illegible. It has come at a most unfortunate time because I have multiple tests next week and I hope that you can take care of this issue.

I have tried a few ways to troubleshoot the PDF, including deleting and resyncing between Scrybble and Obsidian, resyncing between the Remarkable app and tablet, and resetting the tablet. It only affects this PDF which, unfortunately, is the most important one. Thank you!

(emphasis mine)

Related issues

ricklupton/rmscene#27

[Azeirah]

Bug bounty pooling money for large contributions to this issue

• @Azeirah $250

[KMIJPH]

I am working on a project that involves exporting PDF documents with annotations made with the reMarkable 2.
I did not have the time to look into your code, so I don't know how you handle the conversion to PDF, but maybe my approach is of help to you.

1. do something similar to this
2. obtain page dimensions from your PDF file
3. use the widthand height parameters to scale the remarkable output
4. scale back

SCREEN_WIDTH: float = 1404
SCREEN_HEIGHT: float = 1872
SCREEN_DPI: float = 226
SCALE: float = 72.0 / SCREEN_DPI

def rm_to_svg(input: BytesIO, **kwargs: Any) -> str:
    """Converts a remarkable .lines file to a svg string."""
    blocks = read_blocks(input)
    return blocks_to_svg(blocks, **kwargs)

def to_pts(x: float) -> float:
    """Convert unit to pts."""
    return x * SCALE


def from_pts(x: float) -> float:
    """Convert unit from pts."""
    return x / SCALE

# using pypdf (background is a pypdf PageObject)
box = background.mediabox
width, height = from_pts(box.width), from_pts(box.height)
rm_str = rm_to_svg(b, width=width, height=height)
thispage = convert(rm_str.encode(), "svg")[0]  # convert the svg to pdf
thispage.scale_to(to_pts(SCREEN_WIDTH), to_pts(SCREEN_HEIGHT))
thispage.merge_page(background)

Cheers

[andymatuschak]

Hi, all. I encountered this issue as part of a prototype that involved rendering reMarkable lines. The failing scenario is easy to construct, at least for the class of issue I encountered: open a PDF which is in an aspect ratio other than 3:4 (i.e. distinct from the device), and draw in the margins to the edge of the page. You'll also see the failure if you adjust the viewport so that you're drawing "above" or "below" the PDF's crop box. Both remarks/rmc and the solution above from @KMIJPH will produce the wrong output when the "effective" aspect ratio of the annotation layer is distinct from the underlying PDF page.

The good news is that I fixed the problem in my implementation. The bad news is that my implementation is part of a hacky interaction design prototype and is implemented in terms of imperative <canvas> draw calls, rather than SVG or PDF output. I'm sorry not to be able to contribute a patch more directly!

But the key insight which let me fix this bug was to understand how the reMarkable document coordinate system actually works. Once I understood that, I could see that naively layering an SVG on top of the underlying PDF can't work in the general case. The origin of the SVG isn't necessarily the origin of the PDF, and the simple transform-x-by-half-the-page-width adjustment in these implementations only works in some cases.

Here's how the coordinate system works:

• Suppose a PDF with crop box of 300x400pt (you must use the crop box, not the media box, as the source listing in @KMIJPH's comment suggests).
• (0,0) in the reMarkable's coordinate system corresponds to the center of the top of the PDF page.
• The reMarkable's screen is 226 pixels per inch. PDFs points are assumed to measure 72 to an inch, so to get native reMarkable points, we scale by 226 / 72 = 3.139. Our 300x400pt PDF will span ~942 x 1256 points in the reMarkable coordinate system.
• The upper-right hand corner of the PDF will be (942 / 2, 0) = (471, 0).
• The upper-left is at (-471, 0).
• The lower-left and lower-right are at (-471, 1256) and (471, 1256).
• Annotations drawn above the top of the page will have negative y values. Annotations drawn to the left of the center of the page will have negative x values.
• Annotations in the right margin, beyond the PDF's edge, will have coordinates outside the PDF's box: e.g. (500, 0).

The tricky bit here is that if you make an SVG comprising all the annotations, you can't just assume that the upper-left corner of the SVG coincides with the upper-left corner of the PDF. They need to be composited to match the coordinate system I described above.

I hope that helps!

[Azeirah]

@andymatuschak Andy, what an honor to have you here, oh my God!

You literally popped up at the perfect time! I've been busy studying Joe Edelman's values-based social design for meaning. I've been trying to get in contact with him or them for a good time now, but I've been unable to do so for too long.

I've tried

1. Signing up for the mailing list
2. Signing up for turtle community
3. Mailing directly
4. Signing up for the course

None of it is working. All the tech is broken, lmao. I even emailed Joe about 8 years ago about a potential internship with him. Wasn't happening!

I'm working on the constitution of a business model, which is values based. I'd love to get feedback on it from someone with experience, so preferably from someone at the school for social design! Or a graduate (which you are! :D)