"File does not match original!" error when verifying provided example timestamps

[TomasJuocepis]

Running ots verify examples/hello-world.txt.ots or any other timestamps in the examples directory results in "File does not match original!" error.

Running on Windows 10, using Python 3.6.5.

OTS version: v0.6.1dev (built from latest commit: 1b32826)

I presume this might have something to do with windows.

[petertodd]

I wonder if this is related to line endings somehow? Is there any way you could upload the checked out version of the file somewhere? Or hash it?

…

On July 3, 2018 2:49:12 PM UTC, Tomas Juocepis ***@***.***> wrote: Running `ots verify examples/hello-world.txt.ots` or any other timestamps in the examples directory results in "File does not match original!" error. Running on Windows 10, using Python 3.6.5. OTS version: v0.6.1dev (built from latest commit: 1b32826) I presume this might have something to do with windows.

-- https://petertodd.org 'peter'[:-1]@petertodd.org

[TomasJuocepis]

Here's what I got after cloning the repo:

File: hello-world.txt.ots
SHA-1: ce624cf7a40cbcbeb6ca25b6e344e023c1826341
SHA-256: be61ecb940db2e4a6f89fbed59c8d4f7f74fa756cbc1fad1ad7aabcb2bf59c30

File: hello-world.txt
SHA-1: ea7b8499509da0261a19e48a8631a6a506f0de0a
SHA-256: cfce4e2952591e79a0dea1654a92dba4f099d348ab7c176bcd052d69b8929770

[TomasJuocepis]

File hex dumps:

File: hello-world.txt
48 65 6C 6C 6F 20 57 6F 72 6C 64 21 0D 0A

  Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 	
00000000: 48 65 6C 6C 6F 20 57 6F 72 6C 64 21 0D 0A          Hello.World!..

File: hello-world.txt.ots

  Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 	
00000000: 00 4F 70 65 6E 54 69 6D 65 73 74 61 6D 70 73 00    .OpenTimestamps.
00000010: 00 50 72 6F 6F 66 00 BF 89 E2 E8 84 E8 92 94 01    .Proof.?.bh.h...
00000020: 08 03 BA 20 4E 50 D1 26 E4 67 4C 00 5E 04 D8 2E    ..:.NPQ&dgL.^.X.
00000030: 84 C2 13 66 78 0A F1 F4 3B D5 4A 37 81 6B 6A B3    .B.fx.qt;UJ7.kj3
00000040: 40 03 F1 C8 01 01 00 00 00 01 E4 82 F9 D3 2E CC    @.qH......d.yS.L
00000050: 3B A6 57 B6 9D 89 80 10 85 7B 54 45 7A 90 49 79    ;&W6.....{TEz.Iy
00000060: 82 FF 56 F9 7C 4E C5 8E 6F 98 01 00 00 00 6B 48    ..Vy|NE.o.....kH
00000070: 30 45 02 21 00 B2 53 AD D1 D1 CF 90 84 43 38 A4    0E.!.2S-QQO..C8$
00000080: 75 A0 4F F1 3F C9 E7 BD 24 2B 07 76 2D EA 07 F5    u.Oq?Ig=$+.v-j.u
00000090: 60 8B 2D E3 67 02 20 00 B2 68 CA 9C 33 42 B3 76    `.-cg...2hJ.3B3v
000000a0: 9C DD 06 28 91 31 7C DC EF 87 AA C3 10 B6 85 5E    .].(.1|\o.*C.6.^
000000b0: 9D 93 89 8E BB E8 EC 01 21 02 0D 8E 4D 10 7D 2B    ....;hl.!...M.}+
000000c0: 33 9B 00 50 EF DD 4B 4A 09 24 5A A0 56 04 8F 12    3..Po]KJ.$Z.V...
000000d0: 53 96 37 4E A6 A2 AB 07 09 C6 FF FF FF FF 02 65    S.7N&"+..F.....e
000000e0: 33 E6 05 00 00 00 00 19 76 A9 14 0B F0 57 D4 0F    3f......v)..pWT.
000000f0: BB A6 74 48 62 51 5F 5B 55 A2 31 0D E5 77 2F 88    ;&tHbQ_[U"1.ew/.
00000100: AC A0 86 01 00 00 00 00 00 19 76 A9 14 F0 06 88    ,.........v).p..
00000110: AC 00 00 00 00 08 08 F1 20 A9 87 F7 16 C5 33 91    ,......q.).w.E3.
00000120: 3C 31 4C 78 E3 5D 35 88 4C AC 94 3F A4 2C AC 49    <1Lxc]5.L,.?$,,I
00000130: D2 B2 C6 9F 40 03 F8 5F 88 08 08 F1 20 DE C5 5B    [email protected]_...q.^E[
00000140: 34 87 E1 E3 F7 22 A4 9B 55 A7 78 32 15 86 27 85    4.acw"$.U'x2..'.
00000150: F4 A3 AC B3 92 84 60 19 F7 1D C6 4A 9D 08 08 F1    t#,3..`.w.FJ...q
00000160: 20 B2 CA 18 F4 85 E0 80 47 8E 02 5D AB 3D 46 4B    .2J.t.`.G..]+=FK
00000170: 41 6C 0E 1E CB 66 29 C9 AE FC E8 C8 21 4D 04 24    Al..Kf)I.|hH!M.$
00000180: 32 08 08 F0 20 11 B0 E9 06 61 19 6F F4 B0 81 3C    2..p..0i.a.ot0.<
00000190: 3E DA 14 1B AB 5E 91 60 48 37 BD F7 A0 C9 DF 37    >Z..+^.`H7=w.I_7
000001a0: DB 0E 3A 11 98 08 08 F0 20 C3 4B C1 A4 A1 09 3F    [.:....p.CKA$!.?
000001b0: FD 14 8C 01 6B 1E 66 47 42 91 4E 93 9E FA BE 4D    }...k.fGB.N..z>M
000001c0: 3D 35 65 15 91 4B 26 D9 E2 08 08 F0 20 C3 E6 E7    =5e..K&Yb..p.Cfg
000001d0: C3 8C 69 F6 AF 24 C2 BE 34 EB AC 48 25 7E DE 61    C.iv/$B>4k,H%~^a
000001e0: EC 0A 21 B9 53 5E 44 43 27 7B E3 06 46 08 08 F1    l.!9S^DC'{c.F..q
000001f0: 20 07 98 BF 86 06 E0 00 24 E5 D5 D5 4B F0 C9 60    ...?..`.$eUUKpI`
00000200: F6 29 DF B9 DA D6 91 57 45 5B 6F 26 52 C0 E8 DE    v)_9ZV.WE[o&R@h^
00000210: 81 08 08 F0 20 3F 9A DA 6D 60 BA A2 44 00 6B B0    ...p.?.Zm`:"D.k0
00000220: AA D5 14 48 AD 2F AF B9 D4 B6 48 7A 09 99 CF F2    *U.H-//9T6Hz..Or
00000230: 6B 91 F0 F5 36 08 08 F1 20 C7 03 01 9E 95 9A 8D    k.pu6..q.G......
00000240: D3 FA EF 74 89 BB 32 8B A4 85 57 47 58 E7 09 1F    Szot.;2.$.WGXg..
00000250: 01 46 4E B6 58 72 C9 75 C8 08 08 F0 20 CB FE FF    .FN6XrIuH..p.K~.
00000260: F5 13 FF 84 B9 15 E3 FE D6 F9 D7 99 67 66 30 F8    u...9.c~VyW.gf0x
00000270: 36 4E A2 A6 C7 55 7F AD 94 A5 B5 D7 88 08 08 F1    6N"&GU.-.%5W...q
00000280: 20 0B E2 37 09 85 99 13 BA BD 44 60 BB DD F8 ED    ..b7....:=D`;]xm
00000290: 21 3E 7C 87 73 A4 B1 FA CE 30 F8 AC FD F0 93 B7    !>|.s$1zN0x,}p.7
000002a0: 05 08 08 00 05 88 96 0D 73 D7 19 01 03 F7 EF 15

[petertodd]

Yup, that's due to windows line mangling. Not sure why git would do that; check your gitconfig?

…

On July 3, 2018 5:51:32 PM UTC, Tomas Juocepis ***@***.***> wrote: File hex dumps: File: hello-world.txt 48 65 6C 6C 6F 20 57 6F 72 6C 64 21 0D 0A Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 00000000: 48 65 6C 6C 6F 20 57 6F 72 6C 64 21 0D 0A Hello.World!.. File: hello-world.txt.ots Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 00000000: 00 4F 70 65 6E 54 69 6D 65 73 74 61 6D 70 73 00 .OpenTimestamps. 00000010: 00 50 72 6F 6F 66 00 BF 89 E2 E8 84 E8 92 94 01 .Proof.?.bh.h... 00000020: 08 03 BA 20 4E 50 D1 26 E4 67 4C 00 5E 04 D8 2E ..:.NPQ&dgL.^.X. 00000030: 84 C2 13 66 78 0A F1 F4 3B D5 4A 37 81 6B 6A B3 .B.fx.qt;UJ7.kj3 00000040: 40 03 F1 C8 01 01 00 00 00 01 E4 82 F9 D3 2E CC @.qH......d.yS.L 00000050: 3B A6 57 B6 9D 89 80 10 85 7B 54 45 7A 90 49 79 ;&W6.....{TEz.Iy 00000060: 82 FF 56 F9 7C 4E C5 8E 6F 98 01 00 00 00 6B 48 ..Vy|NE.o.....kH 00000070: 30 45 02 21 00 B2 53 AD D1 D1 CF 90 84 43 38 A4 0E.!.2S-QQO..C8$ 00000080: 75 A0 4F F1 3F C9 E7 BD 24 2B 07 76 2D EA 07 F5 u.Oq?Ig=$+.v-j.u 00000090: 60 8B 2D E3 67 02 20 00 B2 68 CA 9C 33 42 B3 76 `.-cg...2hJ.3B3v 000000a0: 9C DD 06 28 91 31 7C DC EF 87 AA C3 10 B6 85 5E .].(.1|\o.*C.6.^ 000000b0: 9D 93 89 8E BB E8 EC 01 21 02 0D 8E 4D 10 7D 2B ....;hl.!...M.}+ 000000c0: 33 9B 00 50 EF DD 4B 4A 09 24 5A A0 56 04 8F 12 3..Po]KJ.$Z.V... 000000d0: 53 96 37 4E A6 A2 AB 07 09 C6 FF FF FF FF 02 65 S.7N&"+..F.....e 000000e0: 33 E6 05 00 00 00 00 19 76 A9 14 0B F0 57 D4 0F 3f......v)..pWT. 000000f0: BB A6 74 48 62 51 5F 5B 55 A2 31 0D E5 77 2F 88 ;&tHbQ_[U"1.ew/. 00000100: AC A0 86 01 00 00 00 00 00 19 76 A9 14 F0 06 88 ,.........v).p.. 00000110: AC 00 00 00 00 08 08 F1 20 A9 87 F7 16 C5 33 91 ,......q.).w.E3. 00000120: 3C 31 4C 78 E3 5D 35 88 4C AC 94 3F A4 2C AC 49 <1Lxc]5.L,.?$,,I 00000130: D2 B2 C6 9F 40 03 F8 5F 88 08 08 F1 20 DE C5 5B ***@***.***_...q.^E[ 00000140: 34 87 E1 E3 F7 22 A4 9B 55 A7 78 32 15 86 27 85 4.acw"$.U'x2..'. 00000150: F4 A3 AC B3 92 84 60 19 F7 1D C6 4A 9D 08 08 F1 t#,3..`.w.FJ...q 00000160: 20 B2 CA 18 F4 85 E0 80 47 8E 02 5D AB 3D 46 4B .2J.t.`.G..]+=FK 00000170: 41 6C 0E 1E CB 66 29 C9 AE FC E8 C8 21 4D 04 24 Al..Kf)I.|hH!M.$ 00000180: 32 08 08 F0 20 11 B0 E9 06 61 19 6F F4 B0 81 3C 2..p..0i.a.ot0.< 00000190: 3E DA 14 1B AB 5E 91 60 48 37 BD F7 A0 C9 DF 37 >Z..+^.`H7=w.I_7 000001a0: DB 0E 3A 11 98 08 08 F0 20 C3 4B C1 A4 A1 09 3F [.:....p.CKA$!.? 000001b0: FD 14 8C 01 6B 1E 66 47 42 91 4E 93 9E FA BE 4D }...k.fGB.N..z>M 000001c0: 3D 35 65 15 91 4B 26 D9 E2 08 08 F0 20 C3 E6 E7 =5e..K&Yb..p.Cfg 000001d0: C3 8C 69 F6 AF 24 C2 BE 34 EB AC 48 25 7E DE 61 C.iv/$B>4k,H%~^a 000001e0: EC 0A 21 B9 53 5E 44 43 27 7B E3 06 46 08 08 F1 l.!9S^DC'{c.F..q 000001f0: 20 07 98 BF 86 06 E0 00 24 E5 D5 D5 4B F0 C9 60 ...?..`.$eUUKpI` 00000200: F6 29 DF B9 DA D6 91 57 45 5B 6F 26 52 C0 E8 DE ***@***.***^ 00000210: 81 08 08 F0 20 3F 9A DA 6D 60 BA A2 44 00 6B B0 ...p.?.Zm`:"D.k0 00000220: AA D5 14 48 AD 2F AF B9 D4 B6 48 7A 09 99 CF F2 *U.H-//9T6Hz..Or 00000230: 6B 91 F0 F5 36 08 08 F1 20 C7 03 01 9E 95 9A 8D k.pu6..q.G...... 00000240: D3 FA EF 74 89 BB 32 8B A4 85 57 47 58 E7 09 1F Szot.;2.$.WGXg.. 00000250: 01 46 4E B6 58 72 C9 75 C8 08 08 F0 20 CB FE FF .FN6XrIuH..p.K~. 00000260: F5 13 FF 84 B9 15 E3 FE D6 F9 D7 99 67 66 30 F8 u...9.c~VyW.gf0x 00000270: 36 4E A2 A6 C7 55 7F AD 94 A5 B5 D7 88 08 08 F1 6N"&GU.-.%5W...q 00000280: 20 0B E2 37 09 85 99 13 BA BD 44 60 BB DD F8 ED ..b7....:=D`;]xm 00000290: 21 3E 7C 87 73 A4 B1 FA CE 30 F8 AC FD F0 93 B7 !>|.s$1zN0x,}p.7 000002a0: 05 08 08 00 05 88 96 0D 73 D7 19 01 03 F7 EF 15

-- https://petertodd.org 'peter'[:-1]@petertodd.org

[TomasJuocepis]

@petertodd

You are right, 0D 0A is the windows /r/n line ending. I think by default git is configured to convert the line endings when pulling/pushing so that files work properly within windows.

Do you think OTS should support /r/n line endings? I think it would be useful.

[petertodd]

That's not possible unfortunately, as it'd become a way to manipulate a file without invalidating the timestamp. You'll just have to figure out how to fix your broken Git windows setup.

…

On July 3, 2018 5:56:19 PM UTC, Tomas Juocepis ***@***.***> wrote: @petertodd You are right, `0D 0A` is the windows /r/n line ending. I think by default git is configured to convert the line endings when pulling/pushing so that files work properly within windows. Do you think OTS should support /r/n line endings? I think it would be useful.

-- https://petertodd.org 'peter'[:-1]@petertodd.org

[petertodd]

Oh, and to be clear, OTS does support windows style line endings. It's just that it doesn't and can't ignore the line ending style if it has been changed from one type to another: those are modifications of the original file, invalidating the timestamp.

[TomasJuocepis]

Understandable. What if when file does not match the original, before throwing the usual error, OTS would check if the only difference is the line endings and if that turns out to be the case, a more user-friendly error could be thrown explaining the problem with the line endings? The common case is for verification to pass, so this would not be an overhead for verification when the regular check passes.

Line ending differences are a general PITA so any assistance from the software in that regard would be a nice UX improvement.

[petertodd]

OTS doesn't have the original file, so to check if line endings were the difference it'd have to rehash the file with converted line endings and try afat. I think that's getting excessive, and won't always be possible due to pipes. Better to just fix broken setups IMO.

…

On July 6, 2018 3:44:01 AM UTC, Tomas Juocepis ***@***.***> wrote: Understandable. What if when file does not match the original, before throwing the usual error, OTS would check if the only difference is the line endings and if that turns out to be the case, a more user-friendly error could be thrown explaining the problem with the line endings? The common case is for verification to pass, so this would not be an overhead for verification when the regular check passes. Line ending differences are a general PITA so any assistance from the software in that regard would be a nice UX improvement.

-- https://petertodd.org 'peter'[:-1]@petertodd.org