tus 1.0 compatibility?

[kvz]

Hi, have/would you consider(ed) https://tus.io compatibility?

[pgaertig]

Hi, Absolutely there is an option to gradually adopt NBU as TUS-compatible server mostly for benefit of available client libraries (uppy!). BTW I was considering TUS for other private project a few years ago, however still there was a need of resumable SHA1/CRC32 for efficient content addressable storage.

[kvz]

That's awesome! Would be great to list you over at https://tus.io/implementations.html, I'm sure a good share of people would prefer plugging tus support into their existing nginx setup, vs running dedicated upload servers in addition to their webservers.

If there's anything that we (as tus creators, but unfortunately no lua/nginx experts) can do to assist I'd love to hear.

PS, have you seen our checksum extension? Would that cater to your crc32 needs? (admittedly I'm not entirely sure yet what the objective there)

[pgaertig]

You use checksums as chunk integrity check, am I right? Here, otther than integrity, CRC32 is calculated to feed on-the-fly resumable ZIP downloads with mod_zip, then SHA1 makes use in content-addressable storages, deduplication etc. Both are currently calculated on-the-fly over entire stream with resumability. That especially pays off with large (>1GB) files where back-end doesn't have to access files again to recalculate these checksums.

Aside, I think TUS-only fork would be more appropriate as mixing two possibly conflicting protocols would increase the effort and complexity of this module. By the way Lua/nginx knowledge is less than week effort, worth to comprehend for some other hiperf nginx plumbing. Just joined your slack to keep in touch.

[kvz]

That sounds really good! And welcome to our community 👌 /cc @Acconut

[Acconut]

You use checksums as chunk integrity check, am I right?

Yes, you are.

Here, otther than integrity, CRC32 is calculated to feed on-the-fly resumable ZIP downloads with mod_zip, then SHA1 makes use in content-addressable storages, deduplication etc. Both are currently calculated on-the-fly over entire stream with resumability. That especially pays off with large (>1GB) files where back-end doesn't have to access files again to recalculate these checksums.

That's a very interesting situation. However, I believe that computing the CRC32 and SHA1 is the responsibility of only the server. What role should the client have in this case?

Aside, I think TUS-only fork would be more appropriate as mixing two possibly conflicting protocols would increase the effort and complexity of this module.

I agree, this makes sense.

Just joined your slack to keep in touch.

Welcome :)

[mmatuska]

I have started looking into this and done some first coding.

@pgaertig tus requires the upload offset ("Upload-Offset") to be reported to the client on a HEAD request (and on PATCH request of course). How can I get that information from the storage backend to the request processor in headers state? If this issue is resolved it should be pretty easy to add tus compatibility. An OPTIONS request will deliver necessary information to the tus client and all other request will need to include the "Tus-Resumable: 1.0.0" header.

The Upload-Offset combined with Content-Length and Upload-Length can be used to calculate the range values. This is pretty portable.

In tus there is no Session-ID, but a final URL path is used instead (e.g. http://somehost/upload/abcdefgh12345678). We could consider the path as the Session-ID, this will work and make it compatilbe with current code.

[Acconut]

@mmatuska Nice plan 👍 Just wanted to offer my help if you have a question regarding tus itself!

[mmatuska]

@Acconut looks like reimplementing from scratch is going to be easier. But this module is a big inspiration for that. I am considering introducing a lua-resty-tus module. For starters I will support creation, checksum and termination extensions.

By the way protocol.md is missing the description of the creation-with-upload extension.

[mmatuska]

@Acconut https://github.com/mmatuska/lua-tus-server

[Acconut]

Amazing work!

By the way protocol.md is missing the description of the creation-with-upload extension.

That's correct. It still lives inside a PR (tus/tus-resumable-upload-protocol#88) since we haven't been able to test its effectiveness in a proper production environment.