Add json module

[michalmuskala]

This implements the json module as specified in EEP 68.

[github-actions]

CT Test Results

    2 files     94 suites   34m 31s ⏱️
2 041 tests 1 993 ✅ 48 💤 0 ❌
2 350 runs  2 300 ✅ 50 💤 0 ❌

Results for commit c714e21.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• Windows Installer

// Erlang/OTP Github Action Bot

[dgud]

Some initial comments
Docs are a bit sparse :-)
Misses meta-data since 27.

But overall looks good to me.

[juhlig]

AFAIK, property tests are usually put in their own suite, using the ct_property_test framework as described here. At least all the property tests for other modules in OTP that I know of are organized this way. For example, see stdlib/test/base64_property_test_SUITE.erl and the corresponding stdlib/test/property_test/base64_prop.erl.

[michalmuskala]

This changes the string escaping implementation to read multiple bytes at once, and applies several other encoding optimisations. With those changes, in my benchmarks on an M1 mac, the new json module significantly outperforms Jason by 1.5-2.5x, and jiffy (except for one case where it's on-par).
This includes implementing custom UTF8 validation that significantly outperforms /utf8 patterns in case where we need to know the length of the codepoint, but don't need to know its value.
Unfortunately constructing binaries with private_append turned out to still be slower than building an IOlist of parts - even if benchmarked with a final iolist_to_binary call.

Btw, in the implementation of UTF8 validation using tuples and element calls to do lookups, rather than functions with integer patterns seems to be considerably faster - I believe this is because of optimisations the compiler tries to do by fusing certain clauses, which in this case leads to a pesimisation.

[josevalim]

This includes implementing custom UTF8 validation that significantly outperforms /utf8 patterns in case where we need to know the length of the codepoint, but don't need to know its value.

I can see such a function being useful in other scenarios. Perhaps it makes sense to export it as part of unicode?

[michalmuskala]

This includes implementing custom UTF8 validation that significantly outperforms /utf8 patterns in case where we need to know the length of the codepoint, but don't need to know its value.

I can see such a function being useful in other scenarios. Perhaps it makes sense to export it as part of unicode?

Unfortunately this function being efficient in here very much depends on it being placed in the same module as the caller allowing the compiler to analyse and optimise them together. I'm not convinced this would be particularly efficient as a standalone function. I'd prefer to focus on further optimising /utf8 patterns, so the "intuitive" way of doing things is also the fastest.

[codeadict]

This includes implementing custom UTF8 validation that significantly outperforms /utf8 patterns in case where we need to know the length of the codepoint, but don't need to know its value.

I can see such a function being useful in other scenarios. Perhaps it makes sense to export it as part of unicode?

Would this help with UTF-8 validation in Cowboy and Bandit?

[michalmuskala]

I've pushed commits that tune the performance of decoder (and tiny changes to encoder). This also includes the change to decoder to remove the empty_object and empty_array settings in favour of calling callbacks, e.g. ArrayFinish(ArrayStart(Acc)).

Current benchmark results for an M1 mac can be found in https://gist.github.com/michalmuskala/00e615f5de4305027abecffed45af07b

The decoder is the fastest out of all libraries I've checked, except for jiffy in couple specific data sets (notably lots of tiny strings and unicode-heavy data). In other cases the decoder is usually ~30% faster than Jason and jiffy (and up to 2x faster) and 2x faster than jsone (and up to 5x faster).

I've also improved documentation and added some more examples.

[michalmuskala]

Would this help with UTF-8 validation in Cowboy and Bandit?

cowboy already does this - this is actually how I learned about this technique, though I'm pretty sure my implementation is more performant.

https://github.com/ninenines/cowlib/blob/cc04201c1d0e1d5603cd1cde037ab729b192634c/src/cow_ws.erl#L581-L588

[essen]

Would this help with UTF-8 validation in Cowboy and Bandit?

cowboy already does this - this is actually how I learned about this technique, though I'm pretty sure my implementation is more performant.

Nice! I might have to revisit this, the VM has changed a lot this past decade.

[michalmuskala]

I've squashed the commits and rebased on top of latest master

[lpil]

Congratulations and thank you @michalmuskala!

[gonzalobf]

This changes the string escaping implementation to read multiple bytes at once, and applies several other encoding optimisations. With those changes, in my benchmarks on an M1 mac, the new json module significantly outperforms Jason by 1.5-2.5x, and jiffy (except for one case where it's on-par). This includes implementing custom UTF8 validation that significantly outperforms /utf8 patterns in case where we need to know the length of the codepoint, but don't need to know its value. Unfortunately constructing binaries with private_append turned out to still be slower than building an IOlist of parts - even if benchmarked with a final iolist_to_binary call.

Hey @michalmuskala,

Great work! Could you please share with us the code of the benchmark you ran?

Thanks