Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New add_crc pack function #7

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

New add_crc pack function #7

wants to merge 12 commits into from

Commits on Jun 28, 2012

  1. wrap pod, add header dependenies

    Reini Urban committed Jun 28, 2012
    Configuration menu
    Copy the full SHA
    3d66eac View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2012

  1. New add_crc feature

    Use empty reserved msgpack 0xc6 type for a crc32 as uint32 at the end of the buffer.
    See msgpack/msgpack#114 pull request.
    
    Optional CRC tag for basic security and data corruption. If pack provides a crc checksum,
    unpack should check the result against the given crc and report an error otherwise.
    The crc tag must be the last tag in the buffer. The crc checksum does not include itself
    and its type tag.
    Reini Urban committed Jun 29, 2012
    Configuration menu
    Copy the full SHA
    2caccfa View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2012

  1. add crc32 lookuptable variant

    Seperate simple crc32.c (byte advance).
    It is about 5% slower than the inlined macro version.
    Reini Urban committed Jul 20, 2012
    Configuration menu
    Copy the full SHA
    fc0c196 View commit details
    Browse the repository at this point in the history
  2. use zlib crc instead, only 2% overhead

    With zlib the crc overhead is only 2% for small buffers.
    Previously it was 10%.
    
    with -lz:
    $ pb benchmark/serialize.pl; pb benchmark/deserialize.pl
    -- serialize
    JSON::XS: 2.32
    Data::MessagePack: 0.46_01
    Storable: 2.35
    Benchmark: running json, mp, mp_crc, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.06 usr +  0.03 sys =  1.09 CPU) @ 136436.70/s (n=148716)
            mp:  1 wallclock secs ( 1.02 usr +  0.01 sys =  1.03 CPU) @ 185578.64/s (n=191146)
        mp_crc:  0 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 162293.40/s (n=172031)
      storable:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 91021.90/s (n=95573)
                 Rate storable     json   mp_crc       mp
    storable  91022/s       --     -33%     -44%     -51%
    json     136437/s      50%       --     -16%     -26%
    mp_crc   162293/s      78%      19%       --     -13%
    mp       185579/s     104%      36%      14%       --
    -- deserialize
    JSON::XS: 2.32
    Data::MessagePack: 0.46_01
    Storable: 2.35
    Benchmark: running json, mp, mp_crc, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.03 usr +  0.01 sys =  1.04 CPU) @ 97302.88/s (n=101195)
            mp:  1 wallclock secs ( 1.12 usr +  0.01 sys =  1.13 CPU) @ 126866.37/s (n=143359)
        mp_crc:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 124841.51/s (n=132332)
      storable:  1 wallclock secs ( 1.07 usr +  0.00 sys =  1.07 CPU) @ 80387.85/s (n=86015)
                 Rate storable     json   mp_crc       mp
    storable  80388/s       --     -17%     -36%     -37%
    json      97303/s      21%       --     -22%     -23%
    mp_crc   124842/s      55%      28%       --      -2%
    mp       126866/s      58%      30%       2%       --
    
    Compared to the old implementation, which had 10% overhead.
    And 18% with packing, but this difference is only the perl
    XS function call, not crc itself.
    
    $ pb benchmark/deserialize.pl
    -- deserialize
    JSON::XS: 2.32
    Data::MessagePack: 0.46_01
    Storable: 2.35
    Benchmark: running json, mp, mp_crc, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.09 usr +  0.00 sys =  1.09 CPU) @ 98642.20/s (n=107520)
            mp:  1 wallclock secs ( 1.11 usr +  0.00 sys =  1.11 CPU) @ 129153.15/s (n=143360)
        mp_crc:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 117027.62/s (n=122879)
      storable:  1 wallclock secs ( 1.08 usr +  0.00 sys =  1.08 CPU) @ 79643.52/s (n=86015)
                 Rate storable     json   mp_crc       mp
    storable  79644/s       --     -19%     -32%     -38%
    json      98642/s      24%       --     -16%     -24%
    mp_crc   117028/s      47%      19%       --      -9%
    mp       129153/s      62%      31%      10%       --
    Reini Urban committed Jul 20, 2012
    Configuration menu
    Copy the full SHA
    9368657 View commit details
    Browse the repository at this point in the history
  3. benchmark/deserialize.pl: add mp_crc (2% vs 10% slower)

    Reini Urban committed Jul 20, 2012
    Configuration menu
    Copy the full SHA
    7f1df22 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2012

  1. crc: adjust to msgpack-c - _msgpack_crc_user macro

    Reini Urban committed Jul 23, 2012
    Configuration menu
    Copy the full SHA
    d9d523b View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2012

  1. add roundtrip and multi-size benchmark tests

    Reini Urban committed Aug 15, 2012
    Configuration menu
    Copy the full SHA
    f7f8393 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2012

  1. fix comment typo in xs-src/pack.c

    Reini Urban committed Oct 16, 2012
    Configuration menu
    Copy the full SHA
    485b3fd View commit details
    Browse the repository at this point in the history
  2. Merge branch 'master' into crc

    Fixed conflicts:
    	Makefile.PL
    	lib/Data/MessagePack.pm
    	xs-src/pack.c
    Reini Urban committed Oct 16, 2012
    Configuration menu
    Copy the full SHA
    7ec41e5 View commit details
    Browse the repository at this point in the history
  3. define HAVE_ZLIB in sysdep.h

    Properly cast x->cur
    Reini Urban committed Oct 16, 2012
    Configuration menu
    Copy the full SHA
    ab53eb3 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f105563 View commit details
    Browse the repository at this point in the history
  5. helper to create static wordlist.pl

    Reini Urban committed Oct 16, 2012
    Configuration menu
    Copy the full SHA
    3a000b0 View commit details
    Browse the repository at this point in the history