avx blake

[spapinistarkware]

This change is 

[spapinistarkware]

• Remove old merkle #531
• FRI using simple merkle #530
• Dumb down FRI #529
• avx merkle #528
• simple merkle benchmark #527
• Use simple merkle tree #526
• Simple mixed merkle tree #525
• avx blake #518 👈
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @spapinistarkware and the rest of your teammates on Graphite

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.40%. Comparing base (83002ed) to head (da3de7d).

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #518      +/-   ##
==========================================
+ Coverage   95.17%   95.40%   +0.22%     
==========================================
  Files          63       65       +2     
  Lines        9229     9678     +449     
  Branches     9229     9678     +449     
==========================================
+ Hits         8784     9233     +449     
  Misses        387      387              
  Partials       58       58              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ohad-starkware]

Reviewable status: 0 of 6 files reviewed, 1 unresolved discussion (waiting on @spapinistarkware)

---

src/commitment_scheme/blake2_hash.rs line 108 at r1 (raw file):

    fn finalize(mut self) -> Blake2sHash {
        if self.current_len != 0 {
            self.update(&[0; 64]);

finalize is not a regular compression,
maybe we should expose a finalize without a finalization flag

---

src/commitment_scheme/blake2_hash.rs line 114 at r1 (raw file):

    fn finalize_reset(&mut self) -> Blake2sHash {
        let hash = self.clone().finalize();

Suggestion:

let hash = Blake2sHash(unsafe{ std::mem::transmute_copy(&self.state)})

---

src/commitment_scheme/blake2s_avx.rs line 216 at r1 (raw file):

}

/// # Safety

doc/make private

---

src/commitment_scheme/blake2s_ref.rs line 1 at r1 (raw file):

//! Based on bla bla's impl. TODO: add the link.

just a reminder to not forget these

---

src/core/proof_of_work.rs line 100 at r1 (raw file):

    #[test]
    fn test_verify_proof_of_work_success() {

why delete this?

[spapinistarkware]

Reviewable status: 0 of 6 files reviewed, 1 unresolved discussion (waiting on @ohad-starkware)

---

src/commitment_scheme/blake2_hash.rs line 108 at r1 (raw file):

Previously, ohad-starkware (Ohad) wrote…

finalize is not a regular compression,
maybe we should expose a finalize without a finalization flag

For stark, we dont care about finalization.

[spapinistarkware]

Reviewable status: 0 of 6 files reviewed, 1 unresolved discussion (waiting on @ohad-starkware)

---

src/core/proof_of_work.rs line 100 at r1 (raw file):

Previously, ohad-starkware (Ohad) wrote…

why delete this?

This test depends on concrete values of the hash, so it's not really a POW test. The 3rd tests in this file is enough IMO

[ohad-starkware]

Reviewed 2 of 5 files at r2.
Reviewable status: 2 of 7 files reviewed, 3 unresolved discussions (waiting on @spapinistarkware)

---

src/commitment_scheme/blake2_hash.rs line 94 at r2 (raw file):

    fn update(&mut self, mut data: &[u8]) {
        while self.pending_len + data.len() >= 64 {

use Self::BLOCK_SIZE
(in the lower level stuff I think raw numbers is fine)

---

src/commitment_scheme/blake2_hash.rs line 101 at r2 (raw file):

            let words =
                unsafe { std::mem::transmute::<&[u8; 64], &[u32; 16]>(&self.pending_buffer) };
            blake2s_ref::compress(&mut self.state, words, 0, 0, 0, 0);

if count arguments = 0 is intentional, document

---

src/commitment_scheme/blake2_hash.rs line 139 at r2 (raw file):

            dst = dst.add(16 * 32);
        }
        // Handle the remainder.

below too

Suggestion:

        }
        
        // Handle the remainder.

---

src/commitment_scheme/blake2_hash.rs line 163 at r2 (raw file):

unsafe fn compress16(inputs: &[*const u8; 16], single_input_length_bytes: usize) -> [u8; 16 * 32] {
    let mut states = [set1(0); 8];
    let mut inputs = inputs.map(|input| input);

or take ownership

Suggestion:

*inputs

---

src/commitment_scheme/blake2_hash.rs line 166 at r2 (raw file):

    // Compress in chunks of 64 bytes.
    for _j in (0..single_input_length_bytes).step_by(64) {

Suggestion:

_

[spapinistarkware]

Reviewable status: 0 of 7 files reviewed, 3 unresolved discussions (waiting on @ohad-starkware)

---

src/commitment_scheme/blake2_hash.rs line 139 at r2 (raw file):

Previously, ohad-starkware (Ohad) wrote…

below too

Done.

---

src/commitment_scheme/blake2_hash.rs line 163 at r2 (raw file):

Previously, ohad-starkware (Ohad) wrote…

or take ownership

Done.

---

src/commitment_scheme/blake2_hash.rs line 166 at r2 (raw file):

    // Compress in chunks of 64 bytes.
    for _j in (0..single_input_length_bytes).step_by(64) {

Done.

[shaharsamocha7]

Reviewable status: 0 of 7 files reviewed, 8 unresolved discussions (waiting on @andrewmilson, @ohad-starkware, and @spapinistarkware)

---

src/commitment_scheme/blake2_hash.rs line 67 at r4 (raw file):

/// Wrapper for the blake2s Hashing functionalities.
/// This blake2s hasher is intended to be sued only when the structure of the input is agreed upon.

used

Code quote:

sued

---

src/commitment_scheme/blake2_hash.rs line 71 at r4 (raw file):

/// The blake2s permutation is used as a hash chain, with no counts, flags or proper padding.
#[derive(Clone, Debug)]
pub struct Blake2sHasher {

We should do finialize for the channel hash.
Only the merkle can reduce it

Code quote:

pub struct Blake2sHasher {

---

src/commitment_scheme/blake2_hash.rs line 73 at r4 (raw file):

pub struct Blake2sHasher {
    state: [u32; 8],
    pending_buffer: [u8; 64],

Document that this is double than the state size.

Code quote:

pending_buffer: [u8; 64],

---

src/commitment_scheme/hasher.rs line 81 at r4 (raw file):

            })
    }
}

revert

Code quote:

    ) {
        data.iter()
            .map(|p| std::slice::from_raw_parts(*p, single_input_length_bytes))
            .zip(
                dst.iter()
                    .map(|p| std::slice::from_raw_parts_mut(*p, Self::OUTPUT_SIZE)),
            )
            .for_each(|(input, out)| {
                let mut hasher = Self::new();
                hasher.update(input);
                out.clone_from_slice(&hasher.finalize().into());
            })
    }
}

---

src/core/proof_of_work.rs line 106 at r4 (raw file):

        proof_of_work_prover.verify(&mut channel, &proof).unwrap();
    }

revert

Code quote:

    #[test]
    fn test_verify_proof_of_work_success() {
        let mut channel = Blake2sChannel::new(Blake2sHash::from(vec![0; 32]));
        let proof_of_work_prover = ProofOfWork { n_bits: 11 };
        let proof = ProofOfWorkProof { nonce: 133 };

        proof_of_work_prover.verify(&mut channel, &proof).unwrap();
    }

[spapinistarkware]

Reviewable status: 0 of 7 files reviewed, 8 unresolved discussions (waiting on @andrewmilson, @ohad-starkware, and @shaharsamocha7)

---

src/commitment_scheme/blake2_hash.rs line 67 at r4 (raw file):

Previously, shaharsamocha7 wrote…

used

Done.

---

src/commitment_scheme/blake2_hash.rs line 71 at r4 (raw file):

Previously, shaharsamocha7 wrote…

We should do finialize for the channel hash.
Only the merkle can reduce it

Done.

---

src/commitment_scheme/blake2_hash.rs line 73 at r4 (raw file):

Previously, shaharsamocha7 wrote…

Document that this is double than the state size.

Done.

---

src/commitment_scheme/hasher.rs line 81 at r4 (raw file):

Previously, shaharsamocha7 wrote…

revert

Done.

---

src/core/proof_of_work.rs line 106 at r4 (raw file):

Previously, shaharsamocha7 wrote…

revert

Done.

[shaharsamocha7]

Reviewed 1 of 5 files at r2, 3 of 3 files at r5.
Reviewable status: 4 of 7 files reviewed, 8 unresolved discussions (waiting on @andrewmilson and @ohad-starkware)

[shaharsamocha7]

Reviewed 1 of 3 files at r4, all commit messages.
Reviewable status: 5 of 7 files reviewed, 6 unresolved discussions (waiting on @andrewmilson, @ohad-starkware, and @spapinistarkware)

a discussion (no related file):
move this logic to avx_backend

---

---

src/commitment_scheme/blake2s_avx.rs line 179 at r5 (raw file):

}

const EVENS_CONCAT_EVENS: __m512i = unsafe {

Can this be unified with the fft into transpose_utils?

Code quote:

const EVENS_CONCAT_EVENS: __m512i = unsafe {

---

src/commitment_scheme/blake2s_avx.rs line 209 at r5 (raw file):

/// # Safety
pub unsafe fn transpose_msgs(mut data: [__m512i; 16]) -> [__m512i; 16] {
    // Transpose by applying 4 times the index permutation:

applying 4 times rotr(index, 1)
Maybe write the result of applying that 4 times

Code quote:

// Transpose by applying 4 times the index permutation:

[spapinistarkware]

Reviewable status: 5 of 7 files reviewed, 6 unresolved discussions (waiting on @andrewmilson, @ohad-starkware, and @shaharsamocha7)

a discussion (no related file):

Previously, shaharsamocha7 wrote…

move this logic to avx_backend

Done.

---

---

src/commitment_scheme/blake2s_avx.rs line 179 at r5 (raw file):

Previously, shaharsamocha7 wrote…

Can this be unified with the fft into transpose_utils?

Done.

---

src/commitment_scheme/blake2s_avx.rs line 209 at r5 (raw file):

Previously, shaharsamocha7 wrote…

applying 4 times rotr(index, 1)
Maybe write the result of applying that 4 times

Done.

[shaharsamocha7]

src/core/backend/avx512/blake2s_avx.rs line 190 at r6 (raw file):

    //   abcd:0123 => 3abc:d012
    // where abcd:0123 denotes that 0b 0123 word in the 0b abcd chunk.
    // In other words, rotate the index to the right by 1.

Each _m512i chunk contains 16 u32 words.
Index abcd:xyzw, refers to a specific word in data as follows:
abcd - chunk index (in base 2)
xyzw - word offset (in base 2)

Code quote:

    // Transpose by applying 4 times the index permutation:
    //   abcd:0123 => 3abc:d012
    // where abcd:0123 denotes that 0b 0123 word in the 0b abcd chunk.
    // In other words, rotate the index to the right by 1.

[shaharsamocha7]

Reviewed 1 of 3 files at r4, 4 of 5 files at r6, all commit messages.
Reviewable status: 9 of 10 files reviewed, 4 unresolved discussions (waiting on @andrewmilson, @ohad-starkware, and @spapinistarkware)

[spapinistarkware]

Reviewable status: 9 of 10 files reviewed, 4 unresolved discussions (waiting on @andrewmilson, @ohad-starkware, and @shaharsamocha7)

---

src/core/backend/avx512/blake2s_avx.rs line 190 at r6 (raw file):

Previously, shaharsamocha7 wrote…

Each _m512i chunk contains 16 u32 words.
Index abcd:xyzw, refers to a specific word in data as follows:
abcd - chunk index (in base 2)
xyzw - word offset (in base 2)

Done.

[shaharsamocha7]

Reviewed 1 of 5 files at r6, 1 of 1 files at r7, all commit messages.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @andrewmilson and @ohad-starkware)

Fixed