Skip to content

Commit

Permalink
Release v0.1.4 (#912)
Browse files Browse the repository at this point in the history
* fix: createReservation lock (#825)

* fix: createReservation lock

* fix: additional locking places

* fix: acquire lock

* chore: feedback

Co-authored-by: markspanbroek <[email protected]>
Signed-off-by: Adam Uhlíř <[email protected]>

* feat: withLock template and fixed tests

* fix: use proc for MockReservations constructor

* chore: feedback

Co-authored-by: Eric <[email protected]>
Signed-off-by: Adam Uhlíř <[email protected]>

* chore: feedback implementation

---------

Signed-off-by: Adam Uhlíř <[email protected]>
Co-authored-by: markspanbroek <[email protected]>
Co-authored-by: Eric <[email protected]>

* Block deletion with ref count & repostore refactor (#631)

* Fix StoreStream so it doesn't return parity bytes  (#838)

* fix storestream so it doesn\'t return parity bits for protected/verifiable manifests

* use Cid.example instead of creating a mock manually

* Fix verifiable manifest initialization (#839)

* fix verifiable manifest initialization

* fix linearstrategy, use verifiableStrategy to select blocks for slots

* check for both strategies in attribute inheritance test

* ci: add verify_circuit=true to the releases (#840)

* provisional fix so EC errors do not crash the node on download (#841)

* prevent node crashing with `not val.isNil` (#843)

* bump nim-leopard to handle no parity data (#845)

* Fix verifiable manifest constructor (#844)

* Fix verifiable manifest constructor

* Add integration test for verifiable manifest download

Add integration test for testing download of verifiable dataset after creating request for storage

* add missing import

* add testecbug to integration suite

* Remove hardhat instance from integration test

* change description, drop echo

---------

Co-authored-by: Eric <[email protected]>
Co-authored-by: gmega <[email protected]>

* Bump Nim to 1.6.21 (#851)

* bump Nim to 1.6.21 (range type reset fixes)

* remove incompatible versions from compiler matrix

* feat(rest): adds erasure coding constraints when requesting storage (#848)

* Rest API: add erasure coding constraints when requesting storage

* clean up

* Make error message for "dataset too small" more informative.

* fix API integration test

---------

Co-authored-by: gmega <[email protected]>

* Prover workshop band-aid (#853)

* add prover bandaid

* Improve error message text

Co-authored-by: Eric <[email protected]>
Signed-off-by: Giuliano Mega <[email protected]>

---------

Signed-off-by: Giuliano Mega <[email protected]>
Co-authored-by: Eric <[email protected]>

* Bandaid for failing erasure coding (#855)

* Update Release workflow (#858)

Signed-off-by: Slava <[email protected]>

* Fixes prover behavior with singleton proof trees (#859)

* add logs and test

* add Merkle proof checks

* factor out Circom input normalization, fix proof input serialization

* add test and update existing ones

* update circuit assets

* add back trace message

* switch contracts to fix branch

* update codex-contracts-eth to latest

* do not expose prove with prenormalized inputs

* Chronos v4 Update (v3 Compat Mode) (#814)

* add changes to use chronos v4 in compat mode

* switch chronos to compat fix branch

* use nimbus-build-system with configurable Nim repo

* add missing imports

* add missing await

* bump compat

* pin nim version in Makefile

* add await instead of asyncSpawn to advertisement queue loop

* bump DHT to v0.5.0

* allow error state of `onBatch` to propagate upwards in test code

* pin Nim compiler commit to avoid fetching stale branch

* make CI build against branch head instead of merge

* fix handling of return values in testslotqueue

* Downgrade to gcc 13 on Windows (#874)

* Downgrade to gcc 13 on Windows

Signed-off-by: Slava <[email protected]>

* Increase build job timeout to 90 minutes

Signed-off-by: Slava <[email protected]>

---------

Signed-off-by: Slava <[email protected]>

* Add MIT/Apache licenses (#861)

* Add MIT/Apache licenses

* Center "Apache License"

Signed-off-by: Giuliano Mega <[email protected]>

* remove wrong legal entity; rename apache license file

---------

Signed-off-by: Giuliano Mega <[email protected]>

* Add OPTIONS endpoint to allow the content-type header for the upload endpoint (#869)

* Add OPTIONS endpoint to allow the content-type header
exec git commit --amend --no-edit -S

* Remove useless header "Access-Control-Headers" and add cache

Signed-off-by: Arnaud <[email protected]>

---------

Signed-off-by: Arnaud <[email protected]>
Co-authored-by: Giuliano Mega <[email protected]>

* chore: add `downtimeProduct` config parameter (#867)

* chore: add `downtimeProduct` config parameter

* bump codex-contracts-eth to master

* Support CORS preflight requests when the storage request api returns an error  (#878)

* Add CORS headers when the REST API is returning an error

* Use the allowedOrigin instead of the wilcard when setting the origin

Signed-off-by: Arnaud <[email protected]>

---------

Signed-off-by: Arnaud <[email protected]>

* refactor(marketplace): generic querying of historical marketplace events (#872)

* refactor(marketplace): move marketplace events to the Market abstraction

Move marketplace contract events to the Market abstraction so the types can be shared across all modules that call the Market abstraction.

* Remove unneeded conversion

* Switch to generic implementation of event querying

* change parent type to MarketplaceEvent

* Remove extra license file (#876)

* remove extra license

* center "apache license"

* Update advertising (#862)

* Setting up advertiser

* Wires up advertiser

* cleanup

* test compiles

* tests pass

* setting up test for advertiser

* Finishes advertiser tests

* fixes commonstore tests

* Review comments by Giuliano

* Race condition found by Giuliano

* Review comment by Dmitriy

Co-authored-by: Dmitriy Ryajov <[email protected]>
Signed-off-by: Ben Bierens <[email protected]>

* fixes tests

---------

Signed-off-by: Ben Bierens <[email protected]>
Co-authored-by: Dmitriy Ryajov <[email protected]>

* feat: add `--payout-address` (#870)

* feat: add `--payout-address`

Allows SPs to be paid out to a separate address, keeping their profits secure.
Supports codex-storage/codex-contracts-eth#144 in the nim-codex client.

* Remove optional payoutAddress

Change --payout-address so that it is no longer optional. There is no longer an overload in `Marketplace.sol` for `fillSlot` accepting no `payoutAddress`.

* Update integration tests to include --payout-address

* move payoutAddress from fillSlot to freeSlot

* Update integration tests to use required payoutAddress

- to make payoutAddress required, the integration tests needed to avoid building the cli params until just before starting the node, otherwise if cli params were added ad-hoc, there would be an error after a non-required parameter was added before a required parameter.

* support client payout address

- withdrawFunds requires a withdrawAddress parameter, directs payouts for withdrawing of client funds (for a cancelled request) to go to that address.

* fix integration test

adds --payout-address to validators

* refactor: support withdrawFunds and freeSlot optional parameters

- withdrawFunds has an optional parameter for withdrawRecipient
- freeSlot has optional parameters for rewardRecipient and collateralRecipient
- change --payout-address to --reward-recipient to match contract signature naming

* Revert "Update integration tests to include --payout-address"

This reverts commit 8f9535c.
There are some valid improvements to the integration tests, but they can be handled in a separate PR.

* small fix

* bump contracts to fix marketplace spec

* bump codex-contracts-eth, now rebased on master

* bump codex-contracts-eth

now that feat/reward-address has been merged to master

* clean up, comments

* Rework circuit downloader (#882)

* Introduces a start method to prover

* Moves backend creation into start method

* sets up three paths for backend initialization

* Extracts backend initialization to backend-factory

* Implements loading backend from cli files or previously downloaded local files

* Wires up downloading and unzipping

* functional implementation

* Fixes testprover.nim

* Sets up tests for backendfactory

* includes libzip-dev

* pulls in updated contracts

* removes integration cli tests for r1cs, wasm, and zkey file arguments.

* Fixes issue where inner-scope values are lost before returning

* sets local proof verification for dist-test images

* Adds two traces and bumps nim-ethers

* Adds separate path for circuit files

* Create circuit dir if not exists

* fix: make sure requestStorage is mined

* fix: correct place to plug confirm

* test: fixing contracts tests

* Restores gitmodules

* restores nim-datastore reference

* Sets up downloader exe

* sets up tool skeleton

* implements getting of circuit hash

* Implements downloader tool

* sets up test skeleton

* Implements test for cirdl

* includes testTools in testAll

* Cleanup building.md

* cleans up previous downloader implementation

* cleans up testbackendfactory

* moves start of prover into node.nim

* Fills in arguments in example command

* Initializes backend in prover constructor

* Restores tests

* Restores tests for cli instructions

* Review comments by Dmitriy, part 1

* Quotes path in download instruction.

* replaces curl with chronos http session

* Moves cirdl build output to 'build' folder.

* Fixes chronicles log output

* Add cirdl support to the codex Dockerfile

Signed-off-by: Slava <[email protected]>

* Add cirdl support to the docker entrypoint

Signed-off-by: Slava <[email protected]>

* Add cirdl support to the release workflow

Signed-off-by: Slava <[email protected]>

* Disable verify_circuit flag for releases

Signed-off-by: Slava <[email protected]>

* Removes backendFactory placeholder type

* wip

* Replaces zip library with status-im/zippy library (which supports zip and tar)

* Updates cirdl to not change circuitdir folder

* Switches from zip to tar.gz

* Review comments by Dmitriy

* updates codex-contracts-eth

* Adds testTools to CI

* Adds check for access to config.circuitdir

* Update fixture circuit zkey

* Update matrix to run tools tests on Windows

* Adds 'deps' dependency for cirdl

* Adjust docker-entrypoint.sh to use CODEX_CIRCUIT_DIR env var

* Review comments by Giuliano

---------

Signed-off-by: Slava <[email protected]>
Co-authored-by: Adam Uhlíř <[email protected]>
Co-authored-by: Veaceslav Doina <[email protected]>

* Support CORS for POST and PATCH availability endpoints (#897)

* Adds testnet marketplace address to known deployments (#911)

* API tweaks for OpenAPI, errors and endpoints (#886)

* All sort of tweaks

* docs: availability's minPrice doc

* Revert changes to the two node test example

* Change default EC params in REST API

Change default EC params in REST API to 3 nodes and 1 tolerance.

Adjust integration tests to honour these settings.

---------

Co-authored-by: Eric <[email protected]>

---------

Signed-off-by: Adam Uhlíř <[email protected]>
Signed-off-by: Giuliano Mega <[email protected]>
Signed-off-by: Slava <[email protected]>
Signed-off-by: Arnaud <[email protected]>
Signed-off-by: Ben Bierens <[email protected]>
Co-authored-by: Adam Uhlíř <[email protected]>
Co-authored-by: markspanbroek <[email protected]>
Co-authored-by: Eric <[email protected]>
Co-authored-by: Tomasz Bekas <[email protected]>
Co-authored-by: Giuliano Mega <[email protected]>
Co-authored-by: Arnaud <[email protected]>
Co-authored-by: Ben Bierens <[email protected]>
Co-authored-by: Dmitriy Ryajov <[email protected]>
Co-authored-by: Arnaud <[email protected]>
  • Loading branch information
10 people authored Sep 24, 2024
1 parent 89917d4 commit 484124d
Show file tree
Hide file tree
Showing 91 changed files with 1,691 additions and 557 deletions.
6 changes: 6 additions & 0 deletions .github/actions/nimbus-build-system/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,12 @@ runs:
mingw-w64-i686-ntldd-git
mingw-w64-i686-rust
- name: MSYS2 (Windows All) - Downgrade to gcc 13
if: inputs.os == 'windows'
shell: ${{ inputs.shell }} {0}
run: |
pacman -U --noconfirm https://repo.msys2.org/mingw/ucrt64/mingw-w64-ucrt-x86_64-gcc-13.2.0-6-any.pkg.tar.zst https://repo.msys2.org/mingw/ucrt64/mingw-w64-ucrt-x86_64-gcc-libs-13.2.0-6-any.pkg.tar.zst
- name: Derive environment variables
shell: ${{ inputs.shell }} {0}
run: |
Expand Down
9 changes: 7 additions & 2 deletions .github/workflows/ci-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:

name: '${{ matrix.os }}-${{ matrix.cpu }}-${{ matrix.nim_version }}-${{ matrix.tests }}'
runs-on: ${{ matrix.builder }}
timeout-minutes: 80
timeout-minutes: 90
steps:
- name: Checkout sources
uses: actions/checkout@v4
Expand All @@ -53,7 +53,7 @@ jobs:
node-version: 18.15

- name: Start Ethereum node with Codex contracts
if: matrix.tests == 'contract' || matrix.tests == 'integration' || matrix.tests == 'all'
if: matrix.tests == 'contract' || matrix.tests == 'integration' || matrix.tests == 'tools' || matrix.tests == 'all'
working-directory: vendor/codex-contracts-eth
env:
MSYS2_PATH_TYPE: inherit
Expand All @@ -79,6 +79,11 @@ jobs:
path: tests/integration/logs/
retention-days: 1

## Part 4 Tools ##
- name: Tools tests
if: matrix.tests == 'tools' || matrix.tests == 'all'
run: make -j${ncpu} testTools

status:
if: always()
needs: [build]
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ jobs:
os {windows}, cpu {amd64}, builder {windows-latest}, tests {unittest}, nim_version {${{ env.nim_version }}}, shell {msys2}
os {windows}, cpu {amd64}, builder {windows-latest}, tests {contract}, nim_version {${{ env.nim_version }}}, shell {msys2}
os {windows}, cpu {amd64}, builder {windows-latest}, tests {integration}, nim_version {${{ env.nim_version }}}, shell {msys2}
os {windows}, cpu {amd64}, builder {windows-latest}, tests {tools}, nim_version {${{ env.nim_version }}}, shell {msys2}
build:
needs: matrix
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docker-dist-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
name: Build and Push
uses: ./.github/workflows/docker-reusable.yml
with:
nimflags: '-d:disableMarchNative -d:codex_enable_api_debug_peers=true -d:codex_enable_proof_failures=true -d:codex_enable_log_counter=true'
nimflags: '-d:disableMarchNative -d:codex_enable_api_debug_peers=true -d:codex_enable_proof_failures=true -d:codex_enable_log_counter=true -d:verify_circuit=true'
nat_ip_auto: true
tag_latest: ${{ github.ref_name == github.event.repository.default_branch || startsWith(github.ref, 'refs/tags/') }}
tag_suffix: dist-tests
Expand Down
102 changes: 67 additions & 35 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@ on:

env:
cache_nonce: 0 # Allows for easily busting actions/cache caches
nim_version: v1.6.14
nim_version: pinned
rust_version: 1.78.0
binary_base: codex
nim_flags: '-d:verify_circuit=true'
upload_to_codex: false
codex_binary_base: codex
cirdl_binary_base: cirdl
build_dir: build
nim_flags: ''
windows_libs: 'libstdc++-6.dll libgomp-1.dll libgcc_s_seh-1.dll libwinpthread-1.dll'

jobs:
# Matrix
Expand Down Expand Up @@ -69,19 +71,48 @@ jobs:
macos*) os_name="darwin" ;;
windows*) os_name="windows" ;;
esac
binary="${{ env.binary_base }}-${{ github.ref_name }}-${os_name}-${{ matrix.cpu }}"
[[ ${os_name} == "windows" ]] && binary="${binary}.exe"
echo "binary=${binary}" >>$GITHUB_ENV
codex_binary="${{ env.codex_binary_base }}-${{ github.ref_name }}-${os_name}-${{ matrix.cpu }}"
cirdl_binary="${{ env.cirdl_binary_base }}-${{ github.ref_name }}-${os_name}-${{ matrix.cpu }}"
if [[ ${os_name} == "windows" ]]; then
codex_binary="${codex_binary}.exe"
cirdl_binary="${cirdl_binary}.exe"
fi
echo "codex_binary=${codex_binary}" >>$GITHUB_ENV
echo "cirdl_binary=${cirdl_binary}" >>$GITHUB_ENV
- name: Release - Build
run: |
make NIMFLAGS="--out:${{ env.binary }} ${{ env.nim_flags }}"
- name: Release - Upload binaries
make NIMFLAGS="--out:${{ env.build_dir }}/${{ env.codex_binary }} ${{ env.nim_flags }}"
make cirdl NIMFLAGS="--out:${{ env.build_dir }}/${{ env.cirdl_binary }} ${{ env.nim_flags }}"
- name: Release - Libraries
run: |
if [[ "${{ matrix.os }}" == "windows" ]]; then
for lib in ${{ env.windows_libs }}; do
cp -v "${MINGW_PREFIX}/bin/${lib}" "${{ env.build_dir }}"
done
fi
- name: Release - Upload codex build artifacts
uses: actions/upload-artifact@v4
with:
name: release-${{ env.codex_binary }}
path: ${{ env.build_dir }}/${{ env.codex_binary_base }}*
retention-days: 1

- name: Release - Upload cirdl build artifacts
uses: actions/upload-artifact@v4
with:
name: release-${{ env.cirdl_binary }}
path: ${{ env.build_dir }}/${{ env.cirdl_binary_base }}*
retention-days: 1

- name: Release - Upload windows libs
if: matrix.os == 'windows'
uses: actions/upload-artifact@v4
with:
name: release-${{ env.binary }}
path: ${{ env.binary }}
name: release-${{ matrix.os }}-libs
path: ${{ env.build_dir }}/*.dll
retention-days: 1

# Release
Expand All @@ -100,46 +131,47 @@ jobs:
- name: Release - Compress and checksum
run: |
cd /tmp/release
prepare() {
# Checksum
checksum() {
arc="${1}"
sha256sum "${arc}" >"${arc}.sha256"
# Upload to Codex
if [[ "${{ env.upload_to_codex }}" == "true" ]]; then
codex_endpoints="${{ secrets.CODEX_ENDPOINTS }}"
codex_username="${{ secrets.CODEX_USERNAME }}"
codex_password="${{ secrets.CODEX_PASSWORD }}"
for endpoint in ${codex_endpoints}; do
echo "::add-mask::${endpoint}"
cid=$(curl -X POST \
"${endpoint}/api/codex/v1/data" \
-u "${codex_username}":"${codex_password}" \
-H "content-type: application/octet-stream" \
-T "${arc}")
echo "${cid}" >"${arc}.cid"
done
fi
}
# Compress and prepare
for file in *; do
for file in ${{ env.codex_binary_base }}* ${{ env.cirdl_binary_base }}*; do
if [[ "${file}" == *".exe"* ]]; then
# Windows - binary only
arc="${file%.*}.zip"
zip "${arc}" "${file}"
checksum "${arc}"
# Windows - binary and libs
arc="${file%.*}-libs.zip"
zip "${arc}" "${file}" ${{ env.windows_libs }}
rm -f "${file}"
prepare "${arc}"
checksum "${arc}"
else
# Linux/macOS
arc="${file}.tar.gz"
chmod 755 "${file}"
tar cfz "${arc}" "${file}"
rm -f "${file}"
prepare "${arc}"
checksum "${arc}"
fi
done
rm -f ${{ env.windows_libs }}
- name: Release - Upload compressed artifacts and checksums
uses: actions/upload-artifact@v4
with:
name: archives-and-checksums
path: /tmp/release/
retention-days: 1

- name: Release
uses: softprops/action-gh-release@v2
if: startsWith(github.ref, 'refs/tags/')
with:
files: |
/tmp/release/*
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
!*.*
*.exe

!LICENSE*
!Makefile

nimcache/
Expand Down
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -215,3 +215,6 @@
[submodule "vendor/nim-leveldbstatic"]
path = vendor/nim-leveldbstatic
url = https://github.com/codex-storage/nim-leveldb.git
[submodule "vendor/nim-zippy"]
path = vendor/nim-zippy
url = https://github.com/status-im/nim-zippy.git
12 changes: 11 additions & 1 deletion BUILDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ The current implementation of Codex's zero-knowledge proving circuit requires th
On a bare bones installation of Debian (or a distribution derived from Debian, such as Ubuntu), run

```shell
apt-get update && apt-get install build-essential cmake curl git rustc cargo
$ apt-get update && apt-get install build-essential cmake curl git rustc cargo
```

Non-Debian distributions have different package managers: `apk`, `dnf`, `pacman`, `rpm`, `yum`, etc.
Expand Down Expand Up @@ -157,6 +157,16 @@ In Bash run
make test
```

### Tools

#### Circuit download tool

To build the circuit download tool located in `tools/cirdl` run:

```shell
make cirdl
```

### testAll

#### Prerequisites
Expand Down
Loading

0 comments on commit 484124d

Please sign in to comment.