feat: rebuild only necessary canisters #3710

vporton · 2024-04-14T03:21:13Z

Description

DFX doesn't compile canisters for which all dependencies are elder than the .wasm file. This results in big compilation speedups.

The PR also adds tracing and comments intended for future improvement of compilation speed.

This PR is a WIP. There is potential for further speedup and there is missing logging. More testing is necessary.

How Has This Been Tested?

I run it before the .wasm file has been created and after it with a significant speedup. Also dfx build -vv correctly does not output any moc compilation logs, when invoking dfx build for the second time.

Later I did bats make_like.bash, add_dependency.bash, broken_canister_dep.bash, and dotenv.bash automated testing.

Checklist:

The title of this PR complies with Conventional Commits.
I have edited the CHANGELOG accordingly.
I have made corresponding changes to the documentation.

vporton · 2024-04-14T20:29:16Z

dfx deploy of my current DFX version running in https://github.com/vporton/zondirectory2 repo causes an infinite loop (but dfx build --all doesn't).

vporton · 2024-04-16T07:12:18Z

The last commit (79faa772dd3b8a7683ed9670a1447ecd0b2893ba) seems working!

Also note that unlike the original (master) code, my code does recompile dependent canisters (canister:* dependencies) when a dependency changed.

The code needs refactoring and more serious testing.

vporton · 2024-04-16T07:36:29Z

Testing shows a bug: When specifying a dependency to build: dfx build dependency, the dependent canister may recompile, too.

vporton · 2024-04-16T16:16:00Z

The latest version commit 030f9af902515edd70d60acde5d2cb547c200b40 works as expected on simple (manual) tests (with more tests than the past time).

You may try to use it for your home projects, but don't try it for mission-critical tasks or if you fear to be lost in wrong recompilations, because errors are still possible.

vporton · 2024-04-16T20:42:52Z

Fixed crash on dfx build --all (creates a new bug, about compiling non-Motoko canister: dependencies).

…ary-dev

vporton · 2024-05-31T13:50:41Z

This MR has been "battle-tested" a little. It worked as expected in developing https://github.com/vporton/zondirectory2 and in testing of https://github.com/vporton/joining-proxy-rust without any visible errors.

…ild-only-necessary

vporton · 2024-06-09T15:49:43Z

There has been found yet a bug:

Compiling vporton/zondirectory2@b145a4c it was output:

...

The command '"moc-wrapper" "src/backend/personhood.mo" "-o" "/home/porton/Projects/zondirectory/.dfx/local/canisters/personhood/personhood.wasm" "-c" "--debug" "--idl" "--stable-types" "--public-metadata" "candid:service" "--public-metadata" "candid:args" "--actor-idl" "/home/porton/Projects/zondirectory/.dfx/local/canisters/idl/" "--actor-alias" "CanDBIndex" "bkyz2-fmaaa-aaaaa-qaaaq-cai" "--actor-alias" "ic_eth" "avqkn-guaaa-aaaaa-qaaea-cai" "--package" "base" ".mops/[email protected]/src" "--package" "candb" ".mops/_github/candb#no-blob/src" "--package" "nacdb" ".mops/[email protected]/src" "--package" "stable-rbtree" ".mops/[email protected]/src" "--package" "stable-buffer" ".mops/_github/stable-buffer#v0.2.0/src" "--package" "prng" ".mops/[email protected]/src" "--package" "stableheapbtreemap" ".mops/[email protected]/src" "--package" "sha2" ".mops/[email protected]/src" "--package" "stablebuffer" ".mops/_github/stablebuffer#v0.2.0/src" "--package" "StableBuffer" ".mops/_github/StableBuffer#v0.2.0/src" "--package" "icrc1" ".mops/[email protected]/src" "--package" "array" ".mops/_github/array#main/src" "--package" "StableTrieMap" ".mops/_github/StableTrieMap#main/src" "--package" "itertools" ".mops/[email protected]/src" "--package" "base-0.7.3" ".mops/_github/base-0.7.3#aafcdee0c8328087aeed506e64aa2ff4ed329b47/src" "--package" "xtendedNumbers" ".mops/_github/xtendedNumbers#v1.1.0/src" "--package" "motoko-lib" ".mops/_github/motoko-lib#0.7/src" "--package" "map7" ".mops/_github/map7#v7.0.0/src" "--package" "sha" ".mops/_github/sha#master/src" "--package" "matchers" ".mops/_github/matchers#v1.3.0/src" "--package" "map" ".mops/[email protected]/src" "--package" "btree" ".mops/_github/btree#v0.3.3/src" "--package" "CanDBMulti" ".mops/_github/CanDBMulti#v0.10.0@86d00711216349ba4879370d6313d44a26d16f61/src" "--package" "xtended-numbers" ".mops/[email protected]/src" "--package" "json.mo" ".mops/[email protected]/src" "--package" "parser-combinators" ".mops/_github/parser-combinators#v0.1.2/src" "--package" "nacdb-reorder" ".mops/[email protected]/src" "--package" "passport-client" ".mops/[email protected]/src" "--package" "sequence" ".mops/_github/sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "--package" "stable-hash-map" ".mops/_github/stable-hash-map#v0.2.1@eb7edf4127233f2b1f3732ede7e2c55827ac6886/src" "--package" "encoding" ".mops/_github/encoding#v0.3.2@8e0fe1d8f5c2d284e77d719703c42e0e271839b1/src" "--package" "candy" ".mops/_github/candy#v0.3.0@907a4e7363aac6c6a4e114ebc73e3d3f21e138af/src" "--package" "stablebuffer_1_3_0" ".mops/_github/stablebuffer_1_3_0#v1.3.0@acdde6bb5b939227997cebdbb8919d2e6da8691c/src" "--package" "map9" ".mops/_github/map9#v9.0.1@10b68f6ea8df5e72dfa4c07a50c8bb60a916c233/src" "--package" "candb-multi" ".mops/[email protected]/src" "--package" "test" ".mops/[email protected]/src" "--package" "date.mo" ".mops/_github/date.mo#master@cfe51ad7d8a576d53e701136d4234233b94970eb/src" "--package" "testing" ".mops/_github/testing#main/src" "--package" "fmt" ".mops/_github/fmt#v0.1.0/src" "--package" "motoko-sequence" ".mops/_github/motoko-sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "-fshared-code" "--actor-idl" ".dfx/local/lsp"' failed with exit status 'exit status: 1'. Stdout:

Stderr: src/backend/personhood.mo:5.1-5.40: import error [M0009], file ".dfx/local/lsp/bkyz2-fmaaa-aaaaa-qaaaq-cai.did" does not exist

src/backend/personhood.mo:5.1-5.40 is about CanDBIndex (bkyz2-fmaaa-aaaaa-qaaaq-cai).

vporton · 2024-06-09T15:53:19Z

I've removed "that's weird" from my comment, because it has been written by me in error.

vporton · 2024-06-09T16:02:42Z

Note that the above error was caused by dfx start --background --clean && dfx deploy frontend.

vporton · 2024-06-09T17:57:45Z

There has been found yet a bug:

I run dfx deploy frontend 15 times and wasn't able to reprise the above bug. Probably, I just installed a wrong version of dfx and there is no bug in this version. I will keep using it in practice.

vporton · 2024-06-16T12:54:23Z

I've fixed a serious bug.

vporton · 2024-06-25T15:58:04Z

Found bugs, while attempting deploying a real app to ic network:

ic_eth (Rust) canister was decided to be compiled after a Motoko canister dependent on it.
Trying to deploy a real app to ic network:

Caused by: The command '"moc-wrapper" "/home/porton/Projects/zondirectory/src/backend/personhood.mo" "-o" "/home/porton/Projects/zondirectory/.dfx/ic/canisters/personhood/personhood.wasm" "-c" "--debug" "--idl" "--stable-types" "--public-metadata" "candid:service" "--public-metadata" "candid:args" "--actor-idl" "/home/porton/Projects/zondirectory/.dfx/ic/canisters/idl/" "--actor-alias" "CanDBIndex" "n674v-zqaaa-aaaap-qbpza-cai" "--actor-alias" "ic_eth" "jrzee-liaaa-aaaap-qhnoa-cai" "--package" "base" ".mops/[email protected]/src" "--package" "candb" ".mops/_github/candb#no-blob/src" "--package" "nacdb" ".mops/[email protected]/src" "--package" "stable-rbtree" ".mops/[email protected]/src" "--package" "stable-buffer" ".mops/_github/stable-buffer#v0.2.0/src" "--package" "prng" ".mops/[email protected]/src" "--package" "stableheapbtreemap" ".mops/[email protected]/src" "--package" "sha2" ".mops/[email protected]/src" "--package" "stablebuffer" ".mops/_github/stablebuffer#v0.2.0/src" "--package" "StableBuffer" ".mops/_github/StableBuffer#v0.2.0/src" "--package" "icrc1" ".mops/[email protected]/src" "--package" "array" ".mops/_github/array#main/src" "--package" "StableTrieMap" ".mops/_github/StableTrieMap#main/src" "--package" "itertools" ".mops/[email protected]/src" "--package" "base-0.7.3" ".mops/_github/base-0.7.3#aafcdee0c8328087aeed506e64aa2ff4ed329b47/src" "--package" "xtendedNumbers" ".mops/_github/xtendedNumbers#v1.1.0/src" "--package" "motoko-lib" ".mops/_github/motoko-lib#0.7/src" "--package" "map7" ".mops/_github/map7#v7.0.0/src" "--package" "sha" ".mops/_github/sha#master/src" "--package" "matchers" ".mops/_github/matchers#v1.3.0/src" "--package" "map" ".mops/[email protected]/src" "--package" "btree" ".mops/_github/btree#v0.3.3/src" "--package" "xtended-numbers" ".mops/[email protected]/src" "--package" "json.mo" ".mops/_github/json.mo#master@49612cf3865dc24bca7bb0e8b6fcf275d11a8cb2/src" "--package" "parser-combinators" ".mops/_github/parser-combinators#main/src" "--package" "nacdb-reorder" ".mops/[email protected]/src" "--package" "cycles-simple" ".mops/[email protected]/src" "--package" "stablehashmap" ".mops/[email protected]/src" "--package" "candb-multi" ".mops/[email protected]/src" "--package" "stable-hash-map" ".mops/_github/stable-hash-map#v0.2.1@eb7edf4127233f2b1f3732ede7e2c55827ac6886/src" "--package" "encoding" ".mops/_github/encoding#v0.3.2@8e0fe1d8f5c2d284e77d719703c42e0e271839b1/src" "--package" "candy" ".mops/_github/candy#v0.3.0@907a4e7363aac6c6a4e114ebc73e3d3f21e138af/src" "--package" "stablebuffer_1_3_0" ".mops/_github/stablebuffer_1_3_0#v1.3.0@acdde6bb5b939227997cebdbb8919d2e6da8691c/src" "--package" "map9" ".mops/_github/map9#v9.0.1@10b68f6ea8df5e72dfa4c07a50c8bb60a916c233/src" "--package" "join-proxy-motoko" ".mops/_github/join-proxy-motoko#master@3582456b29cb81e0e304f7df2a3935da670725f6/src" "--package" "parser-combinators.mo" ".mops/_github/parser-combinators.mo#master@f87c9417ab145017fb1f83e19ec2fc45588dbd41/src" "--package" "passport-client" ".mops/[email protected]/src" "--package" "sequence" ".mops/_github/sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "--package" "test" ".mops/[email protected]/src" "--package" "date.mo" ".mops/_github/date.mo#master@cfe51ad7d8a576d53e701136d4234233b94970eb/src" "--package" "testing" ".mops/_github/testing#main/src" "--package" "fmt" ".mops/_github/fmt#v0.1.0/src" "--package" "motoko-sequence" ".mops/_github/motoko-sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src"' failed with exit status 'exit status: 1'.
Stdout:

Stderr:
/home/porton/Projects/zondirectory/src/backend/personhood.mo:5.1-5.40: import error [M0009], file "/home/porton/Projects/zondirectory/.dfx/ic/canisters/idl/n674v-zqaaa-aaaap-qbpza-cai.did" does not exist
/home/porton/Projects/zondirectory/src/backend/personhood.mo:6.1-6.32: import error [M0009], file "/home/porton/Projects/zondirectory/.dfx/ic/canisters/idl/jrzee-liaaa-aaaap-qhnoa-cai.did" does not exist
/home/porton/Projects/zondirectory/src/backend/personhood.mo:9.1-9.56: import error [M0009], file "/home/porton/Projects/zondirectory/src/libs/configs/passport.config/lib.mo" does not exist

vporton requested review from chenyan-dfinity and a team as code owners April 14, 2024 03:21

github-actions bot added the external-contributor label Apr 14, 2024

vporton force-pushed the build-only-necessary branch from 4be4fa8 to 030f9af Compare April 16, 2024 15:58

vporton added 21 commits April 22, 2024 16:13

bug fix

3535510

removed tracing

3104ff6

bug fix

a07d998

bug fix

7033768

removed commented out code

9a70bbe

comment typo

a62f989

bug fix

bc800c1

code formatting

174eaf6

fixes

8792959

comment

88f5d98

misc

88e7b64

removed tracing

a8f18a2

started to work on automated testing

3ce3ddc

test breaking dependency compiling

c1b6831

fix bug in tests

65e79f2

trying to rectify a test

18a369d

misc

3b184ca

bug fix

108b233

misc

769f89c

more tests

cd89676

comment typo

b2c7ade

vporton added 15 commits May 25, 2024 14:14

Merge remote-tracking branch 'upstream/master' into build-only-necess…

2134dc4

…ary-dev

more error context

27c9e4d

fixed a regression (after adding more error contexts)

8e9d0e0

possible bug fix

94e656b

debugging

2cfbcf2

debugging

39ced7e

"deploy writes all environment variables to .env" test passed

33fd5b5

removed warning

89cbac6

removed an overestimated TODO comment

3b57d0c

clippy

c8fe66d

cargo fmt

fdd2ce3

removed commented out code

222654c

corrected test

568cd6b

bug fix

be01eea

Merge branch 'master' into build-only-necessary

fb9a28a

vporton requested a review from a team as a code owner May 31, 2024 13:49

vporton added 2 commits May 31, 2024 17:18

removed an apparently fixed FIXME

9338d4b

Merge branch 'build-only-necessary' of github.com:vporton/sdk into bu…

80c6f1a

…ild-only-necessary

vporton added 4 commits June 9, 2024 19:14

removed an outdated TODO comment

5731e86

removed an outdated TODO comment

1e057d0

more logging

78352ae

TODO comment

bb51187

bug fix

7299111

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: rebuild only necessary canisters #3710

feat: rebuild only necessary canisters #3710

vporton commented Apr 14, 2024 •

edited

Loading

vporton commented Apr 14, 2024

vporton commented Apr 16, 2024

vporton commented Apr 16, 2024 •

edited

Loading

vporton commented Apr 16, 2024

vporton commented Apr 16, 2024

vporton commented May 31, 2024 •

edited

Loading

vporton commented Jun 9, 2024 •

edited

Loading

vporton commented Jun 9, 2024

vporton commented Jun 9, 2024

vporton commented Jun 9, 2024

vporton commented Jun 16, 2024

vporton commented Jun 25, 2024

feat: rebuild only necessary canisters #3710

Are you sure you want to change the base?

feat: rebuild only necessary canisters #3710

Conversation

vporton commented Apr 14, 2024 • edited Loading

Description

How Has This Been Tested?

Checklist:

vporton commented Apr 14, 2024

vporton commented Apr 16, 2024

vporton commented Apr 16, 2024 • edited Loading

vporton commented Apr 16, 2024

vporton commented Apr 16, 2024

vporton commented May 31, 2024 • edited Loading

vporton commented Jun 9, 2024 • edited Loading

vporton commented Jun 9, 2024

vporton commented Jun 9, 2024

vporton commented Jun 9, 2024

vporton commented Jun 16, 2024

vporton commented Jun 25, 2024

vporton commented Apr 14, 2024 •

edited

Loading

vporton commented Apr 16, 2024 •

edited

Loading

vporton commented May 31, 2024 •

edited

Loading

vporton commented Jun 9, 2024 •

edited

Loading