Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keystone in CRIB #14326

Open
wants to merge 120 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
120 commits
Select commit Hold shift + click to select a range
48f5f91
Add node api wrapper for ergonomic cmd usage
HenryNguyen5 Sep 4, 2024
11f32e9
Add streams trigger template
HenryNguyen5 Sep 4, 2024
80523fe
Add mock external adapter for v03 mercury
HenryNguyen5 Sep 4, 2024
cd6d861
First pass of streams trigger provisioning
HenryNguyen5 Sep 4, 2024
a2a3c29
WIP: Add capabilities registry provisioner script
HenryNguyen5 Sep 7, 2024
d7b7081
Update nix flake
HenryNguyen5 Sep 13, 2024
d98f548
Fixup provisioning scripts
HenryNguyen5 Sep 13, 2024
f5ee544
Change default chainid to be 1337
HenryNguyen5 Sep 16, 2024
bb138d2
Add nil check for balances
HenryNguyen5 Sep 16, 2024
7b43199
Add ability to skip tls verification for local dev
HenryNguyen5 Sep 16, 2024
09874a5
Gently fail on not loading contracts for ocr job deletion
HenryNguyen5 Sep 16, 2024
2172e06
fixup! Change default chainid to be 1337
HenryNguyen5 Sep 16, 2024
620cb39
Formatting
HenryNguyen5 Sep 17, 2024
7627829
Change ocr file flag default
HenryNguyen5 Sep 17, 2024
b959698
Allow for multiple OCR2KB selection in key fetching
HenryNguyen5 Sep 17, 2024
32f48cf
Support on/offchain transmitter OCR3 config generation
HenryNguyen5 Sep 17, 2024
13e7396
Properly reset clientmethod on each invocation
HenryNguyen5 Sep 17, 2024
637e8ea
Add mercury contract deployment feature
HenryNguyen5 Sep 17, 2024
41f0e72
Get oracles to successfully connect to each other
HenryNguyen5 Sep 18, 2024
7a2ce4f
Keep OCR3 and OCR2 config separate
HenryNguyen5 Sep 18, 2024
e56d314
Add goreleaser setup for mock ea
HenryNguyen5 Sep 19, 2024
d369298
Add support for updating bridges
HenryNguyen5 Sep 19, 2024
9208dae
Add UpdateBridge CLI command
HenryNguyen5 Sep 19, 2024
8db8f46
Cleanup comments
HenryNguyen5 Sep 20, 2024
0540f1b
Fix typo
HenryNguyen5 Sep 20, 2024
21a10de
Add revert detection and revert reason extraction
HenryNguyen5 Sep 20, 2024
2dbd69c
Add missing env field to CR struct
HenryNguyen5 Sep 20, 2024
780f986
Fix CR deployment bugs
HenryNguyen5 Sep 20, 2024
1d274cc
Fix trigger capability typo
HenryNguyen5 Sep 20, 2024
5410d20
Add external registry and capability p2p config gen
HenryNguyen5 Sep 20, 2024
b0b182b
Add redial support for logging in
HenryNguyen5 Sep 20, 2024
9972924
Fix capability registration
HenryNguyen5 Sep 20, 2024
7dfa9d6
HACK: Add keystone workflow deployment to streams trigger cmd
HenryNguyen5 Sep 24, 2024
3295aaa
Typo
HenryNguyen5 Sep 24, 2024
ce5687f
Log ocr3 config more extensively
HenryNguyen5 Sep 24, 2024
086f0a0
Set isPublic to false for all-in-one DON
HenryNguyen5 Sep 24, 2024
c55811f
Use nodeapi for deleting ocr3 jobs
HenryNguyen5 Sep 25, 2024
487b2f0
Have mock EA return consistent prices every 10 sec
HenryNguyen5 Sep 25, 2024
d1dc8b7
Remove pluginconfig to properly enable trigger
HenryNguyen5 Sep 25, 2024
55b2dd7
Add additional logging
HenryNguyen5 Sep 25, 2024
dc1f874
Fix rebase errors
HenryNguyen5 Oct 9, 2024
502903b
Shim ksdeploy types
HenryNguyen5 Oct 9, 2024
844477e
Fix goreleaser config for mock ea
HenryNguyen5 Oct 10, 2024
14b1e74
Dont depend on cgo for mock EA
HenryNguyen5 Oct 10, 2024
f01c252
Handle aptos key creation
HenryNguyen5 Oct 10, 2024
6bd18c0
Tune mercury OCR rounds to be less freq
HenryNguyen5 Oct 12, 2024
95d69bd
Use deployments rather than pods
HenryNguyen5 Oct 12, 2024
e315b96
Overhaul node host + url handling
HenryNguyen5 Oct 17, 2024
171a953
Add dummy encryption public key to nodes
HenryNguyen5 Oct 17, 2024
20a1c7e
Add missing ctx
HenryNguyen5 Oct 17, 2024
1fd4abd
Initial multidon support
HenryNguyen5 Oct 18, 2024
987197d
Fix ingress generation for postprevision
HenryNguyen5 Oct 20, 2024
a104e3d
Fix argument ordering
HenryNguyen5 Oct 20, 2024
77ea1d0
Fix nodelist sorting and evmconfig.workflow configuration
HenryNguyen5 Oct 20, 2024
d9d25bc
Update tests
HenryNguyen5 Oct 22, 2024
88448af
Assign keystone workflows to workflow nodes
HenryNguyen5 Oct 22, 2024
7138e93
Expose capabilities on WorkflowDON
HenryNguyen5 Oct 22, 2024
7f2768e
Skip bootstrap node for keystone workflows
HenryNguyen5 Oct 22, 2024
6e6375b
Refactor nodelists + pubkeys -> nodesets
HenryNguyen5 Oct 22, 2024
57c89b0
Skip adding bootstrap nodes to capability registry
HenryNguyen5 Oct 22, 2024
9965574
Skip bootstrap node for mercury OCR config
HenryNguyen5 Oct 22, 2024
c7b9512
Formatting
HenryNguyen5 Oct 22, 2024
d77f141
Fix stale print statement
HenryNguyen5 Oct 22, 2024
7e3c78f
Bump nodeset size to minimum 5 since > 2F+1
HenryNguyen5 Oct 22, 2024
dd1d920
Use service name for DNS resolution
HenryNguyen5 Oct 22, 2024
f3f5d60
Update tests
HenryNguyen5 Oct 23, 2024
b1558bd
Fix missing / incorrect fields for keystone workflow
HenryNguyen5 Oct 23, 2024
0248f10
Update gomods
HenryNguyen5 Oct 23, 2024
499d27f
Simplify node key handling
HenryNguyen5 Oct 24, 2024
1bb5605
Formatting
HenryNguyen5 Oct 25, 2024
3d69400
Add test mocks
HenryNguyen5 Oct 29, 2024
7b7ca44
Refactor - Use single entrypoint for provisiong keystone in CRIB
HenryNguyen5 Oct 29, 2024
47f93b9
Add OCR3 caching
HenryNguyen5 Oct 29, 2024
779a060
Refactor provisioning flags and improve argument validation in keysto…
HenryNguyen5 Oct 29, 2024
b4aada0
Refactor: Remove stale references and fix refactor related bugs
HenryNguyen5 Oct 30, 2024
e63bca3
Create artefacts dir if it doesnt exist
HenryNguyen5 Oct 30, 2024
6107529
Simplify jobspec and bridge handling
HenryNguyen5 Oct 30, 2024
254d4f2
Remove unneeded token transfer calls
HenryNguyen5 Oct 30, 2024
403b135
Refactor: Cleanup logging, add more tx caching
HenryNguyen5 Oct 30, 2024
31bab6c
Fix post-rebase errors
HenryNguyen5 Oct 30, 2024
8a91cf2
Fix OCR3 digest comparison
HenryNguyen5 Oct 30, 2024
541ea0e
Remove extra cmd cruft
HenryNguyen5 Oct 30, 2024
f86f770
Remove unused func
HenryNguyen5 Oct 30, 2024
74d7e9b
Undo transmitter changes
HenryNguyen5 Oct 30, 2024
b42d0b1
Revert "Add test mocks"
HenryNguyen5 Oct 30, 2024
8e7d9d5
Fix caching
HenryNguyen5 Oct 30, 2024
6d230da
Remove deprecated assertion call
HenryNguyen5 Oct 31, 2024
a524394
Update gomod
HenryNguyen5 Oct 31, 2024
aeed963
Fix linter warns
HenryNguyen5 Nov 1, 2024
9168733
Add changeset
HenryNguyen5 Nov 1, 2024
39fa035
Add additional logging around node sets and key fetching
HenryNguyen5 Nov 1, 2024
9f251f0
Run gomodtidy
HenryNguyen5 Nov 1, 2024
82c30fa
Fix additional lints
HenryNguyen5 Nov 1, 2024
19404f1
Harden API request logic
HenryNguyen5 Nov 1, 2024
3386abc
Readme WIP
HenryNguyen5 Nov 5, 2024
c5a8411
Clean up lints, remove old readme
HenryNguyen5 Nov 8, 2024
62f9939
Increase retry interval
HenryNguyen5 Nov 12, 2024
ed1151c
Update goreleaser to 2.4.4-pro
HenryNguyen5 Nov 12, 2024
cd8b4a6
Handle non postfix path
HenryNguyen5 Nov 12, 2024
47bd57a
Resolve darwin shell hook from git root
HenryNguyen5 Nov 12, 2024
ee094d8
Merge branch 'develop' of https://github.com/smartcontractkit/chainli…
HenryNguyen5 Nov 16, 2024
da696fd
Bump streams trigger cap to 1.1.0
HenryNguyen5 Nov 17, 2024
7d875dc
Create toolkit sub-cli
HenryNguyen5 Nov 17, 2024
3deac1d
Cleanup toolkit
HenryNguyen5 Nov 18, 2024
cb46afc
Reverse URLs for nodelists
HenryNguyen5 Nov 18, 2024
987b8b5
Update snapshots
HenryNguyen5 Nov 18, 2024
0eb1a29
Remove unneeded gosec ignore
HenryNguyen5 Nov 18, 2024
f0c9fff
Merge branch 'develop' of https://github.com/smartcontractkit/chainli…
HenryNguyen5 Nov 18, 2024
8626cab
Update gomods
HenryNguyen5 Nov 18, 2024
376a354
Log when we set ocr3 config
HenryNguyen5 Nov 19, 2024
a84d5d2
Cleanup argument parsing
HenryNguyen5 Nov 19, 2024
83b9d34
Fix nodes list parsing
HenryNguyen5 Nov 19, 2024
e27c821
Fix lints + address feedback
HenryNguyen5 Nov 20, 2024
e6c4a3c
Merge branch 'develop' of https://github.com/smartcontractkit/chainli…
HenryNguyen5 Nov 20, 2024
4aea6f7
Merge branch 'develop' of https://github.com/smartcontractkit/chainli…
HenryNguyen5 Dec 12, 2024
de1a8f5
Update gomods to point to this branches pseudo version
HenryNguyen5 Dec 12, 2024
a05aab5
Update gomods to point to this branches pseudo version
HenryNguyen5 Dec 13, 2024
f4bb112
Merge branch 'develop' of https://github.com/smartcontractkit/chainli…
HenryNguyen5 Dec 13, 2024
a35f536
Merge branch 'develop' of https://github.com/smartcontractkit/chainli…
HenryNguyen5 Dec 20, 2024
2b0a459
Bump wrappers to 1.1.0
HenryNguyen5 Dec 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/loud-birds-remain.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"chainlink": minor
---

#internal Add unexposed shell cmd for updating a bridge
23 changes: 23 additions & 0 deletions core/cmd/bridge_commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,29 @@ func (s *Shell) CreateBridge(c *cli.Context) (err error) {
return s.renderAPIResponse(resp, &BridgePresenter{})
}

func (s *Shell) UpdateBridge(c *cli.Context) (err error) {
if !c.Args().Present() {
return s.errorOut(errors.New("must pass the name of the bridge to be updated"))
}
bridgeName := c.Args().First()
buf, err := getBufferFromJSON(c.Args().Get(1))
if err != nil {
return s.errorOut(err)
}

resp, err := s.HTTP.Patch(s.ctx(), "/v2/bridge_types/"+bridgeName, buf)
if err != nil {
return s.errorOut(err)
}
defer func() {
if cerr := resp.Body.Close(); cerr != nil {
err = multierr.Append(err, cerr)
}
}()

return s.renderAPIResponse(resp, &BridgePresenter{})
}

// RemoveBridge removes a specific Bridge by name.
func (s *Shell) RemoveBridge(c *cli.Context) (err error) {
if !c.Args().Present() {
Expand Down
42 changes: 42 additions & 0 deletions core/cmd/bridge_commands_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package cmd_test
import (
"bytes"
"flag"
"fmt"
"testing"
"time"

Expand Down Expand Up @@ -191,3 +192,44 @@ func TestShell_RemoveBridge(t *testing.T) {
assert.Equal(t, bt.URL.String(), p.URL)
assert.Equal(t, bt.Confirmations, p.Confirmations)
}
func TestShell_UpdateBridge(t *testing.T) {
t.Parallel()

app := startNewApplicationV2(t, nil)
client, _ := app.NewShellAndRenderer()
name := testutils.RandomizeName("updatebridge")

bt := &bridges.BridgeType{
Name: bridges.MustParseBridgeName(name),
URL: cltest.WebURL(t, "https://testing.com/bridges"),
Confirmations: 0,
}
require.NoError(t, app.BridgeORM().CreateBridgeType(testutils.Context(t), bt))
tests := []struct {
name string
args []string
errored bool
}{
{"NoArgs", []string{}, true},
{"OnlyName", []string{name}, true},
{"ValidUpdate", []string{name, fmt.Sprintf(`{ "name": "%s", "url": "http://localhost:3000/updated" }`, name)}, false},
{"InvalidJSON", []string{name, `{ "url": "http://localhost:3000/updated"`}, true},
}

for _, tt := range tests {
test := tt
t.Run(test.name, func(t *testing.T) {
set := flag.NewFlagSet("bridge", 0)
flagSetApplyFromAction(client.UpdateBridge, set, "")

require.NoError(t, set.Parse(test.args))

c := cli.NewContext(nil, set, nil)
if test.errored {
assert.Error(t, client.UpdateBridge(c))
} else {
assert.NoError(t, client.UpdateBridge(c))
}
})
}
}
16 changes: 12 additions & 4 deletions core/scripts/common/helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,12 @@ package common
import (
"context"
"crypto/ecdsa"
"crypto/tls"
"encoding/hex"
"flag"
"fmt"
"math/big"
"net/http"
"os"
"strconv"
"strings"
Expand Down Expand Up @@ -69,11 +71,17 @@ func SetupEnv(overrideNonce bool) Environment {
panic("need account key")
}

ec, err := ethclient.Dial(ethURL)
PanicErr(err)

jsonRPCClient, err := rpc.Dial(ethURL)
insecureSkipVerify := os.Getenv("INSECURE_SKIP_VERIFY") == "true"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed to add this since CRIB can run locally via kind, which uses https but a locally generated cert

tr := &http.Transport{
// User enables this at their own risk!
// #nosec G402
TLSClientConfig: &tls.Config{InsecureSkipVerify: insecureSkipVerify},
}
httpClient := &http.Client{Transport: tr}
rpcConfig := rpc.WithHTTPClient(httpClient)
jsonRPCClient, err := rpc.DialOptions(context.Background(), ethURL, rpcConfig)
PanicErr(err)
ec := ethclient.NewClient(jsonRPCClient)

chainID, err := strconv.ParseInt(chainIDEnv, 10, 64)
PanicErr(err)
Expand Down
12 changes: 6 additions & 6 deletions core/scripts/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ replace github.com/smartcontractkit/chainlink/deployment => ../../deployment
// Using a separate `require` here to avoid surrounding line changes
// creating potential merge conflicts.
require (
github.com/smartcontractkit/chainlink/deployment v0.0.0-20241206210521-125d98cdaf66
github.com/smartcontractkit/chainlink/v2 v2.0.0-20241206210521-125d98cdaf66
github.com/smartcontractkit/chainlink/deployment v0.0.0-20241212011003-de1a8f5e5b42
github.com/smartcontractkit/chainlink/v2 v2.0.0-20241212011003-de1a8f5e5b42
)

require (
Expand All @@ -34,14 +34,18 @@ require (
github.com/shopspring/decimal v1.4.0
github.com/smartcontractkit/chainlink-automation v0.8.1
github.com/smartcontractkit/chainlink-common v0.4.1-0.20241217120918-bbe318cd0760
github.com/smartcontractkit/chainlink-data-streams v0.1.1-0.20241216163550-fa030d178ba3
github.com/smartcontractkit/chainlink-testing-framework/lib v1.50.13
github.com/smartcontractkit/libocr v0.0.0-20241007185508-adbe57025f12
github.com/spf13/cobra v1.8.1
github.com/spf13/viper v1.19.0
github.com/stretchr/testify v1.9.0
github.com/umbracle/ethgo v0.1.3
github.com/umbracle/fastrlp v0.0.0-20220527094140-59d5dd30e722
github.com/urfave/cli v1.22.14
go.uber.org/zap v1.27.0
google.golang.org/protobuf v1.35.1
gopkg.in/yaml.v3 v3.0.1
k8s.io/api v0.31.1
k8s.io/apimachinery v0.31.1
k8s.io/client-go v0.31.1
Expand Down Expand Up @@ -305,13 +309,11 @@ require (
github.com/smartcontractkit/chain-selectors v1.0.34 // indirect
github.com/smartcontractkit/chainlink-ccip v0.0.0-20241218114855-f74219171000 // indirect
github.com/smartcontractkit/chainlink-cosmos v0.5.2-0.20241202195413-82468150ac1e // indirect
github.com/smartcontractkit/chainlink-data-streams v0.1.1-0.20241216163550-fa030d178ba3 // indirect
github.com/smartcontractkit/chainlink-feeds v0.1.1 // indirect
github.com/smartcontractkit/chainlink-protos/job-distributor v0.6.0 // indirect
github.com/smartcontractkit/chainlink-protos/orchestrator v0.4.0 // indirect
github.com/smartcontractkit/chainlink-solana v1.1.1-0.20241219173444-150f7443fdd3 // indirect
github.com/smartcontractkit/chainlink-starknet/relayer v0.1.1-0.20241202202529-2033490e77b8 // indirect
github.com/smartcontractkit/chainlink-testing-framework/lib v1.50.13 // indirect
github.com/smartcontractkit/grpc-proxy v0.0.0-20240830132753-a7e17fec5ab7 // indirect
github.com/smartcontractkit/tdh2/go/ocr2/decryptionplugin v0.0.0-20241009055228-33d0c0bf38de // indirect
github.com/smartcontractkit/tdh2/go/tdh2 v0.0.0-20241009055228-33d0c0bf38de // indirect
Expand Down Expand Up @@ -378,7 +380,6 @@ require (
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/ratelimit v0.3.1 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/arch v0.11.0 // indirect
golang.org/x/crypto v0.31.0 // indirect
golang.org/x/exp v0.0.0-20241009180824-f66d83c29e7c // indirect
Expand All @@ -402,7 +403,6 @@ require (
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/kube-openapi v0.0.0-20240709000822-3c01b740850f // indirect
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect
Expand Down
11 changes: 0 additions & 11 deletions core/scripts/keystone/01_deploy_contracts-sample.sh

This file was deleted.

7 changes: 0 additions & 7 deletions core/scripts/keystone/02_deploy_jobspecs-sample.sh

This file was deleted.

6 changes: 0 additions & 6 deletions core/scripts/keystone/03_gen_crib-sample.sh

This file was deleted.

3 changes: 0 additions & 3 deletions core/scripts/keystone/04_delete_ocr3_jobs-sample.sh

This file was deleted.

This file was deleted.

91 changes: 0 additions & 91 deletions core/scripts/keystone/README.md

This file was deleted.

1 change: 0 additions & 1 deletion core/scripts/keystone/artefacts/README.md

This file was deleted.

8 changes: 2 additions & 6 deletions core/scripts/keystone/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,9 @@ type command interface {

func main() {
commands := []command{
src.NewDeployContractsCommand(),
src.NewDeployJobSpecsCommand(),
src.NewGenerateCribClusterOverridesCommand(),
src.NewDeleteJobsCommand(),
src.NewProvisionKeystoneCommand(),
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reviewers: This is the main entrypoint, CRIB scripts call into this command only to provision everything. All other commands below are essentially subcmds of this one

src.NewDeployAndInitializeCapabilitiesRegistryCommand(),
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cedric-cordenier Can we remove this? I kept it around because I wasnt sure if the team needs this. NewProvisionCapabilitesRegistryCommand handles provisioning CR for crib

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we can remove this if NewProvisionCapabilitesRegistryCommand handles it.

src.NewDeployWorkflowsCommand(),
src.NewDeleteWorkflowsCommand(),
src.NewToolkit(),
}

commandsList := func(commands []command) string {
Expand Down
Loading
Loading