Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP for dynamic coordinator selector #333

Merged
merged 94 commits into from
Nov 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
71e6af4
WIP for dynamic coordinator selector
hosie Oct 24, 2024
26218b5
separate thread for assemble coordination
hosie Oct 25, 2024
7654556
multiple iterations
hosie Oct 25, 2024
8e0429e
build tags
hosie Oct 25, 2024
8d72b3e
Merge branch 'domain-conf' into dynamic-coorinator-selection
hosie Oct 25, 2024
73215f5
post merge consolidation
hosie Oct 26, 2024
25df7f4
domain name
hosie Oct 27, 2024
1666684
logs
hosie Oct 27, 2024
2bab9c9
assert consistent view of the world state
hosie Oct 27, 2024
e7673ff
pendingEndorsementRequest struct
hosie Oct 27, 2024
3cf78b9
idempotent endorsement requests
hosie Oct 27, 2024
0561a5c
endorsement not included in revert response
hosie Oct 27, 2024
464ef19
callback to original assembling node
hosie Oct 31, 2024
5181b99
retry assemble errors
hosie Oct 31, 2024
22397c5
component tests for concurrent transactions
hosie Oct 31, 2024
f35bcaa
try component tests on postgres
hosie Nov 2, 2024
ceb2076
Merge branch 'main' into dynamic-coordinator-selection-merge
hosie Nov 2, 2024
fb36a6a
post merge fixup
hosie Nov 2, 2024
454d366
couple sign to assemble
hosie Nov 3, 2024
bcc3074
refactor
hosie Nov 3, 2024
5330614
check error
hosie Nov 3, 2024
6e6a91b
Merge branch 'main' into dynamic-coordinator-selection-merge
hosie Nov 4, 2024
28c3d6e
Merge branch 'main' into dynamic-coordinator-selection-merge
hosie Nov 4, 2024
836dc5e
TestPrivacyGroupEndorsementConcurrent
hosie Nov 8, 2024
82acdd9
logging
hosie Nov 8, 2024
0c161a7
unassigned var
hosie Nov 8, 2024
05d7d9c
unused argument
hosie Nov 8, 2024
04c2be2
comment typo
hosie Nov 8, 2024
28281a7
delete component_test_long
hosie Nov 8, 2024
e6360be
Merge branch 'main' into dynamic-coordinator-selection-merge
hosie Nov 8, 2024
55968cd
status for prepared
hosie Nov 9, 2024
5714358
test refactor
hosie Nov 9, 2024
ee4e891
skip uncessary code branch
hosie Nov 9, 2024
015d138
lint
hosie Nov 9, 2024
cfbf8f3
replace fakes with mocks
hosie Nov 9, 2024
a6de3ad
Merge branch 'main' into dynamic-coordinator-selection-merge
hosie Nov 9, 2024
f696665
merge branch 'main' into dynamic-coorinator-selection
hosie Nov 9, 2024
3463ef3
Merge branch 'dynamic-coordinator-selection-merge' into dynamic-coori…
hosie Nov 9, 2024
3dffd8b
test refactor
hosie Nov 9, 2024
8e2f727
retry delegate request
hosie Nov 9, 2024
d3c5b2c
remove panic
hosie Nov 9, 2024
e9da995
PostgreSQL DB migrations to move from signer_nonce to pub_txn_id
peterbroadhurst Nov 10, 2024
0d1f556
logging
hosie Nov 10, 2024
a3ee9ce
test dynamic coordinator
hosie Nov 10, 2024
1bf6031
ignore mocks in coverage
hosie Nov 10, 2024
753fa96
remove dead code
hosie Nov 10, 2024
939b1af
delegation acknowledgment
hosie Nov 10, 2024
9ed1d27
SQLite migration to create new DB (no data migration)
peterbroadhurst Nov 11, 2024
1d6a764
common utilities for mocking
hosie Nov 11, 2024
3110640
Update logic for new approach to nonce allocation
peterbroadhurst Nov 11, 2024
c887ab2
logging
hosie Nov 11, 2024
4f7b587
better status
hosie Nov 14, 2024
3666ea4
merge branch 'main' into dynamic-coorinator-selection
hosie Nov 14, 2024
ba5fd8b
parametrize concurrency test
hosie Nov 14, 2024
1995aec
RangeBoundary test
hosie Nov 14, 2024
a1c7d4e
Work through changes for new nonce allocation system
peterbroadhurst Nov 15, 2024
ac8aa8a
Working through DB interactions
peterbroadhurst Nov 15, 2024
51aeb5e
Work through UPDATE semantics with multiple rows in one update
peterbroadhurst Nov 16, 2024
8e73aa4
Work through tests in public tx mgr
peterbroadhurst Nov 16, 2024
6233ce4
Update and simplify DB TX model with public TX mgr changes
peterbroadhurst Nov 17, 2024
7879de9
Work through private TX manager changes for late nonce alloc
peterbroadhurst Nov 17, 2024
22330db
Rollback change in #403 that causes ethclient test to take 30s
peterbroadhurst Nov 17, 2024
a1d1f7f
Add configurable startup retry for blockchain node connection
peterbroadhurst Nov 17, 2024
5d3bba7
remove unused function
hosie Nov 17, 2024
4e3c792
Merge branch 'main' into dynamic-coorinator-selection
hosie Nov 17, 2024
e2b084a
Unit test complete for PSQL and SQLite with late nonce assign
peterbroadhurst Nov 17, 2024
fd71386
Add post-commit notification to public Tx manager to dispatch
peterbroadhurst Nov 17, 2024
0204192
Propagate DB TX to contract lookup
peterbroadhurst Nov 17, 2024
983ad9c
Work through last tweaks for component test build pass
peterbroadhurst Nov 17, 2024
bcda725
Work through last issues with E2E tests
peterbroadhurst Nov 17, 2024
751443a
Merge branch 'main' of https://github.com/kaleido-io/paladin into pub…
peterbroadhurst Nov 17, 2024
dd6c859
Finalize comment
peterbroadhurst Nov 17, 2024
963f7a3
Re-enabled commented out tests
peterbroadhurst Nov 17, 2024
1374804
Start debug server on localhost:6060 by default
peterbroadhurst Nov 18, 2024
df8860f
Support a HTTP request to get a Java dump too
peterbroadhurst Nov 18, 2024
ca47eb2
Synchronize calls to onNext per requrements in gRPC docs
peterbroadhurst Nov 18, 2024
4421ff2
Move to enabling debug server in operator, not config
peterbroadhurst Nov 19, 2024
30df922
Add test to do parallel submission, and ensure we do not attempt gas …
peterbroadhurst Nov 19, 2024
56235d9
Enough gas for Zeto transactions
peterbroadhurst Nov 19, 2024
f6556d5
New parallel test passing
peterbroadhurst Nov 19, 2024
d4b0ede
Update operator/test/e2e/e2e_noto_pente_test.go
hosie Nov 19, 2024
64df95d
Clean up defunct field
peterbroadhurst Nov 19, 2024
98c95ab
Merge branch 'hardening' of https://github.com/kaleido-io/paladin int…
peterbroadhurst Nov 19, 2024
89a2d1e
Merge pull request #443 from LF-Decentralized-Trust-labs/hardening
peterbroadhurst Nov 19, 2024
43a6668
Merge branch 'dynamic-coorinator-selection' of https://github.com/kal…
peterbroadhurst Nov 19, 2024
21c93ee
Update tests for queued submission
peterbroadhurst Nov 19, 2024
a6a31e4
Allow full TX pool usage by a single address ()
peterbroadhurst Nov 19, 2024
fd6c572
Allow re-delegation based on blockheight
peterbroadhurst Nov 19, 2024
69511f8
Move to hash on endorsement set for coordinator election ahead of nex…
peterbroadhurst Nov 20, 2024
5a23657
Re-enable the others tests
peterbroadhurst Nov 20, 2024
686e021
Merge pull request #441 from LF-Decentralized-Trust-labs/pubtx-submit…
hosie Nov 20, 2024
fbd7d71
Merge remote-tracking branch 'origin/main' into HEAD
awrichar Nov 20, 2024
7a795a4
Update on-conflict processing for prepared states
peterbroadhurst Nov 20, 2024
b77767e
Merge branch 'dynamic-coorinator-selection' of https://github.com/kal…
peterbroadhurst Nov 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions buildSrc/src/main/groovy/Mockery.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,11 @@ class Mockery extends DefaultTask {
@Optional
String outputPackage

@Input
boolean inpackage = false

@OutputDirectory
@Optional
File outputDir

void inputDir(Object dir) {
Expand All @@ -82,6 +86,10 @@ class Mockery extends DefaultTask {
outputDir = project.file(output)
}

void inpackage(boolean inPackage) {
inpackage = inPackage
}

protected void configure(ExecSpec spec) {
if (inputDir != null) {
spec.args '--dir', inputDir
Expand All @@ -98,6 +106,9 @@ class Mockery extends DefaultTask {
if (outputDir != null) {
spec.args '--output', outputDir
}
if (inpackage) {
spec.args '--inpackage'
}
}

}
Expand Down
2 changes: 2 additions & 0 deletions config/pkg/pldconf/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,12 @@ type PaladinConfig struct {
TransportManagerConfig `json:",inline"`
RegistryManagerConfig `json:",inline"`
KeyManagerConfig `json:",inline"`
Startup StartupConfig `json:"startup"`
Log LogConfig `json:"log"`
Blockchain EthClientConfig `json:"blockchain"`
DB DBConfig `json:"db"`
RPCServer RPCServerConfig `json:"rpcServer"`
DebugServer DebugServerConfig `json:"debugServer"`
StateStore StateStoreConfig `json:"statestore"`
BlockIndexer BlockIndexerConfig `json:"blockIndexer"`
TempDir *string `json:"tempDir"`
Expand Down
4 changes: 4 additions & 0 deletions config/pkg/pldconf/domainmgr.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ package pldconf

import (
"github.com/kaleido-io/paladin/config/pkg/confutil"
"github.com/kaleido-io/paladin/toolkit/pkg/tktypes"
)

// Intended to be embedded at root level of paladin config
Expand All @@ -35,8 +36,11 @@ type DomainConfig struct {
Config map[string]any `json:"config"`
RegistryAddress string `json:"registryAddress"`
AllowSigning bool `json:"allowSigning"`
DefaultGasLimit *uint64 `json:"defaultGasLimit"`
}

var DefaultDefaultGasLimit tktypes.HexUint64 = 4000000 // high gas limit by default (accommodating zkp transactions)

var ContractCacheDefaults = &CacheConfig{
Capacity: confutil.P(1000),
}
Expand Down
9 changes: 9 additions & 0 deletions config/pkg/pldconf/httpserver.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,12 @@ type StaticServerConfig struct {
StaticPath string `json:"staticPath"` // Path to the static files in the server FS e.g /app/ui
URLPath string `json:"urlPath"` // URL path to serve the static files e.g /ui -> http://host:port/ui
}

type DebugServerConfig struct {
Enabled *bool `json:"enabled"`
HTTPServerConfig
}

var DebugServerDefaults = &DebugServerConfig{
Enabled: confutil.P(false),
}
28 changes: 17 additions & 11 deletions config/pkg/pldconf/privatetxmgr.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,19 +37,25 @@ var DistributerWriterConfigDefaults = FlushWriterConfig{

var PrivateTxManagerDefaults = &PrivateTxManagerConfig{
Sequencer: PrivateTxManagerSequencerConfig{
MaxConcurrentProcess: confutil.P(500),
EvaluationInterval: confutil.P("5m"),
PersistenceRetryTimeout: confutil.P("5s"),
StaleTimeout: confutil.P("10m"),
MaxPendingEvents: confutil.P(500),
MaxConcurrentProcess: confutil.P(500),
MaxInflightTransactions: confutil.P(500),
EvaluationInterval: confutil.P("5m"),
PersistenceRetryTimeout: confutil.P("5s"),
StaleTimeout: confutil.P("10m"),
MaxPendingEvents: confutil.P(500),
RoundRobinCoordinatorBlockRangeSize: confutil.P(100),
AssembleRequestTimeout: confutil.P("1s"),
},
RequestTimeout: confutil.P("15s"),
RequestTimeout: confutil.P("1s"),
}

type PrivateTxManagerSequencerConfig struct {
MaxConcurrentProcess *int `json:"maxConcurrentProcess,omitempty"`
MaxPendingEvents *int `json:"maxPendingEvents,omitempty"`
EvaluationInterval *string `json:"evalInterval,omitempty"`
PersistenceRetryTimeout *string `json:"persistenceRetryTimeout,omitempty"`
StaleTimeout *string `json:"staleTimeout,omitempty"`
MaxConcurrentProcess *int `json:"maxConcurrentProcess,omitempty"`
MaxInflightTransactions *int `json:"maxInflightTransactions,omitempty"`
MaxPendingEvents *int `json:"maxPendingEvents,omitempty"`
EvaluationInterval *string `json:"evalInterval,omitempty"`
PersistenceRetryTimeout *string `json:"persistenceRetryTimeout,omitempty"`
StaleTimeout *string `json:"staleTimeout,omitempty"`
RoundRobinCoordinatorBlockRangeSize *int `json:"roundRobinCoordinatorBlockRangeSize,omitempty"`
AssembleRequestTimeout *string `json:"assembleRequestTimeout,omitempty"`
}
36 changes: 36 additions & 0 deletions config/pkg/pldconf/startup.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// Copyright © 2024 Kaleido, Inc.
//
// SPDX-License-Identifier: Apache-2.0
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package pldconf

import (
"github.com/kaleido-io/paladin/config/pkg/confutil"
)

type StartupConfig struct {
BlockchainConnectRetry RetryConfigWithMax `json:"blockchainConnectRetry"`
}

var StartupConfigDefaults = StartupConfig{
BlockchainConnectRetry: RetryConfigWithMax{
RetryConfig: RetryConfig{
InitialDelay: confutil.P("500ms"),
MaxDelay: confutil.P("2s"),
Factor: confutil.P(2.0),
},
MaxAttempts: confutil.P(10),
},
}
14 changes: 7 additions & 7 deletions core/go/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,10 @@ ext {
include "mocks/**/*.go"
}

targetCoverage = 93
targetCoverage = 92.5
maxCoverageBarGap = 1
coverageExcludedPackages = [
'github.com/kaleido-io/paladin/core/internal/privatetxnmgr/ptmgrtypes/mock_transaction_flow.go',
'github.com/kaleido-io/paladin/core/pkg/proto',
'github.com/kaleido-io/paladin/core/pkg/proto/transaction',
'github.com/kaleido-io/paladin/toolkit/prototk',
Expand Down Expand Up @@ -211,12 +212,6 @@ task makeMocks(type: Mockery, dependsOn: [":installMockery", protoc, goGet]) {
outputPackage 'componentmocks'
outputDir 'mocks/componentmocks'
}
mock {
inputDir 'internal/components'
name 'PublicTxBatch'
outputPackage 'componentmocks'
outputDir 'mocks/componentmocks'
}
mock {
inputDir "go list -f {{.Dir}} github.com/kaleido-io/paladin/toolkit/pkg/rpcserver".execute().text.trim()
includeAll true
Expand Down Expand Up @@ -247,6 +242,11 @@ task makeMocks(type: Mockery, dependsOn: [":installMockery", protoc, goGet]) {
outputPackage 'preparedtxdistributionmocks'
outputDir 'mocks/preparedtxdistributionmocks'
}
mock {
inputDir 'internal/privatetxnmgr/ptmgrtypes'
name "TransactionFlow"
inpackage true
}
}

task clean(type: Delete, dependsOn: ':testinfra:stopTestInfra') {
Expand Down
Loading