Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pruning logic #4194

Merged
merged 65 commits into from
Oct 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
542e612
add parallel code
jackzhhuang Aug 28, 2024
a4f0fe3
parallel execution
jackzhhuang Aug 28, 2024
0f0aa78
create chain when the parents are ready in parallel execution
jackzhhuang Aug 28, 2024
11102f9
set executing state in sender
jackzhhuang Aug 28, 2024
22b78f9
add 10000 buffer for parallel execution
jackzhhuang Aug 29, 2024
2455ef6
add log for saving time
jackzhhuang Aug 29, 2024
ca45ef7
add some test
jackzhhuang Sep 2, 2024
e11b137
add false testing case
jackzhhuang Sep 2, 2024
6ca87e3
add more testing code
jackzhhuang Sep 2, 2024
67741c3
add check data
jackzhhuang Sep 3, 2024
86f1817
add verify blue block in verifier
jackzhhuang Sep 3, 2024
c35bd07
add some code
jackzhhuang Sep 4, 2024
f8c3795
add verification
jackzhhuang Sep 4, 2024
93e71f7
fix some bugs
jackzhhuang Sep 4, 2024
df381de
add yeilding after execution for processing the main chain in other s…
jackzhhuang Sep 4, 2024
d757bd6
fmt and clippy
jackzhhuang Sep 5, 2024
fd6d2bb
rebase master
jackzhhuang Sep 5, 2024
0f70ba1
3300000 will be version 1 in vega
jackzhhuang Sep 6, 2024
a75f2f2
add pruning logic and compatible logic
jackzhhuang Jul 21, 2024
d3f8514
fix clippy
jackzhhuang Aug 6, 2024
8f1e647
fix test case
jackzhhuang Aug 6, 2024
296d04a
remove some single chain test case
jackzhhuang Aug 6, 2024
9a94908
fix clippy
jackzhhuang Aug 6, 2024
91a997b
fix flexdag's test case
jackzhhuang Sep 10, 2024
06fbd3e
fix clippy
jackzhhuang Aug 6, 2024
8049c16
add rational random to pass the deserialization verification
jackzhhuang Aug 7, 2024
d32d12f
merge dag-master
jackzhhuang Aug 7, 2024
676e861
fix bugs: blockmeta changes into latest version
jackzhhuang Aug 8, 2024
788cec3
add update db code
jackzhhuang Aug 8, 2024
7689b48
add dag db update
jackzhhuang Aug 8, 2024
e0ec991
update dag db
jackzhhuang Aug 8, 2024
75241a6
update dag db
jackzhhuang Aug 8, 2024
4a39acd
uncomment the pruning code
jackzhhuang Aug 13, 2024
43339b2
rebase sync parallel3
jackzhhuang Sep 10, 2024
07ad098
fix compiling
jackzhhuang Sep 10, 2024
f08687a
db update for dag state
jackzhhuang Sep 11, 2024
4cfc2cc
fix connection
jackzhhuang Sep 11, 2024
b987a31
add verify pruning
jackzhhuang Sep 12, 2024
d55c678
add pruning height
jackzhhuang Sep 13, 2024
954c243
no checking the pruning point if the main header still dose not have …
jackzhhuang Sep 13, 2024
e8f123f
add is ancestor of command for reachability viewing
jackzhhuang Sep 13, 2024
dd1ee2a
add command file
jackzhhuang Sep 13, 2024
109e0bd
use 850000
jackzhhuang Sep 13, 2024
cbc7c55
add rpc json new command
jackzhhuang Sep 13, 2024
153c30d
add some log for debug
jackzhhuang Sep 14, 2024
165690d
if this is the first pruning point, the previous one will be genesis
jackzhhuang Sep 18, 2024
61414cd
save the dag state using the pruning point as the key and if it is 0,…
jackzhhuang Sep 18, 2024
ec341bc
merge dag master
jackzhhuang Sep 18, 2024
7d24357
get the tips by genesis id if the pruning point is 0
jackzhhuang Sep 18, 2024
d5f2762
fix test case as prievious version did
jackzhhuang Sep 18, 2024
a70db10
use genesis id to get the ghost data
jackzhhuang Sep 18, 2024
fd96b21
add genesis in registry in test_miner_service
jackzhhuang Sep 18, 2024
38232f7
use 1000 as parallel buffer
jackzhhuang Sep 19, 2024
f603fc3
remove some logs
jackzhhuang Sep 19, 2024
6e6032d
add pruning arguments in pruning methods to custom the network config
jackzhhuang Sep 20, 2024
41d5def
add test case for pruning calculation and pruning
jackzhhuang Sep 23, 2024
f5807c8
add write lock when saving the tips
jackzhhuang Sep 24, 2024
2be8656
add test pruning for chain
jackzhhuang Sep 24, 2024
b7d157d
add test case
jackzhhuang Sep 24, 2024
2d3745e
if no fork, execute at once
jackzhhuang Sep 25, 2024
07de3ce
add test code
jackzhhuang Sep 27, 2024
998637f
add some rpc param in yaml's cmd
jackzhhuang Sep 27, 2024
ccffd25
use macro to define the parallel count
jackzhhuang Sep 29, 2024
ba5d0b2
use 4090000 as pruning beginning
jackzhhuang Sep 30, 2024
218e7a0
fix the code for robot's comments
jackzhhuang Sep 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion chain/api/src/chain.rs
Original file line number Diff line number Diff line change
Expand Up @@ -106,14 +106,16 @@ pub trait ChainReader {
access_path: Option<AccessPath>,
) -> Result<Option<TransactionInfoWithProof>>;

fn current_tips_hash(&self) -> Result<Vec<HashValue>>;
fn current_tips_hash(&self, pruning_point: HashValue) -> Result<Vec<HashValue>>;
fn has_dag_block(&self, header_id: HashValue) -> Result<bool>;
fn check_chain_type(&self) -> Result<ChainType>;
fn verify_and_ghostdata(
&self,
uncles: &[BlockHeader],
header: &BlockHeader,
) -> Result<GhostdagData>;
fn is_dag_ancestor_of(&self, ancestor: HashValue, descendants: Vec<HashValue>) -> Result<bool>;
fn get_pruning_height(&self) -> BlockNumber;
}

pub trait ChainWriter {
Expand Down
7 changes: 6 additions & 1 deletion chain/api/src/message.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
use crate::{ChainType, TransactionInfoWithProof};
use anyhow::Result;
use starcoin_crypto::HashValue;
use starcoin_dag::consensusdb::consenses_state::DagStateView;
use starcoin_dag::consensusdb::consenses_state::{DagStateView, ReachabilityView};
use starcoin_dag::types::ghostdata::GhostdagData;
use starcoin_service_registry::ServiceRequest;
use starcoin_types::transaction::RichTransactionInfo;
Expand Down Expand Up @@ -68,6 +68,10 @@ pub enum ChainRequest {
GetDagStateView,
CheckChainType,
GetGhostdagData(HashValue),
IsAncestorOfCommand {
ancestor: HashValue,
descendants: Vec<HashValue>,
},
}

impl ServiceRequest for ChainRequest {
Expand Down Expand Up @@ -99,4 +103,5 @@ pub enum ChainResponse {
DagStateView(Box<DagStateView>),
CheckChainType(ChainType),
GhostdagDataOption(Box<Option<GhostdagData>>),
IsAncestorOfCommand { reachability_view: ReachabilityView },
}
24 changes: 23 additions & 1 deletion chain/api/src/service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use crate::message::{ChainRequest, ChainResponse};
use crate::{ChainType, TransactionInfoWithProof};
use anyhow::{bail, Result};
use starcoin_crypto::HashValue;
use starcoin_dag::consensusdb::consenses_state::DagStateView;
use starcoin_dag::consensusdb::consenses_state::{DagStateView, ReachabilityView};
use starcoin_dag::types::ghostdata::GhostdagData;
use starcoin_service_registry::{ActorService, ServiceHandler, ServiceRef};
use starcoin_types::contract_event::{ContractEvent, ContractEventInfo};
Expand Down Expand Up @@ -149,6 +149,11 @@ pub trait ChainAsyncService:
async fn get_dag_state(&self) -> Result<DagStateView>;
async fn check_chain_type(&self) -> Result<ChainType>;
async fn get_ghostdagdata(&self, id: HashValue) -> Result<Option<GhostdagData>>;
async fn is_ancestor_of(
&self,
ancestor: HashValue,
descendants: Vec<HashValue>,
) -> Result<starcoin_dag::consensusdb::consenses_state::ReachabilityView>;
}

#[async_trait::async_trait]
Expand Down Expand Up @@ -486,4 +491,21 @@ where
bail!("failed to get ghostdag data")
}
}
async fn is_ancestor_of(
&self,
ancestor: HashValue,
descendants: Vec<HashValue>,
) -> Result<ReachabilityView> {
let response = self
.send(ChainRequest::IsAncestorOfCommand {
ancestor,
descendants,
})
.await??;
if let ChainResponse::IsAncestorOfCommand { reachability_view } = response {
Ok(reachability_view)
} else {
bail!("failed to get ghostdag data")
}
}
}
111 changes: 109 additions & 2 deletions chain/mock/src/mock_chain.rs
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
// Copyright (c) The Starcoin Core Contributors
// SPDX-License-Identifier: Apache-2.0

use anyhow::Result;
use anyhow::{format_err, Result};
use starcoin_account_api::AccountInfo;
use starcoin_chain::{BlockChain, ChainReader, ChainWriter};
use starcoin_config::ChainNetwork;
use starcoin_consensus::Consensus;
use starcoin_crypto::HashValue;
use starcoin_dag::blockdag::BlockDAG;
use starcoin_dag::blockdag::{BlockDAG, MineNewDagBlockInfo};
use starcoin_genesis::Genesis;
use starcoin_logger::prelude::*;
use starcoin_storage::Storage;
use starcoin_types::block::{Block, BlockHeader};
use starcoin_types::blockhash::KType;
use starcoin_types::startup_info::ChainInfo;
use std::sync::Arc;
use std::vec;
Expand Down Expand Up @@ -39,6 +40,26 @@ impl MockChain {
Ok(Self::new_inner(net, chain, miner, storage))
}

pub fn new_with_params(
net: ChainNetwork,
k: KType,
pruning_depth: u64,
pruning_finality: u64,
) -> Result<Self> {
let (storage, chain_info, _, dag) =
Genesis::init_storage_for_test_with_param(&net, k, pruning_depth, pruning_finality)?;

let chain = BlockChain::new(
net.time_service(),
chain_info.head().id(),
storage.clone(),
None,
dag,
)?;
let miner = AccountInfo::random();
Ok(Self::new_inner(net, chain, miner, storage))
}

pub fn new_with_storage(
net: ChainNetwork,
storage: Arc<Storage>,
Expand Down Expand Up @@ -178,6 +199,26 @@ impl MockChain {
.create_block(template, self.net.time_service().as_ref())
}

pub fn produce_block_by_params(
&mut self,
parent_header: BlockHeader,
tips: Vec<HashValue>,
pruning_point: HashValue,
) -> Result<Block> {
let (template, _) = self.head.create_block_template_by_header(
*self.miner.address(),
parent_header,
vec![],
vec![],
None,
tips,
pruning_point,
)?;
self.head
.consensus()
.create_block(template, self.net.time_service().as_ref())
}

pub fn produce_block_by_tips(
&mut self,
parent_header: BlockHeader,
Expand All @@ -197,6 +238,72 @@ impl MockChain {
.create_block(template, self.net.time_service().as_ref())
}

pub fn produce_block_for_pruning(&mut self) -> Result<Block> {
let tips = self.head.get_dag_state()?.tips;
let ghostdata = self.head.dag().ghost_dag_manager().ghostdag(&tips)?;
let selected_header = self
.head()
.get_storage()
.get_block_header_by_hash(ghostdata.selected_parent)?
.ok_or_else(|| {
format_err!(
"Cannot find block header by hash: {:?}",
ghostdata.selected_parent
)
})?;

let previous_pruning = if selected_header.pruning_point() == HashValue::zero() {
self.head().get_storage().get_genesis()?.unwrap()
} else {
selected_header.pruning_point()
};

let prevous_ghostdata = self
.head()
.dag()
.ghostdata_by_hash(previous_pruning)?
.ok_or_else(|| format_err!("Cannot find ghostdata by hash: {:?}", previous_pruning))?;

let MineNewDagBlockInfo {
tips: pruned_tips,
blue_blocks,
pruning_point,
} = self
.head
.dag()
.calc_mergeset_and_tips(previous_pruning, prevous_ghostdata.as_ref())?;

debug!(
"tips: {:?}, blue_blocks: {:?}, pruning_point: {:?}",
pruned_tips, blue_blocks, pruning_point
);

let (template, _) = self.head.create_block_template_by_header(
*self.miner.address(),
selected_header,
vec![],
blue_blocks
.get(1..)
.unwrap_or(&[])
.iter()
.map(|block_id| {
self.head()
.get_storage()
.get_block_header_by_hash(*block_id)?
.ok_or_else(|| {
format_err!("Block header not found for hash: {:?}", block_id)
})
})
.collect::<Result<Vec<_>>>()?,
None,
pruned_tips,
pruning_point,
)?;
self.head
.consensus()
.create_block(template, self.net.time_service().as_ref())
}

pub fn apply(&mut self, block: Block) -> Result<()> {
self.head.apply(block)?;
Ok(())
Expand Down
6 changes: 6 additions & 0 deletions chain/service/src/chain_service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,12 @@ impl ServiceHandler<Self, ChainRequest> for ChainReaderService {
ChainRequest::GetGhostdagData(id) => Ok(ChainResponse::GhostdagDataOption(Box::new(
self.inner.get_ghostdagdata(id)?,
))),
ChainRequest::IsAncestorOfCommand {
ancestor,
descendants,
} => Ok(ChainResponse::IsAncestorOfCommand {
reachability_view: self.inner.dag.is_ancestor_of(ancestor, descendants)?,
}),
}
}
}
Expand Down
Loading
Loading