working fetcher CLI #60
Conversation
| * For every contract id, store the ledger height, | ||
| * up to which the transactions were fetched. | ||
| */ | ||
| heights: OrderedMap<string, number> |
There was a problem hiding this comment.
Should we use bigint as the value?
There was a problem hiding this comment.
Yeah, I also thought about it, but Stellar SDK is using number so we are compatible with them: https://stellar.github.io/js-stellar-sdk/OperationCallBuilder.html#forLedger
solarkraft/src/fetcher/storage.ts
Outdated
| function getFetcherStateFilename(root: string) { | ||
| mkdirSync(root, { recursive: true }) |
There was a problem hiding this comment.
Not really obvious that this has side-effects from the function name.
Can we just call mkdirSync in saveFetcherState instead of here?
solarkraft/src/fetch.ts
Outdated
| if (args.height < 0) { | ||
| // how to fetch the latest height? | ||
| console.log('not implemented yet, starting with 0') | ||
| lastHeight = 0 |
There was a problem hiding this comment.
Shouldn't this also start from the cached height?
solarkraft/src/fetch.ts
Outdated
| // load and save the state, other fetchers may work concurrently | ||
| fetcherState = loadFetcherState(args.home) | ||
| fetcherState = { ...fetcherState, heights: fetcherState.heights.set(contractId, lastHeight) } | ||
| saveFetcherState(args.home, fetcherState) |
There was a problem hiding this comment.
Don't you get a filesystem race here with concurrent fetchers? 😄
And actually, will the streaming callbacks be processed sequentially?
There was a problem hiding this comment.
yeah, there is a possibility into missing filesystem updates for concurrently running fetchers. Also, for the callbacks in theory.
solarkraft/src/fetch.ts
Outdated
| if (nEvents % HEIGHT_FETCHING_PERIOD === 0) { | ||
| // Fetch the height of the current message. | ||
| // Note that messages may come slightly out of order, so the heights are not precise. |
There was a problem hiding this comment.
Not entirely sure what this block does.
afaict, every 100 messages, it will actually fetch the tx blockheight, and updated the persisted state? May be worth a comment.
There was a problem hiding this comment.
good catch, I will add a comment
Co-authored-by: Thomas Pani <[email protected]>
This PR adds relatively small changes on top of #55. The fetcher is fetching the transactions and saving them in the storage. For example:
$ solarkraft fetch --rpc https://horizon-testnet.stellar.org --id CC22QGTOUMERDNIYN7TPNX3V6EMPHQXVSRR3XY56EADF7YTFISD2ROND --height 1638368 Target contract: CC22QGTOUMERDNIYN7TPNX3V6EMPHQXVSRR3XY56EADF7YTFISD2ROND... Last cached height: 1638598 Fetching fresh transactions from: https://horizon-testnet.stellar.org... Fetching the ledger for 1638368 Fetching transactions indefinitely. Close with Ctrl-C. + save: 1638369 + save: 1638371 + save: 1638370 + save: 1638372 + save: 1638373 + save: 1638374 + save: 1638375 + save: 1638376 + save: 1638378 + save: 1638377 + save: 1638379 + save: 1638380 + save: 1638381 + save: 1638382 + save: 1638383 = at: 1638412 = at: 1638456I will add an integration test for the above later today.