Question/issue: Querying data older than last known sync committee

[EzequielPostan]

please let me know if I should move this to the same thread in #388, I created a separate one because the topic looks wider

As a follow up of #388, I mentioned in that issue:

I would like to use Helios to validate historical data, which means that I am interested in running Helios from the oldest checkpoint possible. Moreover, as time goes by, and more checkpoints are created, I will still want to start from that oldest possible checkpoint.

In #388 we discussed how to sync from checkpoints older than 128 committees old, and now I have a follow up question/issue.
After testing, looking at the logs and exploring the code, I get the following understanding:

• Helios bootstraps from a checkpoint which today can be 128 sync committees old. With the change we discussed Helios fails to sync (invalid sync committee period) #388, it could ask for all the sync committees needed and not limit itself to just 128, allowing for longer bootstrap period.
• Now, I see that in this bootstrap period, Helios is asking for checkpoint headers only, until it catches up with the expected period based on wall clock (and assumes that wall clock is within the checkpoint + 128 committees). Then, it transitions towards asking new headers from optimistic and finalized updates calls (I am still trying to understand the specs on these calls).

When I sent this request to helios

$ curl localhost:8545 --request POST --header "Content-Type: application/json" --data '{ "jsonrpc": "2.0", "id": "1", "method": "eth_getBlockByNumber", "params": ["0x9129e7", false] }' | jq 

I received

{
  "jsonrpc": "2.0",
  "result": null,
  "id": "1"
}

which didn't retrieve the block, while I was expecting Helios to be able to do so (because it is a block at a time after my initial checkpoint)

After exploring the code, I noticed that Helios asks for the blocks to its internal state. But this state is indeed not updated from the initial syncing updates. The state seems to be populated only by updates from the most recent known sync committee (which I understand are the optimistic and finalized headers received)

My questions are:

• Is my understanding correct?
• Does this mean that Helios does not support querying historical data before first optimistic/finalized header received? Which I understand implies that if I re-start Helios, I would lose data (the older headers) even if I can sync from the same checkpoint (because the last sync committee can change while Helios is down)
• If my understanding is correct, is this feature in the scope of Helios? E.g. as a "verbose" mode? Would it be too hard to implement? I imagine it would demand more resources for syncing time (as I am trying to download all block headers from checkpoint)
• Also, just to confirm, are all headers downloaded after sync? or only a subset?

Once again, thanks a lot in advance for the responses

[ncitron]

So only a one header per sync committee period (27 hours) is downloaded while doing the initial sync, and then all headers are downloaded. To download and maintain all of these headers would be much too expensive for a light client to do, nor do I feel it would be in scope to add.

You can however do what you want to do by using Helios as a library. Simply ask Helios for the latest/finalized block header. Then grab the parent_hash from that block, and use an untrusted RPC to fetch the parent block, and use something similar to this to check that the block actually hashes to the correct value. You can recursively do this and go back as far as you want without any additional trust assumptions.