Improve trace production

[sideninja]

Right now trace production is being done on-demand so it can server different possible configurations of the tracers.

We should work on producing most common traces up front and serving them from a local index instead of on-demand. That would significantly lower the load on the node producing the traces on-demand.

[j1010001]

No longer relevant, we are moving back to dry-run (onflow/flow-go#6539) to implement better solution for providing traces.