Optimize the node store writing perf

[jianoaix]

Why are these changes needed?

About half of the node store writing time was spent on unnecessary serialization of chunks.

See the "chunk serialization duration" v.s. "write batch duration", which are the two dominant components of db writing and are roughly the same. With this PR, the "chunk serialization duration" will become negligible.

Checks

• I've made sure the lint is passing in this PR.
• I've made sure the tests are passing. Note that there might be a few flaky tests, in that case, please comment that they are not relevant.
• Testing Strategy
  • Unit tests
  • Integration tests
  • This PR is not tested :(

[pschork]

Nice optimization @jianoaix

Wrt Fiona, do you think that their problematic node had degraded IO that was amplified with redundant serialization?

Also, I feel like some of your profiling code that was removed in a65c410 (ie encode duration, serialization duration) could be useful as prometheus gauge in the operator's grafana dashboard for future debugging needs.

[jianoaix]

Nice optimization @jianoaix

Wrt Fiona, do you think that their problematic node had degraded IO that was amplified with redundant serialization?

Also, I feel like some of your profiling code that was removed in a65c410 (ie encode duration, serialization duration) could be useful as prometheus gauge in the operator's grafana dashboard for future debugging needs.

It is possible, but not certain. I think we could run another test after v0.7.0 (I'm waiting for the reachability check as an additional tool for this).
The metric was added separately so they were removed here to avoid conflict.