VIP-3: Rust Server Read Path

[FelixGV]

Also tweaked the VIP template, main VIP landing page and VIP-1 in order to improve the navigation.

Added guidance on how to embed images in the doc_guide.

The rendered page for this VIP can be seen here: https://felixgv.github.io/venice/docs/proposals/vip-3

[ZacAttack]

So, I won't belabor the rust vs. C++ conversation, I don't think I'm really qualified to die on either hill.

But I do wonder, Rocksdb is written in C++. Today, the Venice team is far more java oriented. If we go for a native read path, we're going to have to build a competency in some non managed language in the team. It strikes me that if we build that competency in the same language that rocksdb is implemented, then we'd gain more introspection within the team in that layer. Admittedly, the situations where we're digging through the guts of rocksdb are few and far in between, but maybe that would start to change?

I kind of liken this to our Kafka clients. We're not doing big changes in the clients, but by being able to read the code and understanding how the clients/Kafka works, we've been able to make design decisions more informed based on that.

[FelixGV]

So, I won't belabor the rust vs. C++ conversation, I don't think I'm really qualified to die on either hill.

But I do wonder, Rocksdb is written in C++. Today, the Venice team is far more java oriented. If we go for a native read path, we're going to have to build a competency in some non managed language in the team. It strikes me that if we build that competency in the same language that rocksdb is implemented, then we'd gain more introspection within the team in that layer. Admittedly, the situations where we're digging through the guts of rocksdb are few and far in between, but maybe that would start to change?

I kind of liken this to our Kafka clients. We're not doing big changes in the clients, but by being able to read the code and understanding how the clients/Kafka works, we've been able to make design decisions more informed based on that.

Hmm... it is true that if the team gains more hands-on experience with C++, we could better understand the RocksDB codebase than today. But I am not sure that this incremental benefit (meaning: we're not starting from zero understanding today) is sufficiently important to warrant the downsides. The downsides being that the learning curve and productivity of C++ are generally considered to be worse than Rust's.

[gaojieliu]

Hmm... it is true that if the team gains more hands-on experience with C++, we could better understand the RocksDB codebase than today. But I am not sure that this incremental benefit (meaning: we're not starting from zero understanding today) is sufficiently important to warrant the downsides. The downsides being that the learning curve and productivity of C++ are generally considered to be worse than Rust's.

When I read RocksDB code, my feeling is that I can certainly understand the code even written in C++, and I think the difficult part is the design principle and architecture of RocksDB.
We didn't get involved in any RocksDB development and we don't read RocksDB that often since it is quite stable, and I think that might be the main reason we don't understand RocksDB that much.
Language is not the biggest barrier from my understanding. RocksDB has been developed by many people in the past 10+ years, understanding the design principle and architecture and techs being used is most challenging part IMHO.

[ahmadbeirkdar]

So, I won't belabor the rust vs. C++ conversation, I don't think I'm really qualified to die on either hill.

But I do wonder, Rocksdb is written in C++. Today, the Venice team is far more java oriented. If we go for a native read path, we're going to have to build a competency in some non managed language in the team. It strikes me that if we build that competency in the same language that rocksdb is implemented, then we'd gain more introspection within the team in that layer. Admittedly, the situations where we're digging through the guts of rocksdb are few and far in between, but maybe that would start to change?

This could hold true, but not sure how deep you folks want to dig into RocksDB, I doubt y'all will do any development on RocksDB. Even so, if the plan using C++ here so the team will have a more hands on experience in C++ because that is what RocksDB is written in, then in my opinion Rust is still a better option as Rust programmers tend to treat and write C++ code in a Rust way vs the old style C++ way. Where as onboarding engineers who previously only used Java or Python, they will write C++ code the Java way which will lead into problems w/ memory leaks and safety. The Rust -> Modern C++ transition is quicker and the safer one. But that is even IF the team decides to work on RocksDB code

[ahmadbeirkdar]

Love the proposal, don't have much to say from a architecture, justification and a higher level aspect it looks promising.

Would love to see how things plan to be implemented, from the choice of the gRPC server to choice of how you folks decide to build the rust server, lots of promising things in that space, especially for clean, robust and explicit external APIs that insure idiomatic Rust code

[adamxchen]

LGTM