Motivation

Wanted to create a distributed filesystem with the following key features

• Highly availability
• Consistency and data integrity: All data includes a checksum of the data. It calculates checksums and writes them along with the data. When reading that data later, it recalculates the checksums. If the checksums do not match, meaning detecting one or more data errors, ZFS will attempt to automatically correct errors with Reed–Solomon codes
• High throughput with large and many files and large number of clients at the same time
• Modular and ergonomic design
• Masterless cluster
• Efficient communication between nodes and with clients using recent tech and hardware optimizations like RDMA, SIMD, QUIC, Apache Arrow, Kafka, Pulsar
• Efficient file transfer and sync using recent tech and hardware optimizations like zero-copy, BitTorrent, DHT, QUIC
• Performance with caching mechanisms provides increased performance with an advanced memory-based read cache
• Efficient distributed metadata storage
• Replication
• Encryption, deduplication, compression
• ACL, authentication, authorization
• History/versioning
• High performance using Rust and modern stack
• Vast collection of clients and libs
• Expose with FUSE and NFS
• Possibility of global distribution
• Great observability and tools