Issue with RocksDB growth in Solana private cluster despite low transaction volume

[ImmuneFOMO]

Good afternoon everyone,

We recently started our Solana cluster for a project where we require hiding some transaction data and disabling commissions. However, we encountered a significant issue with RocksDB growing to over 1 terabyte in just 1-2 weeks, despite having almost zero transactions in the cluster during this time. We've tried various optimization techniques, including adjusting flags in the genesis validator script and limiting transactions per second (TPS), but the problem persists.

Genesis Validator Script:

#!/usr/bin/bash

file_latest_slot="/home/sol/latest_optimistic_slot"
file_latest_hash="/home/sol/latest_optimistic_hash"

# Check for the presence of latest slot and hash files
if [ -s "$file_latest_slot" ] && [ -s "$file_latest_hash" ];
then
    # Path 1: Latest Slot and Hash found
    SLOT_X=$(cat "$file_latest_slot")
    HASH_X=$(cat "$file_latest_hash")

    # Purge slots after the latest snapshot to manage ledger size
    solana-ledger-tool purge -l /var/solana/ledger/ "$SLOT_X"
    
    # Deleting latest slot and hash files
    rm "$file_latest_slot" "$file_latest_hash"

    solana-validator \
        --wait-for-supermajority "$SLOT_X" \
        --expected-bank-hash "$HASH_X" \
        --require-tower \
        --ledger /var/solana/ledger \
        --accounts /var/solana/accounts/ramdisk \
        --rpc-port 8899 \
        --identity /home/sol/config/validator-keypair.json \
        --vote-account /home/sol/config/vote-account-keypair.json \
        --rpc-faucet-address 127.0.0.1:9900 \
        --enable-rpc-transaction-history \
        --enable-extended-tx-metadata-storage \
        --init-complete-file /home/sol/init-completed \
        --no-wait-for-vote-to-start-leader \
        --no-poh-speed-test \
        --no-os-network-limits-test \
        --limit-ledger-size \
        --full-rpc-api \
        --gossip-port 8001 \
        --gossip-host cbc-n0.comudel.com \
        --dynamic-port-range 8000-8020 \
        --log /home/sol/logs/solana-validator.log \
        --log-messages-bytes-limit 10000000 \
        --allow-private-addr \
        --rpc-threads 2  
else
    # Path 2: Starting without latest slot and hash
    [ -e "$file_latest_slot" ] && rm "$file_latest_slot"
    [ -e "$file_latest_hash" ] && rm "$file_latest_hash"
    
    # Normal startup path with TPS reduction measures
    solana-validator \
        --require-tower \
        --ledger /var/solana/ledger \
        --accounts /var/solana/accounts/ramdisk \
        --rpc-port 8899 \
        --identity /home/sol/config/validator-keypair.json \
        --vote-account /home/sol/config/vote-account-keypair.json \
        --rpc-faucet-address 127.0.0.1:9900 \
        --enable-rpc-transaction-history \
        --enable-extended-tx-metadata-storage \
        --init-complete-file /home/sol/init-completed \
        --no-wait-for-vote-to-start-leader \
        --no-poh-speed-test \
        --no-os-network-limits-test \
        --limit-ledger-size \
        --full-rpc-api \
        --gossip-port 8001 \
        --gossip-host cbc-n0.comudel.com \
        --dynamic-port-range 8000-8020 \
        --log /home/sol/logs/solana-validator.log \
        --log-messages-bytes-limit 10000000 \
        --allow-private-addr \
        --rpc-threads 2 
fi

Issue:
Despite our efforts, RocksDB continues to grow rapidly even when the cluster experiences minimal transaction activity. We've considered implementing snapshotting and resetting the cluster when the ledger size reaches certain limits, but we're unsure if this would effectively address the problem.

Any suggestions or solutions for optimizing RocksDB growth in our Solana private cluster would be greatly appreciated.

Thank you for your assistance.

[github-actions]

This repository is no longer in use. Please re-open this issue in the agave repo: https://github.com/anza-xyz/agave