Memento

Memento is a tool used to load and save accounts from old Solana snapshots in Google Cloud Storage.

Why should I use Memento?

• No fighting solana versioning between your geyser plugin and the validator.
• No fighting the rust compiler between your geyser plugin and the validator.
• Easy programmatic access to bulk snapshot ingestion.

Developing

Build

$ cargo b --release

Run

Worker

$ ./target/release/jito-workers --help

Backfill

The following is a backfill client that backfills to clickhouse.

$ ./target/release/jito-replayer-backfill --help

How does it work

• There's a server and client.
• When starting up the server, it will load in all snapshots available across all GCP buckets you provide on the CLI.
• The clients can connect to the server over GRPC to start backfills and request information on the snapshots available.

System Setup

Note: During the development a machine with the following specs was used:

• CPU: AMD Epyc 7313P
• Cores/Threads: 16c / 32t
• RAM: 128 GB
• Drive 0: NVME0N1: 15.36 TB
• Drive 1: NVME1N1: 15.36 TB
• Drive 2: NVME2N1: 15.36 TB

Drive 1 and 2 were mounted as an NVME in RAID0 (directions below).

You can get away with NVMEs if you're dumping to an external database. Also, you should tune the number of workers based on your system memory.

Steps

Kernel modifications for validator. (Some might be unnecessary.)

$ sudo bash -c "cat >/etc/sysctl.d/21-solana-validator.conf <<EOF
# Increase UDP buffer sizes
net.core.rmem_default = 134217728
net.core.rmem_max = 134217728
net.core.wmem_default = 134217728
net.core.wmem_max = 134217728

# Increase memory mapped files limit
vm.max_map_count = 10000000
EOF"

$ sudo bash -c "cat >/etc/security/limits.d/90-solana-nofiles.conf <<EOF
# Increase process file descriptor count limit
* - nofile 1000000
EOF"

$ sudo sysctl -p /etc/sysctl.d/21-solana-validator.conf

Install packages

$ sudo apt update
$ sudo apt install -y \
      unzip \
      mdadm \
      libssl-dev \
      libudev-dev \
      pkg-config \
      zlib1g-dev \
      llvm \
      clang \
      cmake \
      make \
      libprotobuf-dev \
      protobuf-compiler \
      libpq-dev \
      nload \
      iotop \
      apt-transport-https \
      ca-certificates \
      dirmngr \
      --no-install-recommends

Install protobufs

export PROTOC_VERSION=21.12 && \
        export PROTOC_ZIP=protoc-$PROTOC_VERSION-linux-x86_64.zip && \
        curl -Ss -OL https://github.com/google/protobuf/releases/download/v$PROTOC_VERSION/$PROTOC_ZIP \
        && sudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc \
        && sudo unzip -o $PROTOC_ZIP -d /usr/local include/* \
        && rm -f $PROTOC_ZIP

Install rust:

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ source "$HOME/.cargo/env"

Mount NVMEs and setup swap (your system may differ)

$ sudo mdadm --create /dev/md0 --level=0 --raid-devices=2 \
 /dev/nvme1n1 \
 /dev/nvme2n1
$ sudo mkfs.ext4 -F /dev/md0
$ sudo mkdir -p /mnt/disks/ledger
$ sudo mount /dev/md0 /mnt/disks/ledger
$ sudo chmod a+w /mnt/disks/ledger

$ sudo fallocate -l 512G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ sudo swapon --show

Setup and install clickhouse.

$ GNUPGHOME=$(mktemp -d)
$ sudo GNUPGHOME="$GNUPGHOME" gpg --no-default-keyring --keyring /usr/share/keyrings/clickhouse-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 8919F6BD2B48D754
$ sudo rm -r "$GNUPGHOME"
$ sudo chmod +r /usr/share/keyrings/clickhouse-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb stable main" | sudo tee \
    /etc/apt/sources.list.d/clickhouse.list
$ sudo apt-get update
$ sudo apt-get install -y clickhouse-server clickhouse-client
$ sudo systemctl start clickhouse-server