Most SAE implementations are either part of research code (messy) or robust code bases (over-engineered). This sometimes makes it hard to discern the important parts in SAE training. To this end we propose smol-sae; a very small, didactic SAE training library. This library deliberately ignores as much as possible while retaining good base performance.
This repo currently contains the base code for the following SAEs:
- Vanilla (Bricken et al.)
- Normed (Conerly et al.)
- Gated (Rajamanoharan et al.) TODO
- TopK (Gao et al.)
- Tokenized (Dooms et al.)
Please keep in mind that not all code has been thoroughly verified.