Foundation Model Inference

All

3 repositories

FlexLLMGen
Public archive
Running large language models on a single GPU for throughput-oriented scenarios.
machine-learning offloading high-throughput opt gpt-3 large-language-models deep-learning
Python
•
Apache License 2.0
•555•9.2k•52•6•Updated Oct 28, 2024Oct 28, 2024
H2O
Public
[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.
sparsity high-throughput heavy-hitters kv-cache gpt-3 large-language-models
Python
•51•413•31•1•Updated Aug 1, 2024Aug 1, 2024
DejaVu
Public
Python
•40•306•26•1•Updated Apr 2, 2024Apr 2, 2024