Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --backend TRT_LLM_BUILDER option to truss init #1067

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

philipkiely-baseten
Copy link
Member

@philipkiely-baseten philipkiely-baseten commented Aug 2, 2024

🚀 What

Right now, the default model.py in Truss causes an error when trying to push with the engine builder. @htrivedi99 ran into this issue during testing.

This new option for the --backend flag in truss init solves this by providing a better default model.py that does not cause an error.

I also updated the default config when using this option with scaffold for the trt_llm.build.

💻 How

🔬 Testing

Please test by creating a Truss with this model.py and using the engine builder.

pip install .
truss init --backend TRT_LLM_BUILDER --name "init test truss" init-test-truss

@bolasim bolasim removed their request for review August 2, 2024 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant