Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nvidia api #65

Merged
merged 4 commits into from
Jul 17, 2024
Merged

Add nvidia api #65

merged 4 commits into from
Jul 17, 2024

Conversation

Kipok
Copy link
Collaborator

@Kipok Kipok commented Jul 17, 2024

import os
from nemo_skills.inference.server.model import get_model

os.environ['NVIDIA_API_KEY'] = "..."

model = get_model('openai', model="meta/llama3-70b-instruct", base_url="https://integrate.api.nvidia.com/v1")
prompts = [
    'Write a limerick about the wonders of GPU computing.',
]
model.generate(prompts)

or with a system message

import os
from nemo_skills.inference.server.model import get_model

os.environ['NVIDIA_API_KEY'] = "..."

model = get_model('openai', model="meta/llama3-70b-instruct", base_url="https://integrate.api.nvidia.com/v1")
prompts = [
    """<system_start>You're a coding machine, answer in Python only!<system_end>
       <user_start>Write a limerick about the wonders of GPU computing.<user_end>""",
]
model.generate(prompts)

Signed-off-by: Igor Gitman <[email protected]>
nemo_skills/inference/server/model.py Dismissed Show dismissed Hide dismissed
Signed-off-by: Igor Gitman <[email protected]>
@Kipok Kipok requested a review from i-vainn July 17, 2024 18:24
@Kipok
Copy link
Collaborator Author

Kipok commented Jul 17, 2024

Merging without review - if any issues, let's fix them in follow-up PRs

@Kipok Kipok merged commit 8f1bd2c into main Jul 17, 2024
6 checks passed
@Kipok Kipok deleted the igitman/nvidia-api branch September 30, 2024 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant