Sagemaker - Support for custom_attributes for accept_eula for Llama 2

[rhlarora84]

Problem

I could not find the ability to set custom_attributes when using Llama2 on AWS Sagemaker. Sagemaker requires custom_attributes='accept_eula=true' for the Llama2 model.

Error -

File "/opt/conda/lib/python3.11/site-packages/botocore/client.py", line 980, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (424) from primary with message "{
"code":424,
"message":"prediction failure",
"error":"Need to pass custom_attributes='accept_eula=true' as part of header. This means you have read and accept the end-user license agreement (EULA) of the model. EULA can be found in model card description or from https://ai.meta.com/resources/models-and-libraries/llama-downloads/."
}". See https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/jumpstart-dft-meta-textgeneration-llama-2-7b-f in account for more information.

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[JasonWeill]

Thank you for your contribution! This requires langchain-ai/langchain#8326 to be fixed, and for us to upgrade to a newer LangChain version once this is fixed.

[shenshaoyong]

I verified that this feature is ready to use. I can access the llama2 deployed on SageMaker endpoint by using SageMaker Jumpstart.

[thuandoan-hcl]

I verified that this feature is ready to use. I can access the llama2 deployed on SageMaker endpoint by using SageMaker Jumpstart.

Can you share the code? Thanks

[rhlarora84]

@JasonWeill It would be great if credentials_profile_name can be supported as well. This will allow to override the AWS credentials that are used by using named profile.
https://python.langchain.com/docs/integrations/llms/sagemaker

[rhlarora84]

@JasonWeill

I tried Sagemaker endpoint with Langchain and it does support Custom Attributes
llm=SagemakerEndpoint( endpoint_name="jumpstart-dft-llama-codellama-7b-instruct", region_name='us-east-1', model_kwargs={"max_new_tokens": 1500, "top_p": 0.9, "temperature": 0.6}, endpoint_kwargs={"CustomAttributes": 'accept_eula=true'}, content_handler=content_handler )
langchain 0.0.277

[YYyp99]

I also encountered this issue when using Sagemaker's batch transform job, so where should I add this parameter to the batch transform job