Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(langchain): [MLOB-1949] add langchain tests for vertexai and gemini integrations #11568

Draft
wants to merge 34 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
e16e2ba
wip; submit spans to llmobs
ncybul Nov 20, 2024
9e30a14
setup llmobs tests and add vertexai to google generativeai riotfile pkgs
ncybul Nov 20, 2024
a150ed5
enable vertexai integration and fix test bugs
ncybul Nov 20, 2024
f5eedf2
added more llmobs tests
ncybul Nov 20, 2024
05b8f58
add tests
ncybul Nov 22, 2024
ae20d5d
added the rest of vertexai tests
ncybul Nov 24, 2024
2c9e5b9
pass in model instance to kwargs
ncybul Nov 25, 2024
3674605
only submit llmobs payloads for sampled spans
ncybul Nov 25, 2024
0867fe6
remove riot changes
ncybul Nov 25, 2024
2d30b4f
remove unnecessary dict checks for contents object
ncybul Nov 25, 2024
fb3f0fb
move identical helper methods to shared utils file
ncybul Nov 25, 2024
6e50c72
add missing metric key imports
ncybul Nov 25, 2024
e04adc9
update method signatures for _llmobs_set_tags
ncybul Nov 25, 2024
06d1a03
update get metadata helper
ncybul Nov 25, 2024
8b454a1
create helper for extracting system instructions
ncybul Nov 25, 2024
e4b56dc
create helper method to extract messages from content object
ncybul Nov 25, 2024
ca046f9
remove dependency on vertexai import in integration
ncybul Nov 26, 2024
5ba4f17
add tests for various generation config settings
ncybul Nov 26, 2024
7648b85
remove vertexai import in integration utils
ncybul Nov 26, 2024
13918ec
run black
ncybul Nov 26, 2024
11ea7ff
update tests with new history argument
ncybul Nov 26, 2024
a1f1d9f
add release note
ncybul Nov 26, 2024
b7c6726
apply ruff fixes
ncybul Nov 26, 2024
daecc5e
import Part type locally to avoid vertexai dependency
ncybul Nov 26, 2024
34dee34
run black
ncybul Nov 26, 2024
d399cdb
pass in history via kwargs
ncybul Nov 26, 2024
3c8dc1e
make system instruction helper work for gemini integration
ncybul Nov 26, 2024
5e2212a
return early when span errors
ncybul Nov 26, 2024
0f0e6f8
run black
ncybul Nov 27, 2024
9d09bea
apply ruff fixes
ncybul Nov 27, 2024
0fa37f9
add langchain vertexai tests
ncybul Nov 27, 2024
d1badbb
Revert "add langchain vertexai tests"
ncybul Nov 27, 2024
fd7e346
add langchain vertexai tests
ncybul Nov 27, 2024
ee6a63a
add other langchain tests for vertexai integration
ncybul Nov 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,16 @@
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --allow-unsafe --no-annotate .riot/requirements/f12fa99.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/102c18b.in
#
annotated-types==0.7.0
attrs==24.2.0
cachetools==5.5.0
certifi==2024.8.30
charset-normalizer==3.4.0
coverage[toml]==7.6.7
coverage[toml]==7.6.8
docstring-parser==0.16
google-ai-generativelanguage==0.6.13
google-api-core[grpc]==2.23.0
google-auth==2.36.0
google-cloud-aiplatform[all]==1.71.1
Expand All @@ -36,8 +37,8 @@ proto-plus==1.25.0
protobuf==5.28.3
pyasn1==0.6.1
pyasn1-modules==0.4.1
pydantic==2.9.2
pydantic-core==2.23.4
pydantic==2.10.2
pydantic-core==2.27.1
pytest==8.3.3
pytest-asyncio==0.24.0
pytest-cov==6.0.0
Expand Down
123 changes: 123 additions & 0 deletions .riot/requirements/105966d.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile --allow-unsafe --no-annotate .riot/requirements/105966d.in
#
ai21==3.0.0
ai21-tokenizer==0.12.0
aiohappyeyeballs==2.4.3
aiohttp==3.11.8
aiosignal==1.3.1
annotated-types==0.7.0
anthropic==0.26.0
anyio==4.6.2.post1
async-timeout==4.0.3
attrs==24.2.0
boto3==1.34.51
botocore==1.34.51
cachetools==5.5.0
certifi==2024.8.30
charset-normalizer==3.4.0
cohere==5.4.0
coverage[toml]==7.6.8
dataclasses-json==0.6.7
defusedxml==0.7.1
distro==1.9.0
docstring-parser==0.16
exceptiongroup==1.2.2
faiss-cpu==1.8.0
fastavro==1.9.7
filelock==3.16.1
frozenlist==1.5.0
fsspec==2024.10.0
google-ai-generativelanguage==0.6.13
google-api-core[grpc]==2.23.0
google-auth==2.36.0
google-cloud-aiplatform[all]==1.71.1
google-cloud-bigquery==3.27.0
google-cloud-core==2.4.1
google-cloud-resource-manager==1.13.1
google-cloud-storage==2.18.2
google-crc32c==1.6.0
google-resumable-media==2.7.2
googleapis-common-protos[grpc]==1.66.0
greenlet==3.0.3
grpc-google-iam-v1==0.13.1
grpcio==1.68.0
grpcio-status==1.68.0
h11==0.14.0
httpcore==1.0.7
httpx==0.27.2
httpx-sse==0.4.0
huggingface-hub==0.26.2
hypothesis==6.45.0
idna==3.10
iniconfig==2.0.0
jmespath==1.0.1
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.1.20
langchain-anthropic==0.1.11
langchain-aws==0.1.3
langchain-cohere==0.1.4
langchain-community==0.0.38
langchain-core==0.1.52
langchain-google-vertexai==0.1.3
langchain-openai==0.1.6
langchain-pinecone==0.1.0
langchain-text-splitters==0.0.2
langsmith==0.1.147
marshmallow==3.23.1
mock==5.1.0
multidict==6.1.0
mypy-extensions==1.0.0
numexpr==2.8.5
numpy==1.26.4
openai==1.30.3
opentracing==2.4.0
orjson==3.10.12
packaging==23.2
pinecone-client==3.2.2
pluggy==1.5.0
propcache==0.2.0
proto-plus==1.25.0
protobuf==5.28.3
psutil==6.1.0
pyasn1==0.6.1
pyasn1-modules==0.4.1
pydantic==2.9.0
pydantic-core==2.23.2
pytest==8.3.3
pytest-asyncio==0.23.7
pytest-cov==6.0.0
pytest-mock==3.14.0
pytest-randomly==3.10.1
python-dateutil==2.9.0.post0
pyyaml==6.0.2
regex==2024.11.6
requests==2.32.3
requests-toolbelt==1.0.0
rsa==4.9
s3transfer==0.10.4
sentencepiece==0.2.0
shapely==2.0.6
six==1.16.0
sniffio==1.3.1
sortedcontainers==2.4.0
sqlalchemy==2.0.36
tenacity==8.5.0
tiktoken==0.8.0
tokenizers==0.19.1
tomli==2.1.0
tqdm==4.67.1
types-protobuf==4.25.0.20240417
types-requests==2.32.0.20241016
typing-extensions==4.12.2
typing-inspect==0.9.0
tzdata==2024.2
urllib3==2.0.7
vcrpy==5.1.0
vertexai==1.71.1
wrapt==1.17.0
yarl==1.18.0
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,16 @@
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# pip-compile --allow-unsafe --no-annotate .riot/requirements/23eab38.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/107d415.in
#
annotated-types==0.7.0
attrs==24.2.0
cachetools==5.5.0
certifi==2024.8.30
charset-normalizer==3.4.0
coverage[toml]==7.6.7
coverage[toml]==7.6.8
docstring-parser==0.16
google-ai-generativelanguage==0.6.13
google-api-core[grpc]==2.23.0
google-auth==2.36.0
google-cloud-aiplatform[all]==1.71.1
Expand All @@ -36,8 +37,8 @@ proto-plus==1.25.0
protobuf==5.28.3
pyasn1==0.6.1
pyasn1-modules==0.4.1
pydantic==2.9.2
pydantic-core==2.23.4
pydantic==2.10.2
pydantic-core==2.27.1
pytest==8.3.3
pytest-asyncio==0.24.0
pytest-cov==6.0.0
Expand Down
94 changes: 60 additions & 34 deletions .riot/requirements/1761cfc.txt → .riot/requirements/10e47e7.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,42 +2,58 @@
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile --allow-unsafe --no-annotate .riot/requirements/1761cfc.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/10e47e7.in
#
ai21==2.14.1
ai21==3.0.0
ai21-tokenizer==0.12.0
aiohappyeyeballs==2.4.0
aiohttp==3.10.5
aiohappyeyeballs==2.4.3
aiohttp==3.11.7
aiosignal==1.3.1
annotated-types==0.7.0
anthropic==0.26.0
anyio==4.4.0
anyio==4.6.2.post1
async-timeout==4.0.3
attrs==24.2.0
boto3==1.34.51
botocore==1.34.51
cachetools==5.5.0
certifi==2024.8.30
charset-normalizer==3.3.2
cohere==5.9.1
coverage[toml]==7.6.1
charset-normalizer==3.4.0
cohere==5.11.4
coverage[toml]==7.6.8
dataclasses-json==0.6.7
defusedxml==0.7.1
distro==1.9.0
docstring-parser==0.16
exceptiongroup==1.2.2
fastavro==1.9.7
filelock==3.16.0
frozenlist==1.4.1
fsspec==2024.9.0
filelock==3.16.1
frozenlist==1.5.0
fsspec==2024.10.0
google-ai-generativelanguage==0.6.13
google-api-core[grpc]==2.23.0
google-auth==2.36.0
google-cloud-aiplatform[all]==1.71.1
google-cloud-bigquery==3.27.0
google-cloud-core==2.4.1
google-cloud-resource-manager==1.13.1
google-cloud-storage==2.18.2
google-crc32c==1.6.0
google-resumable-media==2.7.2
googleapis-common-protos[grpc]==1.66.0
greenlet==3.0.3
grpc-google-iam-v1==0.13.1
grpcio==1.68.0
grpcio-status==1.68.0
h11==0.14.0
httpcore==1.0.5
httpcore==1.0.7
httpx==0.27.2
httpx-sse==0.4.0
huggingface-hub==0.24.6
huggingface-hub==0.26.2
hypothesis==6.45.0
idna==3.8
idna==3.10
iniconfig==2.0.0
jiter==0.5.0
jiter==0.8.0
jmespath==1.0.1
jsonpatch==1.33
jsonpointer==3.0.0
Expand All @@ -46,52 +62,62 @@ langchain-anthropic==0.1.13
langchain-aws==0.1.6
langchain-cohere==0.1.8
langchain-core==0.2.0
langchain-google-vertexai==1.0.4
langchain-openai==0.1.7
langchain-pinecone==0.1.3
langchain-text-splitters==0.2.1
langsmith==0.1.117
marshmallow==3.22.0
langsmith==0.1.146
marshmallow==3.23.1
mock==5.1.0
multidict==6.1.0
mypy-extensions==1.0.0
numexpr==2.8.5
numpy==1.26.4
openai==1.44.1
openai==1.55.2
opentracing==2.4.0
orjson==3.10.7
orjson==3.10.12
packaging==23.2
parameterized==0.9.0
pinecone-client==5.0.1
pinecone-plugin-inference==1.0.3
pinecone-plugin-inference==1.1.0
pinecone-plugin-interface==0.0.7
pluggy==1.5.0
psutil==6.0.0
pydantic==2.9.1
pydantic-core==2.23.3
propcache==0.2.0
proto-plus==1.25.0
protobuf==5.28.3
psutil==6.1.0
pyasn1==0.6.1
pyasn1-modules==0.4.1
pydantic==2.10.2
pydantic-core==2.27.1
pytest==8.3.3
pytest-asyncio==0.23.7
pytest-cov==5.0.0
pytest-cov==6.0.0
pytest-mock==3.14.0
pytest-randomly==3.10.1
python-dateutil==2.9.0.post0
pyyaml==6.0.2
regex==2024.7.24
regex==2024.11.6
requests==2.32.3
s3transfer==0.10.2
requests-toolbelt==1.0.0
rsa==4.9
s3transfer==0.10.4
sentencepiece==0.2.0
shapely==2.0.6
six==1.16.0
sniffio==1.3.1
sortedcontainers==2.4.0
sqlalchemy==2.0.34
sqlalchemy==2.0.36
tenacity==8.5.0
tiktoken==0.7.0
tokenizers==0.20.0
tomli==2.0.1
tqdm==4.66.5
types-requests==2.32.0.20240907
tiktoken==0.8.0
tokenizers==0.21.0
tomli==2.1.0
tqdm==4.67.1
types-requests==2.32.0.20241016
typing-extensions==4.12.2
typing-inspect==0.9.0
urllib3==2.0.7
vcrpy==5.1.0
wrapt==1.16.0
yarl==1.11.1
vertexai==1.71.1
wrapt==1.17.0
yarl==1.18.0
Loading
Loading