feat: openai vision toolkit #269
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
use 4o-mini bump version
AaronGoldsmith
commented
Nov 18, 2024
Comment on lines
+28
to
+42
| def content_converter(contents: list) -> list[Content]: | ||
| result = [] | ||
| for c in contents: | ||
| if isinstance(c, dict) and 'type' in c: # Structured content logic | ||
| content_type = c.pop('type') | ||
| if content_type in CONTENT_TYPES: | ||
| result.append(CONTENT_TYPES[content_type](**c)) | ||
| elif isinstance(c, Content): # Already a Content instance | ||
| result.append(c) | ||
| elif isinstance(c, str): # Plain text handling | ||
| result.append(Text(text=c)) | ||
| else: | ||
| # Handle unexpected content formats if necessary | ||
| raise ValueError(f"Unsupported content type: {type(c)}") | ||
| return result |
There was a problem hiding this comment.
Before updating content_converter I would get the following:
"Traceback (most recent call last):
File \"/Users/aarong/Development/goose/packages/exchange/src/exchange/exchange.py\", line 149, in call_function
output = json.dumps(tool.function(**tool_use.parameters))
File \"/Users/aarong/Development/goose/src/goose/toolkit/vision.py\", line 22, in describe_image
user_message = Message(role=\"user\", content=[f\"{instructions}: \", image])
File \"<attrs generated init exchange.message.Message>\", line 13, in __init__
_setattr('content', __attr_converter_content(content))
File \"/Users/aarong/Development/goose/packages/exchange/src/exchange/message.py\", line 29, in content_converter\n return [(CONTENT_TYPES[c.pop(\"type\")](**c) if c.__class__ not in CONTENT_TYPES.values() else c) for c in contents]
File \"/Users/aarong/Development/goose/packages/exchange/src/exchange/message.py\", line 29, in <listcomp>\n return [(CONTENT_TYPES[c.pop(\"type\")](**c) if c.__class__ not in CONTENT_TYPES.values() else c) for c in contents]\nAttributeError: 'str' object has no attribute 'pop'
'str' object has no attribute 'pop'", "is_error": true, "type": "ToolResult"}]
add support for local image files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new
ImageUrlcontent type and integrates it across various modules in theexchangepackage. The changes include updates to the content handling logic, message validation, and new tests to ensure the functionality of theImageUrltype. Additionally, a newVisionToolkithas been added to thegoosetoolkit for image analysis using AI capabilities.The toolkit uses the OpenAI Vision API to analyze image URLs and base64-encoded images.
Future enhancements: Support for additional providers (ex: Anthropic)
New Content Type and Integration:
ImageUrlclass to handle image URLs as content and implementedto_dictandsummarymethods inpackages/exchange/src/exchange/content.py.validate_role_and_contentandcontent_converterfunctions inpackages/exchange/src/exchange/message.pyto supportImageUrl. [1] [2]image_contentproperty to theMessageclass to retrieve allImageUrlinstances inpackages/exchange/src/exchange/message.py.Providers and Utilities:
messages_to_openai_specfunction inpackages/exchange/src/exchange/providers/utils.pyto handleImageUrlcontent.Testing Enhancements:
ImageUrlintegration inpackages/exchange/tests/test_image_tool_integration.pyandpackages/exchange/tests/test_message.py. [1] [2]New Toolkit:
VisionToolkitfor image analysis insrc/goose/toolkit/vision.pyand updatedpyproject.tomlto include the new toolkit entry point. [1] [2]If the team prefers having the toolkit and the new content type as separate PRs, I’d be happy to split them up.