-
Notifications
You must be signed in to change notification settings - Fork 14
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add GoogleOpenAISchema with comprehensive test logging (#234)
* Add GoogleOpenAISchema with OpenAI compatibility mode - Add GoogleOpenAISchema struct with documentation - Implement create_chat method with Google API base URL - Implement create_embedding method with Google API base URL - Use GOOGLE_API_KEY for authentication * test: Add tests for GoogleOpenAISchema implementation * test: Enhance GoogleOpenAISchema tests to improve coverage * Move GoogleOpenAISchema tests to llm_openai_schema_def.jl * feat: Add Gemini 1.5 models and aliases to user_preferences.jl Added new models: - gemini-1.5-pro-latest - gemini-1.5-flash-8b-latest - gemini-1.5-flash-latest Added corresponding aliases: - gem15p - gem15f8 - gem15f Set pricing according to Google's pay-as-you-go rates: Input: /bin/bash.075/1M tokens Output: /bin/bash.30/1M tokens * Update Gemini model pricing to reflect correct per-million token rates * revert: Remove unintended changes to test/llm_google.jl * revert: restore test/llm_google.jl to original state from main branch * feat: Add GoogleProvider with proper Bearer token auth and update GoogleOpenAISchema methods - Add GoogleProvider struct with proper Bearer token authentication - Update create_chat to use GoogleProvider and openai_request - Update create_embeddings to use GoogleProvider and openai_request - Maintain consistent URL handling up to /v1beta * feat: Add Gemini models to registry and fix GoogleOpenAISchema tests - Add Gemini 1.5 models (Pro, Flash, Flash 8b) with correct pricing - Fix GoogleOpenAISchema tests to properly handle GOOGLE_API_KEY - Save and restore original API key value during testing * fix: restore gpt-4-turbo-preview model and ensure correct model ordering * Add comprehensive logging to GoogleOpenAISchema test mock servers - Add detailed request header logging - Track authorization header values and expectations - Log request body content and responses - Improve debugging capabilities for test failures * fix: Update auth_header in GoogleProvider to use OpenAIProvider implementation * chore: prepare release v0.63.0 - Update version to 0.63.0 - Add warning about token/cost counting limitations in GoogleOpenAISchema - Update changelog with correct model aliases (gem15p, gem15f, gem15f8) - Remove logging from test/llm_openai_schema_def.jl --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Loading branch information
1 parent
3cd535e
commit faaf9ce
Showing
8 changed files
with
273 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| using Test | ||
| using PromptingTools: GoogleOpenAISchema, AIMessage, aigenerate, aiembed | ||
|
|
||
| @testset "GoogleOpenAISchema" begin | ||
| # Save original API key | ||
| original_api_key = PromptingTools.GOOGLE_API_KEY | ||
|
|
||
|
|
||
| # Test with empty GOOGLE_API_KEY | ||
| PromptingTools.GOOGLE_API_KEY = "" | ||
| PORT = rand(10000:20000) | ||
| echo_server = HTTP.serve!(PORT, verbose = -1) do req | ||
| auth_header = HTTP.header(req, "Authorization") | ||
| @test HTTP.header(req, "Authorization") == "Bearer test_key" | ||
|
|
||
| content = JSON3.read(req.body) | ||
|
|
||
| response = Dict( | ||
| :choices => [ | ||
| Dict(:message => Dict(:content => "Test response"), | ||
| :finish_reason => "stop") | ||
| ], | ||
| :usage => Dict(:total_tokens => 5, | ||
| :prompt_tokens => 5, | ||
| :completion_tokens => 0)) | ||
| return HTTP.Response(200, JSON3.write(response)) | ||
| end | ||
|
|
||
| msg = aigenerate(GoogleOpenAISchema(), | ||
| "Test prompt"; | ||
| api_key = "test_key", | ||
| model = "gemini-1.5-pro-latest", | ||
| api_kwargs = (; url = "http://localhost:$(PORT)")) | ||
|
|
||
| @test msg.content == "Test response" | ||
| @test msg.finish_reason == "stop" | ||
| close(echo_server) | ||
|
|
||
| # Test with non-empty GOOGLE_API_KEY | ||
| PromptingTools.GOOGLE_API_KEY = "env_key" | ||
| PORT = rand(10000:20000) | ||
| echo_server = HTTP.serve!(PORT, verbose = -1) do req | ||
| auth_header = HTTP.header(req, "Authorization") | ||
| @test HTTP.header(req, "Authorization") == "Bearer env_key" | ||
|
|
||
| content = JSON3.read(req.body) | ||
|
|
||
| response = Dict( | ||
| :choices => [ | ||
| Dict(:message => Dict(:content => "Test response"), | ||
| :finish_reason => "stop") | ||
| ], | ||
| :usage => Dict(:total_tokens => 5, | ||
| :prompt_tokens => 5, | ||
| :completion_tokens => 0)) | ||
| return HTTP.Response(200, JSON3.write(response)) | ||
| end | ||
|
|
||
| msg = aigenerate(GoogleOpenAISchema(), | ||
| "Test prompt"; | ||
| api_key = "test_key", # This should be ignored since GOOGLE_API_KEY is set | ||
| model = "gemini-1.5-pro-latest", | ||
| api_kwargs = (; url = "http://localhost:$(PORT)")) | ||
|
|
||
| @test msg.content == "Test response" | ||
| @test msg.finish_reason == "stop" | ||
| close(echo_server) | ||
|
|
||
| # Test embeddings with empty GOOGLE_API_KEY | ||
| PromptingTools.GOOGLE_API_KEY = "" | ||
| PORT = rand(10000:20000) | ||
| echo_server = HTTP.serve!(PORT, verbose = -1) do req | ||
| auth_header = HTTP.header(req, "Authorization") | ||
| @test HTTP.header(req, "Authorization") == "Bearer test_key" | ||
|
|
||
| content = JSON3.read(req.body) | ||
|
|
||
| response = Dict(:data => [Dict(:embedding => ones(128))], | ||
| :usage => Dict(:total_tokens => 5, | ||
| :prompt_tokens => 5, | ||
| :completion_tokens => 0)) | ||
| return HTTP.Response(200, JSON3.write(response)) | ||
| end | ||
|
|
||
| msg = aiembed(GoogleOpenAISchema(), | ||
| "Test prompt"; | ||
| api_key = "test_key", | ||
| model = "gemini-1.5-pro-latest", | ||
| api_kwargs = (; url = "http://localhost:$(PORT)")) | ||
|
|
||
| @test msg.content == ones(128) | ||
| @test msg.tokens == (5, 0) | ||
| close(echo_server) | ||
|
|
||
| # Test embeddings with non-empty GOOGLE_API_KEY | ||
| PromptingTools.GOOGLE_API_KEY = "env_key" | ||
| PORT = rand(10000:20000) | ||
| echo_server = HTTP.serve!(PORT, verbose = -1) do req | ||
| auth_header = HTTP.header(req, "Authorization") | ||
| @test HTTP.header(req, "Authorization") == "Bearer env_key" | ||
|
|
||
| content = JSON3.read(req.body) | ||
|
|
||
| response = Dict(:data => [Dict(:embedding => ones(128))], | ||
| :usage => Dict(:total_tokens => 5, | ||
| :prompt_tokens => 5, | ||
| :completion_tokens => 0)) | ||
| return HTTP.Response(200, JSON3.write(response)) | ||
| end | ||
|
|
||
| msg = aiembed(GoogleOpenAISchema(), | ||
| "Test prompt"; | ||
| api_key = "test_key", # This should be ignored since GOOGLE_API_KEY is set | ||
| model = "gemini-1.5-pro-latest", | ||
| api_kwargs = (; url = "http://localhost:$(PORT)")) | ||
|
|
||
| @test msg.content == ones(128) | ||
| @test msg.tokens == (5, 0) | ||
| close(echo_server) | ||
|
|
||
| # Restore original API key | ||
| PromptingTools.GOOGLE_API_KEY = original_api_key | ||
| end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| Test Coverage Analysis for GoogleOpenAISchema | ||
|
|
||
| Current Issues: | ||
| - 6 uncovered lines in src/llm_openai_schema_defs.jl | ||
| - Patch coverage at 14.29% | ||
|
|
||
| Implementation Analysis: | ||
| 1. GoogleProvider struct (lines 219-223) | ||
| - Basic struct definition, likely covered | ||
| - auth_header method (lines 225-227) might be uncovered | ||
|
|
||
| 2. create_chat method (lines 229-244) | ||
| - Key lines that might be uncovered: | ||
| * Line 235: api_key override logic | ||
| * Line 237: GoogleProvider instantiation | ||
| * Lines 238-243: OpenAI.openai_request call | ||
|
|
||
| 3. create_embeddings method (lines 415-429) | ||
| - Similar pattern to create_chat | ||
| - Potential uncovered lines: | ||
| * Line 421: api_key override logic | ||
| * Line 422: GoogleProvider instantiation | ||
| * Lines 423-428: OpenAI.openai_request call | ||
|
|
||
| Hypotheses for Coverage Issues: | ||
| 1. Streaming callback paths not tested | ||
| - We're using openai_request directly, which might have different behavior | ||
| - Solution: Add tests for streaming scenarios | ||
|
|
||
| 2. Error handling paths not tested | ||
| - No tests for API errors or invalid responses | ||
| - Solution: Add tests with mock error responses | ||
|
|
||
| 3. Provider instantiation edge cases | ||
| - GoogleProvider creation with different URL combinations not tested | ||
| - Solution: Add tests with various URL configurations | ||
|
|
||
| 4. API key override logic not fully tested | ||
| - Need to test all combinations of empty/non-empty GOOGLE_API_KEY | ||
| - Solution: Expand current tests to cover more scenarios | ||
|
|
||
| 5. Request parameter handling not fully tested | ||
| - Different combinations of optional parameters not tested | ||
| - Solution: Add tests with various kwargs combinations | ||
|
|
||
| Most Likely Issue: | ||
| Hypothesis #4 seems most likely - our tests don't fully exercise the API key override logic in both create_chat and create_embeddings methods. The current tests only check basic scenarios, but we need to test edge cases and different combinations of API keys. | ||
|
|
||
| Action Plan: | ||
| 1. Add tests for API key override edge cases | ||
| 2. Add tests for different URL configurations | ||
| 3. Add tests for error scenarios | ||
| 4. Add tests for streaming callbacks | ||
| 5. Add tests for various kwargs combinations |