Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update failure-finish-reason-safety with Vertex responses #21

Merged
merged 3 commits into from
Oct 14, 2024

Conversation

andrewheard
Copy link
Collaborator

@andrewheard andrewheard commented Oct 11, 2024

Replaced streaming-failure-finish-reason-safety.txt, unary-failure-finish-reason-safety-no-content.json and unary-failure-finish-reason-safety.json with responses from Vertex AI. The previous ones appear to be sourced from Google AI (example). These responses have a few new fields in safetyRatings:

  • probabilityScore
  • severity
  • severityScore
  • blocked

Additionally the modelVersion is included in the response, and index is omitted. These responses also did not include promptFeedback.

Copy link

github-actions bot commented Oct 11, 2024

Coverage Diff

GenerateContentResponse: 43
| candidates: 38
| | index: 35 -> 32 🟡
| | safetyRatings: 29
| | | blocked: 0 -> 3 ✅
| | | probability: 29
| | | | LOW: 1 -> 4 🔵
| | | | HIGH: 4 -> 1 🟡
| | | probabilityScore: 7 -> 10 🔵
| | | severity: 7 -> 10 🔵
| | | | HARM_SEVERITY_NEGLIGIBLE: 7 -> 10 🔵
| | | | HARM_SEVERITY_LOW: 0 -> 3 ✅
| | | severityScore: 7 -> 10 🔵
| modelVersion: 0 -> 3 ✅
| promptFeedback: 27 -> 24 🟡
| | safetyRatings: 26 -> 23 🟡
| | | category: 26 -> 23 🟡
| | | | HARM_CATEGORY_HATE_SPEECH: 24 -> 21 🟡
| | | | HARM_CATEGORY_DANGEROUS_CONTENT: 25 -> 22 🟡
| | | | HARM_CATEGORY_HARASSMENT: 26 -> 23 🟡
| | | | HARM_CATEGORY_SEXUALLY_EXPLICIT: 24 -> 21 🟡
| | | probability: 26 -> 23 🟡
| | | | NEGLIGIBLE: 25 -> 22 🟡
| usageMetadata: 12 -> 15 🔵
| | candidatesTokenCount: 11 -> 12 🔵
| | promptTokenCount: 12 -> 15 🔵
| | totalTokenCount: 11 -> 14 🔵
Total Coverage: 39.29% -> 41.43% ✅

Legend:
✅ : total coverage increase
🔵 : files number increase, total coverage unaffected
🟡 : files number decrease, total coverage unaffected
❌ : total coverage decrease

@andrewheard
Copy link
Collaborator Author

@hsubox76 @DellaBitta The previous sample files came from Google AI where the index field was always included. Vertex AI omits default values (index == 0) when serializing its responses as JSON. I filed firebase/firebase-js-sdk#8566 but @dlarocque can likely explain it much better than me (thank you for the debugging help, Daniel!!).

@andrewheard andrewheard merged commit fa4ce87 into main Oct 14, 2024
8 of 9 checks passed
@andrewheard andrewheard deleted the ah/finish-reason-blocked branch October 14, 2024 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants