Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unstructured output #606

Merged
merged 6 commits into from
Aug 7, 2024
Merged

unstructured output #606

merged 6 commits into from
Aug 7, 2024

Conversation

pelikhan
Copy link
Member

@pelikhan pelikhan commented Aug 7, 2024

  • ✨ Introduced a feature enabling JSON schema outputs by setting responseType to json_schema.
  • ⬆️ Updated the version of openai and promptfoo dependencies in package.json of both cli and core package.
  • 📃 Added a new file to document Structured Output that allows generating structured data in JSON format with a JSON schema.
  • 🔧 Updated chat.ts and expander.ts to manage new scenarios of output formatting via JSON schema.
  • 🙈 Added json_schema to the responseType enum in prompt_template.d.ts for the frontend.
  • 📝 Rearranged the JSON mode documentation, introduced schema validation and improved on the structure.
  • ☔ Added tests to ensure the "strict" condition is met within JSON schemas.
  • 🎁 Included new JSON-focused genai.mjs scripts to the sample package.
  • 📗 Updated the markdown documentation as per changes in terms of interface or usage of certain commands/parameters.

generated by pr-describe

Copy link

github-actions bot commented Aug 7, 2024

The pull request seems to be introducing the concept of a "json_schema" response type. The "json_schema" response type is being handled alongside the existing "json_object" type in various parts of the codebase. The changes allow for a stricter JSON Schema that requires all fields.

This PR also includes some refactoring and formatting changes (like using destructuring for import statements and reformatting long lines of code) for better code readability.

Test cases have been added to verify the new strict schema functionality, which is good for maintaining code quality.

From the diff, all changes seem to be implemented correctly and adequately and no functional issues are apparent.

LGTM 🚀

generated by pr-review


You can specify a `responseSchema` in the script metadata which will automatically turn on the JSON mode. The output will be validated against the schema, and GenAIScript will attempt to repair the output is not valid. The script will fail if the output does not match the schema.
You can specify a [schema](/genaiscript/reference/scripts/schemas) through `responseSchema` which will automatically turn on the JSON mode. The output will be validated against the schema, and GenAIScript will attempt to repair the output is not valid. The script will fail if the output does not match the schema.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The phrase "attempt to repair the output is not valid" should be corrected to "attempt to repair if the output is not valid".

generated by pr-docs-review-commit grammar

Note that there are [several restrictions](https://platform.openai.com/docs/guides/structured-outputs/how-to-use) on the schema features supported by this mode.

- `additionalProperties: true` is not supported.
- all optional fields (e.g. not in `required`) will be returned and might be `null`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last line is missing a newline character at the end of the file.

generated by pr-docs-review-commit missing_info

@pelikhan pelikhan merged commit be31abc into main Aug 7, 2024
10 checks passed
@pelikhan pelikhan deleted the strict branch August 7, 2024 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant