Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unstructured output #606

Merged
merged 6 commits into from
Aug 7, 2024
Merged

unstructured output #606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ced3af1
strict output
pelikhan Aug 7, 2024
b748e7b
json_schema support
pelikhan Aug 7, 2024
9284964
updated docs
pelikhan Aug 7, 2024
0541e62
typos
pelikhan Aug 7, 2024
fcad638
typechecks
pelikhan Aug 7, 2024
60e32d9
pr feedback
pelikhan Aug 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 32 additions & 31 deletions THIRD_PARTY_LICENSES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3153,7 +3153,7 @@ MIT License

The following npm package may be included in this product:

- [email protected].1
- [email protected].3

This package contains the following license and notice below:

Expand Down Expand Up @@ -14667,7 +14667,7 @@ Apache-2.0

The following npm package may be included in this product:

- [email protected].30001649
- [email protected].30001650

This package contains the following license and notice below:

Expand Down Expand Up @@ -17132,6 +17132,7 @@ The following npm packages may be included in this product:
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]

These packages each contain the following license and notice below:
Expand Down Expand Up @@ -17860,7 +17861,7 @@ THIS SOFTWARE.

The following npm package may be included in this product:

- [email protected].4
- [email protected].5

This package contains the following license and notice below:

Expand Down Expand Up @@ -19851,11 +19852,11 @@ The following npm packages may be included in this product:
- [email protected]
- [email protected]
- [email protected]
- [email protected].1
- [email protected].3
- [email protected]
- [email protected].1
- [email protected].1
- [email protected].1
- [email protected].3
- [email protected].3
- [email protected].3
- [email protected]
- [email protected]
- [email protected]
Expand Down Expand Up @@ -21461,7 +21462,7 @@ SOFTWARE.
The following npm packages may be included in this product:

- @drauu/[email protected]
- @slidev/[email protected].23
- @slidev/[email protected].24
- @slidev/[email protected]
- [email protected]

Expand Down Expand Up @@ -21493,9 +21494,9 @@ SOFTWARE.

The following npm packages may be included in this product:

- @slidev/[email protected].23
- @slidev/[email protected].23
- @slidev/[email protected].23
- @slidev/[email protected].24
- @slidev/[email protected].24
- @slidev/[email protected].24

These packages each contain the following license and notice below:

Expand Down Expand Up @@ -21525,7 +21526,7 @@ SOFTWARE.

The following npm packages may be included in this product:

- [email protected].1
- [email protected].2
- [email protected]
- [email protected]

Expand Down Expand Up @@ -21649,7 +21650,7 @@ SOFTWARE.

The following npm package may be included in this product:

- @astrojs/[email protected].1
- @astrojs/[email protected].2

This package contains the following license and notice below:

Expand Down Expand Up @@ -22147,8 +22148,8 @@ SOFTWARE.

The following npm packages may be included in this product:

- @floating-ui/[email protected].5
- @floating-ui/[email protected].5
- @floating-ui/[email protected].7
- @floating-ui/[email protected].7

These packages each contain the following license and notice below:

Expand Down Expand Up @@ -22398,7 +22399,7 @@ SOFTWARE.

The following npm package may be included in this product:

- @astrojs/[email protected].1
- @astrojs/[email protected].2

This package contains the following license and notice below:

Expand Down Expand Up @@ -22645,7 +22646,7 @@ SOFTWARE.

The following npm package may be included in this product:

- @astrojs/[email protected].3
- @astrojs/[email protected].4

This package contains the following license and notice below:

Expand Down Expand Up @@ -22826,7 +22827,7 @@ SOFTWARE.

The following npm package may be included in this product:

- @astrojs/[email protected].1
- @astrojs/[email protected].2

This package contains the following license and notice below:

Expand Down Expand Up @@ -26688,16 +26689,16 @@ SOFTWARE.

The following npm packages may be included in this product:

- @vue/[email protected].35
- @vue/[email protected].35
- @vue/[email protected].35
- @vue/[email protected].35
- @vue/[email protected].35
- @vue/[email protected].35
- @vue/[email protected].35
- @vue/[email protected].35
- @vue/[email protected].35
- [email protected].35
- @vue/[email protected].36
- @vue/[email protected].36
- @vue/[email protected].36
- @vue/[email protected].36
- @vue/[email protected].36
- @vue/[email protected].36
- @vue/[email protected].36
- @vue/[email protected].36
- @vue/[email protected].36
- [email protected].36

These packages each contain the following license and notice below:

Expand Down Expand Up @@ -26757,7 +26758,7 @@ THE SOFTWARE.

The following npm package may be included in this product:

- [email protected].2
- [email protected].3

This package contains the following license and notice below:

Expand Down Expand Up @@ -27068,7 +27069,7 @@ THE SOFTWARE.

The following npm package may be included in this product:

- @lvce-editor/verror@1.3.0
- @lvce-editor/verror@1.4.0

This package contains the following license and notice below:

Expand Down Expand Up @@ -27659,7 +27660,7 @@ THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR I

The following npm package may be included in this product:

- [email protected].40
- [email protected].41

This package contains the following license and notice below:

Expand Down
12 changes: 7 additions & 5 deletions demo/genaisrc/genaiscript.d.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 7 additions & 5 deletions docs/genaisrc/genaiscript.d.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/src/content/docs/reference/cli/commands.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ Options:
-td, --test-delay <string> delay between tests in seconds
--no-cache disable LLM result cache
-v, --verbose verbose output
-pv, --promptfoo-version [version] promptfoo version, default is ^0.75.0
-pv, --promptfoo-version [version] promptfoo version, default is ^0.75.2
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
-os, --out-summary <file> append output summary in file
-h, --help display help for command
```
Expand Down
53 changes: 21 additions & 32 deletions docs/src/content/docs/reference/scripts/json-mode.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,52 +10,41 @@ Some models support forcing the output format to a JSON object, like the [JSON O

The generated file name will be `[spec].[template].json`.

## `responseSchema`
```js 'responseType: "json_object"'
script({
responseType: "json_object",
})
```

## Schema validation

You can specify a `responseSchema` in the script metadata which will automatically turn on the JSON mode. The output will be validated against the schema, and GenAIScript will attempt to repair the output is not valid. The script will fail if the output does not match the schema.
You can specify a [schema](/genaiscript/reference/scripts/schemas) through `responseSchema` which will automatically turn on the JSON mode. The output will be validated against the schema, and GenAIScript will attempt to repair the output is not valid. The script will fail if the output does not match the schema.
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The phrase "attempt to repair the output is not valid" should be corrected to "attempt to repair if the output is not valid".

generated by pr-docs-review-commit grammar


```js "responseSchema"
script({
responseType: "json_object",
responseSchema: {
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
type: "object",
properties: {
cities: {
type: "array",
description:
"A list of cities with population and elevation information.",
items: {
type: "object",
description:
"A city with population and elevation information.",
properties: {
name: {
type: "string",
description: "The name of the city.",
},
population: {
type: "number",
description: "The population of the city.",
},
},
required: ["name", "population", "url"],
},
},
name: { type: "string" },
age: { type: "number" },
},
required: ["name", "age"],
},
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
})
```

## `responseType`
## Inline schemas

You can also specify the [schema inline](/genaiscript/reference/scripts/schemas) in the script and use a mixed markdown/data that GenAIScript will parse.
pelikhan marked this conversation as resolved.
Show resolved Hide resolved

You can also enable this mode without a schema by setting `response_type` to `json_object`.
## Structured Output

```javascript
Recent models have added a [structured output](/genaiscript/reference/scripts/structured-output) mode that is more strict than JSON mode. This mode is enabled by setting `responseType` to `json_schema`.
pelikhan marked this conversation as resolved.
Show resolved Hide resolved

```js "responseSchema"
script({
...,
responseType: `json_object`,
responseType: "json_schema",
responseSchema: {...},
})
```

## Inline schemas

You can also specify the [schema inline](/genaiscript/reference/scripts/schemas) in the script and use a mixed markdown/data that GenAIScript will parse.
28 changes: 28 additions & 0 deletions docs/src/content/docs/reference/scripts/structured-output.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
title: Structured Output
sidebar:
order: 12.1
---

Structured output is a feature that allows you to generate structured data in JSON format with a [JSON schema](/genaiscript/reference/scripts/schemas). This is more strict than [JSON mode](/genaiscript/reference/scripts/json-mode) and is supported by `gpt-4o-mini`, `gpt-4o-2024-08-06` and [later models](https://platform.openai.com/docs/guides/structured-outputs/structured-outputs-vs-json-mode).

To enable this mode, set `responseType` to `json_schema`.

```js "responseType: 'json_schema'"
script({
responseType: "json_schema",
responseSchema: {
type: "object",
properties: {
name: { type: "string" },
age: { type: "number" },
},
required: ["name", "age"],
},
})
```

Note that there are [several restrictions](https://platform.openai.com/docs/guides/structured-outputs/how-to-use) on the schema features supported by this mode.
pelikhan marked this conversation as resolved.
Show resolved Hide resolved

- `additionalProperties: true` is not supported.
- all optional fields (e.g. not in `required`) will be returned and might be `null`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last line is missing a newline character at the end of the file.

generated by pr-docs-review-commit missing_info

Loading