Skip to content

Commit

Permalink
data queries (#965)
Browse files Browse the repository at this point in the history
  • Loading branch information
pelikhan authored Dec 23, 2024
1 parent 8a92fd8 commit 3c9e098
Show file tree
Hide file tree
Showing 21 changed files with 2,205 additions and 471 deletions.
24 changes: 24 additions & 0 deletions THIRD_PARTY_LICENSES.md
Original file line number Diff line number Diff line change
Expand Up @@ -7091,6 +7091,30 @@ THIS SOFTWARE.

-----------

The following npm package may be included in this product:

- [email protected]

This package contains the following license:

ISC License

Copyright (c) 2020, Stefan Terdell

Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

-----------

The following npm package may be included in this product:

- [email protected]
Expand Down
2 changes: 1 addition & 1 deletion docs/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"@astrojs/starlight": "^0.30.3",
"astro": "^5.1.1",
"rehype-mermaid": "^3.0.0",
"starlight-blog": "^0.16.0",
"starlight-blog": "^0.16.1",
"starlight-links-validator": "^0.14.0",
"starlight-package-managers": "^0.8.1",
"typescript": "5.7.2",
Expand Down
26 changes: 14 additions & 12 deletions docs/src/content/docs/reference/scripts/content-safety.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,15 @@ GenAIScript has multiple built-in safety features to protect the system from mal

The following safety prompts are included by default when running a prompt, unless the system option is configured:

- [system.safety_harmful_content](/genaiscript/reference/scripts/system#systemsafety_harmful_content), safety prompt against Harmful Content: Hate and Fairness, Sexual, Violence, Self-Harm. See https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/safety-system-message-templates.
- [system.safety_jailbreak](/genaiscript/reference/scripts/system#systemsafety_jailbreak), safety script to ignore prompting instructions in code sections, which are created by the `def` function.
- [system.safety_protected_material](/genaiscript/reference/scripts/system#systemsafety_protected_material) safety prompt against Protected material. See https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/safety-system-message-templates
- [system.safety_harmful_content](/genaiscript/reference/scripts/system#systemsafety_harmful_content), safety prompt against Harmful Content: Hate and Fairness, Sexual, Violence, Self-Harm. See https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/safety-system-message-templates.
- [system.safety_jailbreak](/genaiscript/reference/scripts/system#systemsafety_jailbreak), safety script to ignore prompting instructions in code sections, which are created by the `def` function.
- [system.safety_protected_material](/genaiscript/reference/scripts/system#systemsafety_protected_material) safety prompt against Protected material. See https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/safety-system-message-templates

Other system scripts can be added to the prompt by using the `system` option.

- [system.safety_ungrounded_content_summarization](/genaiscript/reference/scripts/system#systemsafety_ungrounded_content_summarization) safety prompt against ungrounded content in summarization
- [system.safety_canary_word](/genaiscript/reference/scripts/system#systemsafety_canary_word) safety prompt against prompt leaks.
- [system.safety_validate_harmful_content](/genaiscript/reference/scripts/system#systemsafety_validate_harmful_content) runs the `detectHarmfulContent` method to validate the output of the prompt.
- [system.safety_ungrounded_content_summarization](/genaiscript/reference/scripts/system#systemsafety_ungrounded_content_summarization) safety prompt against ungrounded content in summarization
- [system.safety_canary_word](/genaiscript/reference/scripts/system#systemsafety_canary_word) safety prompt against prompt leaks.
- [system.safety_validate_harmful_content](/genaiscript/reference/scripts/system#systemsafety_validate_harmful_content) runs the `detectHarmfulContent` method to validate the output of the prompt.

## Azure AI Content Safety services

Expand Down Expand Up @@ -126,7 +126,9 @@ console.log(resf)
}
```

The [def](/genaiscript/reference/scripts/context) also supports setting a `detectPromptInjection` flag to apply the detection to each file.
The [def](/genaiscript/reference/scripts/context)
and [defData](/genaiscript/reference/scripts/context)
functions supports setting a `detectPromptInjection` flag to apply the detection to each file.

```js
def("FILE", env.files, { detectPromptInjection: true })
Expand All @@ -152,14 +154,14 @@ console.log(harms)

```json
{
harmfulContentDetected: true,
categoriesAnalysis: [
"harmfulContentDetected": true,
"categoriesAnalysis": [
{
category: 'Hate',
severity: 2
"category": "Hate'",
"severity": 2
}, ...
],
chunk: 'you are a very bad person'
"chunk": "you are a very bad person"
}
```

Expand Down
1 change: 1 addition & 0 deletions docs/src/content/docs/reference/scripts/context.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,7 @@ The `defData` function also supports functions to slice the input rows and colum
- `sliceTail`, number of rows to include from the end
- `sliceSample`, number of rows to pick at random
- `distinct`, list of column names to deduplicate the data based on
- `query`, a [jq](https://jqlang.github.io/jq/) query to filter the data
```js
defData("DATA", data, {
Expand Down
16 changes: 13 additions & 3 deletions docs/src/content/docs/reference/scripts/parsers.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ The `parsers.json5` function parses the JSON5 format.
In general, parsing a JSON file as JSON5 does not cause harm, but it might be more forgiving
to syntactic errors. In addition to JSON5, [JSON repair](https://www.npmjs.com/package/jsonrepair) is applied if the initial parse fails.

- JSON5 example
- JSON5 example

```json5
{
Expand Down Expand Up @@ -272,8 +272,8 @@ const fences = parsers.fences("...")
Parses error, warning annotations in various formats
into a list of objects.

- [GitHub Actions](https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions)
- [Azure DevOps Pipeline](https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=bash#example-log-a-warning-about-a-specific-place-in-a-file)
- [GitHub Actions](https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions)
- [Azure DevOps Pipeline](https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=bash#example-log-a-warning-about-a-specific-place-in-a-file)
-

```js
Expand Down Expand Up @@ -322,6 +322,16 @@ used with `defData`.
const d = parsers.tidyData(rows, { sliceSample: 100, sort: "name" })
```
## GROQ
Apply a [GROQ](GROQEvaluate) query to a JSON object.
```js
const d = parsers.GROQ(`*[completed == true && userId == 2]{
title
}`, data)
```
## hash
Utility to hash an object, array into a string that is appropriate for hashing purposes.
Expand Down
4 changes: 4 additions & 0 deletions docs/src/content/docs/reference/scripts/schemas.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,10 @@ const schema = defSchema(...)
$`Use ${schema} for the JSON schema.`
```

## inference

Use `JSONSchema.infer` to generate a JSON schema from a JSON object.

## Validation

When a JSON/YAML payload is generated with the schema identifier,
Expand Down
3 changes: 2 additions & 1 deletion docs/src/content/docs/reference/scripts/system.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2555,7 +2555,8 @@ Plain text output
`````js wrap title="system.output_plaintext"
system({ title: "Plain text output" })
$`## Plain Text Output
Respond in plain text. Do not wrap result in Markdown code fences or XML tags.
Respond in plain text. No yapping, no markdown, no code fences, no XML tags, no string delimiters
wrapping it.
`
`````
Expand Down
Loading

0 comments on commit 3c9e098

Please sign in to comment.