Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

configurable .gitignore in search files #548

Merged
merged 5 commits into from
Jun 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions demo/genaisrc/genaiscript.d.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions docs/genaisrc/genaiscript.d.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
"dependencies": {
"@astrojs/check": "^0.7.0",
"@astrojs/starlight": "^0.24.2",
"astro": "^4.10.2",
"astro": "^4.10.3",
"typescript": "5.4.5"
},
"devDependencies": {
Expand Down
5 changes: 4 additions & 1 deletion docs/src/content/docs/reference/cli/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Runs a GenAIScript against files.

Options:
-ef, --excluded-files <string...> excluded files
-egi, --exclude-git-ignore exclude files that are ignore through the .gitignore file in the workspace root
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
-o, --out <string> output folder. Extra markdown fields for output and trace will also be generated
-rmo, --remove-out remove output folder if it exists
-ot, --out-trace <string> output file for trace
Expand Down Expand Up @@ -56,6 +57,8 @@ Run a tool on a batch of specs

Options:
-ef, --excluded-files <string...> excluded files
-egi, --exclude-git-ignore exclude files that are ignore through the
.gitignore file in the workspace root
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
-o, --out <folder> output folder. Extra markdown fields for
output and trace will also be generated
-rmo, --remove-out remove output folder if it exists
Expand Down Expand Up @@ -115,7 +118,7 @@ Options:
-td, --test-delay <string> delay between tests in seconds
--no-cache disable LLM result cache
-v, --verbose verbose output
-pv, --promptfoo-version [version] promptfoo version, default is ^0.63.2
-pv, --promptfoo-version [version] promptfoo version, default is ^0.65.0
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
-os, --out-summary <file> append output summary in file
-h, --help display help for command
```
Expand Down
32 changes: 20 additions & 12 deletions docs/src/content/docs/reference/cli/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,22 +36,30 @@ See [configuration](/genaiscript/getting-started/configuration).

Excludes the specified files from the file set.

```sh
npx genaiscript run <script> <spec> --excluded-files <excluded-files...>
```sh "--excluded-files <excluded-files...>"
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
npx genaiscript run <script> <files> --excluded-files <excluded-files...>
```

### --exclude-git-ignore

Exclude files ignored by the `.gitignore` file at the workspace root.

```sh "--exclude-git-ignore"
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
npx genaiscript run <script> <files> --exclude-git-ignore
```

### --out <file|directory>

Saves the results in a JSON file, along with markdown files of the output and the trace.

```sh
npx genaiscript run <script> <spec> --out output/results.json
```sh "--out tmp"
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
npx genaiscript run <script> <files> --out out/res.json
```

If `file` does not end with `.json`, the path is treated as a directory path.

```sh
npx genaiscript run <script> <spec> --out output
```sh "--out tmp"
pelikhan marked this conversation as resolved.
Show resolved Hide resolved
npx genaiscript run <script> <files> --out tmp
```

### --json
Expand All @@ -71,29 +79,29 @@ Populate values in the `env.vars` map that can be used when running the prompt.
Save the markdown trace to the specified file.

```sh
npx genaiscript run <script> <spec> --out-trace &lt;file&gt;
npx genaiscript run <script> <files> --out-trace &lt;file&gt;
```

In a GitHub Actions workflow, you can use this feature to save the trace as a step summary (`GITHUB_STEP_SUMMARY`):

```yaml
- name: Run GenAIScript tool on spec
run: |
genaiscript run <script> <spec> --out-trace $GITHUB_STEP_SUMMARY
genaiscript run <script> <files> --out-trace $GITHUB_STEP_SUMMARY
```

### --out-annotations &lt;file&gt;

Emit annotations in the specified file as a JSON array, JSON Lines, [SARIF](https://sarifweb.azurewebsites.net/) or a CSV file if the file ends with `.csv`.

```sh
npx genaiscript run <script> <spec> --out-annotations diags.csv
npx genaiscript run <script> <files> --out-annotations diags.csv
```

Use JSON lines (`.jsonl`) to aggregate annotations from multiple runs in a single file.

```sh
npx genaiscript run <script> <spec> --out-annotations diags.jsonl
npx genaiscript run <script> <files> --out-annotations diags.jsonl
```

### --out-data &lt;file&gt;
Expand All @@ -102,15 +110,15 @@ Emits parsed data as JSON, YAML or JSONL. If a JSON schema is specified
and availabe, the JSON validation result is also stored.

```sh
npx genaiscript run <script> <spec> --out-data data.jsonl
npx genaiscript run <script> <files> --out-data data.jsonl
```

### --out-changelogs &lt;file&gt;

Emit changelogs in the specified file as text.

```sh
npx genaiscript run <script> <spec> --out-changelogs changelogs.txt
npx genaiscript run <script> <files> --out-changelogs changelogs.txt
```

### --prompt
Expand Down
4 changes: 2 additions & 2 deletions genaisrc/genaiscript.d.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 4 additions & 5 deletions packages/cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
"@azure/identity": "^4.2.1",
"dockerode": "^4.0.2",
"pdfjs-dist": "4.3.136",
"promptfoo": "^0.63.2",
"promptfoo": "^0.65.0",
"tree-sitter-wasms": "^0.1.11",
"typescript": "5.4.5",
"web-tree-sitter": "^0.22.2",
Expand All @@ -43,7 +43,7 @@
"@types/dockerode": "^3.3.29",
"@types/fs-extra": "^11.0.4",
"@types/memorystream": "^0.3.4",
"@types/node": "^20.14.2",
"@types/node": "^20.14.5",
"@types/papaparse": "^5.3.14",
"@types/pg": "^8.11.2",
"@types/prompts": "^2.4.9",
Expand All @@ -59,8 +59,7 @@
"genaiscript-core": "*",
"get-stdin": "^9.0.0",
"glob": "^10.4.1",
"ignore": "^5.3.1",
"mammoth": "^1.7.2",
"mammoth": "^1.8.0",
"memorystream": "^0.3.1",
"node-sarif-builder": "^3.1.0",
"openai": "^4.51.0",
Expand All @@ -69,7 +68,7 @@
"prompts": "^2.4.2",
"replace-ext": "^2.0.0",
"semver": "^7.6.2",
"tsx": "^4.15.4",
"tsx": "^4.15.6",
"zx": "^8.1.2"
},
"scripts": {
Expand Down
6 changes: 5 additions & 1 deletion packages/cli/src/batch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ export async function batchScript(
specs: string[],
options: {
excludedFiles: string[]
excludeGitIgnore: boolean
out: string
outSummary: string
removeOut: boolean
Expand Down Expand Up @@ -69,6 +70,7 @@ export async function batchScript(
outSummary,
applyEdits,
excludedFiles,
excludeGitIgnore,
} = options
const outAnnotations = join(out, "annotations.jsonl")
const outData = join(out, "data.jsonl")
Expand All @@ -89,7 +91,9 @@ export async function batchScript(
if (GENAI_JS_REGEX.test(tool)) toolFiles.push(tool)
const specFiles = new Set<string>()
for (const arg of specs) {
const ffs = await host.findFiles(arg)
const ffs = await host.findFiles(arg, {
applyGitIgnore: excludeGitIgnore,
})
if (!ffs.length)
fail(`no files matching ${arg}`, FILES_NOT_FOUND_ERROR_CODE)

Expand Down
8 changes: 8 additions & 0 deletions packages/cli/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,10 @@ export async function cli() {
.description("Runs a GenAIScript against files.")
.arguments("<script> [files...]")
.option("-ef, --excluded-files <string...>", "excluded files")
.option(
"-egi, --exclude-git-ignore",
"exclude files that are ignore through the .gitignore file in the workspace root"
)
.option(
"-o, --out <string>",
"output folder. Extra markdown fields for output and trace will also be generated"
Expand Down Expand Up @@ -155,6 +159,10 @@ export async function cli() {
.description("Run a tool on a batch of specs")
.arguments("<script> [files...]")
.option("-ef, --excluded-files <string...>", "excluded files")
.option(
"-egi, --exclude-git-ignore",
"exclude files that are ignore through the .gitignore file in the workspace root"
)
.option(
"-o, --out <folder>",
"output folder. Extra markdown fields for output and trace will also be generated"
Expand Down
19 changes: 9 additions & 10 deletions packages/cli/src/nodehost.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,17 +21,18 @@ import {
UTF8Encoder,
createBundledParsers,
createFileSystem,
filterGitIgnore,
parseDefaultsFromEnv,
parseTokenFromEnv,
resolveLanguageModel,
setHost,
unique,
} from "genaiscript-core"
import { TextDecoder, TextEncoder } from "util"
import { readFile, unlink, writeFile } from "node:fs/promises"
import { ensureDir, existsSync, remove } from "fs-extra"
import { resolve, dirname } from "node:path"
import { glob } from "glob"
import ignorer from "ignore"
import { debug, error, info, warn } from "./log"
import { execa } from "execa"
import { join } from "node:path"
Expand Down Expand Up @@ -204,21 +205,19 @@ export class NodeHost implements Host {
}
async findFiles(
path: string | string[],
ignore?: string | string[]
options: {
ignore?: string | string[]
applyGitIgnore?: boolean
}
): Promise<string[]> {
const { ignore, applyGitIgnore } = options || {}
let files = await glob(path, {
nodir: true,
windowsPathsNoEscape: true,
ignore,
})
if (existsSync(".gitignore")) {
const gitignore = await readFile(".gitignore", {
encoding: "utf-8",
})
const ig = ignorer().add(gitignore)
files = ig.filter(files)
}
return files
if (applyGitIgnore) files = await filterGitIgnore(files)
return unique(files)
}
async writeFile(name: string, content: Uint8Array): Promise<void> {
await ensureDir(dirname(name))
Expand Down
8 changes: 6 additions & 2 deletions packages/cli/src/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ export async function runScript(
specs: string[],
options: {
excludedFiles: string[]
excludeGitIgnore: boolean
out: string
retry: string
retryDelay: string
Expand Down Expand Up @@ -80,6 +81,7 @@ export async function runScript(
}
) {
const excludedFiles = options.excludedFiles
const excludeGitIgnore = !!options.excludeGitIgnore
const out = options.out
const stream = !options.json && !options.yaml && !out
const skipLLM = !!options.prompt
Expand Down Expand Up @@ -134,7 +136,9 @@ export async function runScript(
for (const arg of specs) {
if (HTTPS_REGEX.test(arg)) files.add(arg)
else {
const ffs = await host.findFiles(arg)
const ffs = await host.findFiles(arg, {
applyGitIgnore: excludeGitIgnore,
})
if (!ffs.length)
fail(`no files matching ${arg}`, FILES_NOT_FOUND_ERROR_CODE)

Expand Down Expand Up @@ -234,7 +238,7 @@ ${Array.from(files)
stats: {
toolCalls: 0,
repairs: 0,
turns: 0
turns: 0,
},
})
} catch (err) {
Expand Down
Loading
Loading