Add pqueue to API #725
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1037,7 +1037,7 @@ The following npm packages may be included in this product: | |
| - @types/[email protected] | ||
| - @types/[email protected] | ||
| - @types/[email protected] | ||
| - @types/node@22.7.0 | ||
| - @types/[email protected] | ||
| - @types/[email protected] | ||
|
|
||
|
|
@@ -1099,7 +1099,7 @@ MIT License | |
|
|
||
| The following npm package may be included in this product: | ||
|
|
||
| - [email protected].7 | ||
|
|
||
| This package contains the following license and notice below: | ||
|
|
||
|
|
@@ -4188,9 +4188,9 @@ The following npm packages may be included in this product: | |
| - @tokenizer/[email protected] | ||
| - [email protected] | ||
| - [email protected] | ||
| - [email protected].7 | ||
| - [email protected].7 | ||
| - [email protected].7 | ||
| - [email protected] | ||
| - [email protected] | ||
| - [email protected] | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| --- | ||
| title: Concurrency | ||
| description: How to run multiple prompts concurrently | ||
| sidebar: | ||
| order: 50 | ||
| --- | ||
|
|
||
| When working with a GenAI, your program will likely be idling waiting for tokens to come back from the LLM. | ||
|
|
||
| ## await and async | ||
|
|
||
| JavaScript has a wonderful support for non-blocking asynchronous APIs using [async functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function). | ||
|
|
||
| ```js | ||
| // takes a while | ||
| async function workM() { ... } | ||
|
|
||
| // let other threads work while this function is running | ||
| await work() | ||
| ``` | ||
|
|
||
| This feature is leveraged in [inline prompts](/genaiscript/reference/scripts/inline-prompts) to wait for a LLM result or run multiple queries concurrently. | ||
|
|
||
| ## Serial vs concurrent execution | ||
|
|
||
| In this example, we run each LLM queries 'serially' using `await`: | ||
|
|
||
| ```js | ||
| const poem = await prompt`write a poem` | ||
| const essay = await prompt`write an essay` | ||
| ``` | ||
|
|
||
| However, we can run all queries 'concurrently' to speed things up: | ||
|
|
||
| ```js | ||
| const [poem, essay] = await Promise.all( | ||
| prompt`write a poem`, | ||
| prompt`write an essay` | ||
| ) | ||
|
There was a problem hiding this comment. Missing function calls for 'prompt' template literals.
There was a problem hiding this comment. Missing function wrappers for 'prompt' calls within 'Promise.all'.
|
||
| ``` | ||
|
|
||
| This works but it may become problematic if you have a lot of entries as you will create a lot of requests concurrently and probably hit some rate limiting boundaries. | ||
| Note that GenAIScript automatically limits the number of concurrent requests to a single model to prevent this scenario. | ||
|
|
||
| ## Promise queue | ||
|
|
||
| The promise queue provides a way to run promises concurrently with a guaranteed concurrency limit, how many are allowed to run at the same time. | ||
| The difference with `Promise.all` is that you wrap each promise in a function. | ||
|
|
||
| ```js | ||
| const queue = host.promiseQueue(3) | ||
| const res = await queue.all( | ||
| () => prompt`write a poem` | ||
| () => prompt`write an essay` | ||
| ) | ||
|
There was a problem hiding this comment. Missing function calls for 'prompt' template literals.
|
||
| ``` | ||
|
|
||
| Use the `mapAll` function to iterate over an array. | ||
|
|
||
| ```js | ||
| const queue = host.promiseQueue(3) | ||
| const summaries = await queue.mapAll( | ||
| env.files, | ||
| (file) => prompt`Summarize ${file}` | ||
| ) | ||
|
There was a problem hiding this comment. Missing function calls for 'prompt' template literals.
|
||
| ``` | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -68,11 +68,11 @@ | |
|
|
||
| `prompt` and `runPrompt` are async functions that can be used in a loop to run multiple prompts concurrently. | ||
|
|
||
| ```js | ||
|
There was a problem hiding this comment. Incorrect usage of 'Promise.all'; missing function wrapper and incorrect argument passing.
|
||
| await Promise.all(env.files, file => prompt`Summarize the ${file}`) | ||
| ``` | ||
|
|
||
| Internally, GenAIScript applies a concurrent limit of 8 per model by default. You can change this limit using the `modelConcurrency` option. | ||
|
There was a problem hiding this comment. The default model concurrency limit has changed from 5 to 8.
|
||
|
|
||
| ```js "modelConcurrency" | ||
| script({ | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -764,10 +764,10 @@ | |
| "@nodelib/fs.scandir" "2.1.5" | ||
| fastq "^1.6.0" | ||
|
|
||
| "@oslojs/encoding@^1.0.0": | ||
| version "1.1.0" | ||
| resolved "https://registry.yarnpkg.com/@oslojs/encoding/-/encoding-1.1.0.tgz#55f3d9a597430a01f2a5ef63c6b42f769f9ce34e" | ||
| integrity sha512-70wQhgYmndg4GCPxPPxPGevRKqTIJ2Nh4OkiMWmDAVYsTQ+Ta7Sq+rPevXyXGdzr30/qZBnyOalCszoMxlyldQ== | ||
|
|
||
| "@pagefind/[email protected]": | ||
| version "1.1.1" | ||
|
|
@@ -800,9 +800,9 @@ | |
| integrity sha512-b7/qPqgIl+lMzkQ8fJt51SfguB396xbIIR+VZ3YrL2tLuyifDJ1wL5mEm+ddmHxJ2Fki340paPcDan9en5OmAw== | ||
|
|
||
| "@rollup/pluginutils@^5.1.0": | ||
| version "5.1.2" | ||
| resolved "https://registry.yarnpkg.com/@rollup/pluginutils/-/pluginutils-5.1.2.tgz#d3bc9f0fea4fd4086aaac6aa102f3fa587ce8bd9" | ||
| integrity sha512-/FIdS3PyZ39bjZlwqFnWqCOVnW7o963LtKMwQOD0NhQqw22gSr2YY1afu3FxRip4ZCZNsD5jq6Aaz6QV3D/Njw== | ||
| dependencies: | ||
| "@types/estree" "^1.0.0" | ||
| estree-walker "^2.0.2" | ||
|
|
@@ -1046,9 +1046,9 @@ | |
| "@types/unist" "*" | ||
|
|
||
| "@types/node@*", "@types/node@>=20": | ||
| version "22.7.0" | ||
| resolved "https://registry.yarnpkg.com/@types/node/-/node-22.7.0.tgz#670aa1874bc836863e5c116f9f2c32416ff27e1f" | ||
| integrity sha512-MOdOibwBs6KW1vfqz2uKMlxq5xAfAZ98SZjO8e3XnAbFnTJtAspqhWk7hrdSAs9/Y14ZWMiy7/MxMUzAOadYEw== | ||
| dependencies: | ||
| undici-types "~6.19.2" | ||
|
|
||
|
|
@@ -1267,9 +1267,9 @@ [email protected]: | |
| ultrahtml "^1.5.3" | ||
|
|
||
| astro@^4.15.8: | ||
| version "4.15.9" | ||
| resolved "https://registry.yarnpkg.com/astro/-/astro-4.15.9.tgz#b7835126a53296f6a5b11352f3e1e31197f8640e" | ||
| integrity sha512-51oXq9qrZ5OPWYmEXt1kGrvWmVeWsx28SgBTzi2XW6iwcnW/wC5ONm6ol6qBGSCF93tQvZplXvuzpaw1injECA== | ||
| dependencies: | ||
| "@astrojs/compiler" "^2.10.3" | ||
| "@astrojs/internal-helpers" "0.4.1" | ||
|
|
@@ -1278,7 +1278,7 @@ astro@^4.15.8: | |
| "@babel/core" "^7.25.2" | ||
| "@babel/plugin-transform-react-jsx" "^7.25.2" | ||
| "@babel/types" "^7.25.6" | ||
| "@oslojs/encoding" "^1.0.0" | ||
| "@rollup/pluginutils" "^5.1.0" | ||
| "@types/babel__core" "^7.20.5" | ||
| "@types/cookie" "^0.6.0" | ||
|
|
@@ -1400,12 +1400,12 @@ braces@^3.0.3, braces@~3.0.2: | |
| fill-range "^7.1.1" | ||
|
|
||
| browserslist@^4.23.1: | ||
| version "4.24.0" | ||
| resolved "https://registry.yarnpkg.com/browserslist/-/browserslist-4.24.0.tgz#a1325fe4bc80b64fda169629fc01b3d6cecd38d4" | ||
| integrity sha512-Rmb62sR1Zpjql25eSanFGEhAxcFwfA1K0GuQcLoaJBAcENegrQut3hYdhXFF1obQfiDyqIW/cLM5HSJ/9k884A== | ||
| dependencies: | ||
| caniuse-lite "^1.0.30001663" | ||
| electron-to-chromium "^1.5.28" | ||
| node-releases "^2.0.18" | ||
| update-browserslist-db "^1.1.0" | ||
|
|
||
|
|
@@ -1414,7 +1414,7 @@ camelcase@^7.0.1: | |
| resolved "https://registry.yarnpkg.com/camelcase/-/camelcase-7.0.1.tgz#f02e50af9fd7782bc8b88a3558c32fd3a388f048" | ||
| integrity sha512-xlx1yCK2Oc1APsPXDL2LdlNP6+uu8OCDdhOBSVT279M/S+y75O30C2VuD8T2ogdePBBl7PfPF4504tnLgX3zfw== | ||
|
|
||
| caniuse-lite@^1.0.30001663: | ||
| version "1.0.30001663" | ||
| resolved "https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30001663.tgz#1529a723505e429fdfd49532e9fc42273ba7fed7" | ||
| integrity sha512-o9C3X27GLKbLeTYZ6HBOLU1tsAcBZsLis28wrVzddShCS16RujjHp9GDHKZqrB3meE0YjhawvMFsGb/igqiPzA== | ||
|
|
@@ -1652,10 +1652,10 @@ eastasianwidth@^0.2.0: | |
| resolved "https://registry.yarnpkg.com/eastasianwidth/-/eastasianwidth-0.2.0.tgz#696ce2ec0aa0e6ea93a397ffcf24aa7840c827cb" | ||
| integrity sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA== | ||
|
|
||
| electron-to-chromium@^1.5.28: | ||
| version "1.5.28" | ||
| resolved "https://registry.yarnpkg.com/electron-to-chromium/-/electron-to-chromium-1.5.28.tgz#aee074e202c6ee8a0030a9c2ef0b3fe9f967d576" | ||
| integrity sha512-VufdJl+rzaKZoYVUijN13QcXVF5dWPZANeFTLNy+OSpHdDL5ynXTF35+60RSBbaQYB1ae723lQXHCrf4pyLsMw== | ||
|
|
||
| emmet@^2.4.3: | ||
| version "2.4.11" | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,15 +19,15 @@ | |
| export async function codeQuery(files: string, query: string) { | ||
| // Find files matching the given pattern, respecting .gitignore rules. | ||
| const ffs = await host.findFiles(files, { | ||
| applyGitIgnore: true, // Ensure .gitignore rules are applied when finding files | ||
|
There was a problem hiding this comment. The comment change on this line does not provide any additional information. It is recommended to keep comments concise and informative. 📝
|
||
| }) | ||
| const captures: any[] = [] // Array to store query result captures | ||
|
|
||
| // Iterate through each matched file | ||
| for (const filename of ffs) { | ||
| logVerbose(`scanning ${filename}`) // Log the current file being scanned | ||
|
|
||
| // Initialize a WorkspaceFile object with filename and undefined content | ||
| const f: WorkspaceFile = { filename, content: undefined } | ||
|
|
||
| // Resolve and load the file content | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Incorrect function name; should be 'workM' instead of 'work'.