transformers.js sample #581
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| const onImport = (file: string) => { | ||
| trace?.itemValue("📦 import", file) | ||
| //trace?.itemValue("📦 import", fileURLToPath(file)) | ||
| } |
There was a problem hiding this comment.
You have commented out the trace function call. This might lead to loss of important debugging information. Please ensure this is intentional. 🕵️♀️
generated by pr-review-commit
commented_code
| } | ||
| onImport(modulePath) |
There was a problem hiding this comment.
The onImport(modulePath) function call seems unnecessary as it is called before the register function where it should be used. This might lead to unexpected behavior. 🧐
generated by pr-review-commit
unnecessary_code
packages/core/src/importprompt.ts
Outdated
| const { tsImport, register } = await import("tsx/esm/api") | ||
| unregister = register({ onImport }) | ||
| const module = await tsImport(modulePath, { | ||
| parentURL, | ||
| //tsconfig: false, | ||
| onImport, | ||
| }) |
There was a problem hiding this comment.
The onImport function is no longer passed as an argument to tsImport. This might cause the import process to not behave as expected. Please ensure this is the intended behavior. 🚀
generated by pr-review-commit
missing_argument
|
The changes involve adjusting how imports are handled and traced in a TypeScript file. It appears the developer is testing a different approach to tracing by commenting out some lines and adding new ones. Here are some potential concerns:
Here is a diff with some improvements, if the original functionality needs to be maintained: diff --git a/packages/core/src/importprompt.ts b/packages/core/src/importprompt.ts
index 1d280a6..ceaadf7 100644
--- a/packages/core/src/importprompt.ts
+++ b/packages/core/src/importprompt.ts
@@ -2,7 +2,7 @@ import { assert } from "console"
import { host } from "./host"
import { logError } from "./util"
import { TraceOptions } from "./trace"
-import { pathToFileURL } from "url"
+import { fileURLToPath, pathToFileURL } from "url"
function resolveGlobal(): any {
if (typeof window !== "undefined")
@@ -46,16 +46,14 @@ export async function importPrompt(
import.meta.url ??
pathToFileURL(__filename ?? host.projectFolder()).toString()
trace?.itemValue(`import`, `${modulePath}, parent: ${parentURL}`)
const onImport = (file: string) => {
trace?.itemValue("📦 import", fileURLToPath(file))
}
onImport(modulePath)
const { tsImport, register } = await import("tsx/esm/api")
unregister = register({ onImport })
const module = await tsImport(modulePath, {
parentURL,
onImport,
})
const main = module.default
if (typeof main === "function") await main(ctx0)Please verify if the changes made are as per the required functionality.
|
| @@ -73,6 +73,7 @@ class ModelManager implements ModelService { | |||
| if (provider === MODEL_PROVIDER_OLLAMA) { | |||
| if (this.pulled.includes(modelid)) return { ok: true } | |||
|
|
|||
| logVerbose(`ollama: pulling ${modelid}...`) | |||
There was a problem hiding this comment.
The logVerbose function is called without any context. It might be better to provide more information about the current state of the application. 🤔
generated by pr-review-commit
log_without_context
| } | ||
| onImport(modulePath) |
There was a problem hiding this comment.
The onImport function is called with modulePath but the result is not used or stored. This could lead to unexpected behavior. 😕
generated by pr-review-commit
unused_function_call
| if (line) trace.log(line) | ||
| if (line) { | ||
| trace.log(line) | ||
| logVerbose(line) |
There was a problem hiding this comment.
The logVerbose function is called with the same argument as trace.log. This could lead to duplicate log entries. 🙃
generated by pr-review-commit
duplicate_logging
|
/genai-describe |
| if (line) trace.log(line) | ||
| if (line) { | ||
| trace.log(line) | ||
| logVerbose(line) |
There was a problem hiding this comment.
The logVerbose function is called inside the trace.log condition. This could lead to excessive logging and performance issues. Consider moving it outside the condition or adding additional checks. 😮
generated by pr-review-commit
log_in_trace
| @@ -0,0 +1,80 @@ | |||
| --- | |||
| title: Transformer.js | |||
There was a problem hiding this comment.
The filename Transformer.js should be transformers.js to match the library's actual name.
generated by pr-docs-review-commit
filename_incorrect
| import sampleSrc from "../../../../../packages/sample/genaisrc/summary-with-transformers.genai?raw" | ||
|
|
||
|
|
||
| HuggingFace [Transformers.js](https://huggingface.co/docs/transformers.js/index) is a JavaScript library |
There was a problem hiding this comment.
The URL provided for Transformers.js documentation is incorrect; it should be https://huggingface.co/transformers/.
generated by pr-docs-review-commit
incorrect_url
| that lets you run pretrained models locally on your machine. The library uses [onnxruntime](https://onnxruntime.ai/) | ||
| to leverage the CPU/GPU capabilities of your hardware. | ||
|
|
||
| In this guide, we will show how to create [summaries](https://huggingface.co/tasks/summarization) using the [Transformers.js](https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.SummarizationPipeline) library. |
There was a problem hiding this comment.
The URL provided for Transformers.js summarization pipeline is incorrect; it should be https://huggingface.co/transformers/main_classes/pipelines.html#transformers.SummarizationPipeline.
generated by pr-docs-review-commit
incorrect_url
|
|
||
| :::tip | ||
|
|
||
| Transformers.js has an extensive list of tasks available. This guide will only cover one but checkout their [documentation](https://huggingface.co/docs/transformers.js/pipelines#tasks) |
There was a problem hiding this comment.
The URL provided for Transformers.js tasks documentation is incorrect; it should be https://huggingface.co/transformers/task_summary.html.
generated by pr-docs-review-commit
incorrect_url
|
|
||
| ## Installation | ||
|
|
||
| Following the [installation instructions](https://huggingface.co/docs/transformers.js/installation), |
There was a problem hiding this comment.
The URL provided for Transformers.js installation instructions is incorrect; it should be https://huggingface.co/transformers/installation.html.
generated by pr-docs-review-commit
incorrect_url
| ## Installation | ||
|
|
||
| Following the [installation instructions](https://huggingface.co/docs/transformers.js/installation), | ||
| we add the [@xenova/transformers](https://www.npmjs.com/package/@xenova/transformers) to the current project. |
There was a problem hiding this comment.
The package name @xenova/transformers is incorrect; it should be @huggingface/transformers.
generated by pr-docs-review-commit
incorrect_package_name
| we add the [@xenova/transformers](https://www.npmjs.com/package/@xenova/transformers) to the current project. | ||
|
|
||
| ```bash | ||
| npm install @xenova/transformers |
There was a problem hiding this comment.
The package name @xenova/transformers is incorrect in the installation command; it should be npm install @huggingface/transformers.
generated by pr-docs-review-commit
incorrect_package_name
| You can also install this library globally to be able to use on any project | ||
|
|
||
| ```bash "-g" | ||
| npm install -g @xenova/transformers |
There was a problem hiding this comment.
The package name @xenova/transformers is incorrect in the global installation command; it should be npm install -g @huggingface/transformers.
generated by pr-docs-review-commit
incorrect_package_name
| You can specify a model name or let the library pick the latest and greatest. | ||
|
|
||
| ```js | ||
| import { pipeline } from "@xenova/transformers" |
There was a problem hiding this comment.
The import statement uses an incorrect package name @xenova/transformers; it should be @huggingface/transformers.
generated by pr-docs-review-commit
incorrect_package_import
|
|
||
| :::note[Migrate your script to `.mjs`] | ||
|
|
||
| To use the `Transformers.js` library, you need to use the `.mjs` extension for your script (or `.mts` for TypeScript support). |
There was a problem hiding this comment.
The advice to migrate scripts to .mjs is incorrect and not necessary for using the Transformers.js library.
generated by pr-docs-review-commit
incorrect_extension_advice
| The example below generates a summary of each input file | ||
| before letting the model generate a full summary. | ||
|
|
||
| <Code |
There was a problem hiding this comment.
The file reference transformers.genai.mjs is incorrect; it should match the actual file name used in the project.
generated by pr-docs-review-commit
incorrect_file_reference
| @@ -0,0 +1,80 @@ | |||
| --- | |||
| title: Transformer.js | |||
There was a problem hiding this comment.
The title field in the metadata section should be 'Transformers.js' instead of 'Transformer.js'.
generated by pr-docs-review-commit
metadata_field_incorrect
| You can specify a model name or let the library pick the latest and greatest. | ||
|
|
||
| ```js | ||
| import { pipeline } from "@xenova/transformers" |
There was a problem hiding this comment.
The import statement should use '@huggingface/transformers' instead of '@xenova/transformers'.
generated by pr-docs-review-commit
incorrect_import
| if (line) trace.log(line) | ||
| if (line) { | ||
| trace.log(line) | ||
| logVerbose(line) |
There was a problem hiding this comment.
The logVerbose function is called right after trace.log with the same argument. This could lead to duplicate logs. Consider removing one of them to avoid unnecessary duplication. 🔄
generated by pr-review-commit
log_duplication
| @@ -36,7 +36,7 @@ See [configuration](/genaiscript/getting-started/configuration). | |||
|
|
|||
| `run` takes one or more [glob](https://en.wikipedia.org/wiki/Glob_(programming)) patterns to match files in the workspace. | |||
|
|
|||
| ```npx sh | |||
| ```bash sh | |||
There was a problem hiding this comment.
The code fence should be just bash instead of bash sh.
generated by pr-docs-review-commit
incorrect_code_fence
/docs/src/content/docs/guidesnamedtransformers-js.mdx, which provides a guide on using the Transformers.js library. This new file contains installation instructions, how to invoke the pipeline, and other useful information. 📚summary-with-transformers.genai.mjsin the/packages/sample/genaisrcdirectory. The script utilizes the Transformers.js library for creating summaries of content. 📜nodehost.tsallows verbose logging for model pulling from Ollama, improving model download tracing. 🎛️importPromptinimportprompt.tswas refactored. Instead of tracing on every import, tracing now only occurs once per module. 🔄runpromptcontext.ts, verbose logging was introduced in thecreateChatTurnGenerationContextfunction. Now, tracing will also output verbose logs for better visibility. ✨package.jsonfile in the/packages/sampledirectory was also updated with a new dev dependency@xenova/transformersto support the use of Transformers.js library. 🔧Please note that there were many minor changes regarding import statements across various files, and those are not included in this summary as per the instructions.