Custom fetch wrapper

[sbarbett]

I added a custom wrapper for fetch to control the headersTimeout value. The default value is 5 minutes, and slower instances of ollama can take longer than that, so the fetch fails. The following environment variable allows you to up it (the default is still 5 min).

services:
  web:
    environment:
      - INFERENCE_FETCH_TIMEOUT_SEC=3600  # 1 hour fetch timeout

There's more info on this in issue 628. I built this locally and tested it against my ollama instance. With the default timeout:

Nov 14 02:35:33 llama ollama[3733]: [GIN] 2024/11/14 - 02:35:33 | 200 |          5m0s |        10.0.0.2 | POST     "/api/chat"

Hard stop at 5 min. The inference job fails.

workers-1      | 2024-11-14T02:35:33.346Z error: [inference][10] inference job failed: TypeError: fetch failed
workers-1      | TypeError: fetch failed
workers-1      |     at node:internal/deps/undici/undici:13392:13
workers-1      |     at async post (/app/node_modules/ollama/dist/shared/ollama.9c897541.cjs:114:20)
workers-1      |     at async Ollama.processStreamableRequest (/app/node_modules/ollama/dist/shared/ollama.9c897541.cjs:232:25)
workers-1      |     at async OllamaInferenceClient.runModel (/app/packages/shared/inference.ts:2:3386)
workers-1      |     at async OllamaInferenceClient.inferFromText (/app/packages/shared/inference.ts:2:4136)
workers-1      |     at async inferTagsFromText (/app/apps/workers/openaiWorker.ts:6:3135)
workers-1      |     at async inferTags (/app/apps/workers/openaiWorker.ts:6:3370)
workers-1      |     at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6863)
workers-1      |     at async Runner.runOnce (/app/node_modules/liteque/dist/runner.js:2:2578)

Here is the same job after upping the fetch timeout.

Nov 14 02:52:23 llama ollama[3733]: [GIN] 2024/11/14 - 02:52:23 | 200 |         7m31s |        10.0.0.2 | POST     "/api/chat"

workers-1      | 2024-11-14T02:55:55.614Z info: [search][13] Completed successfully

---

One other small tweak: The docker-compose.dev.yml file needed the env file declared under the worker service.

---

Thanks for making this useful app.