feat: make Chromium install configurable

[Patrick-Erichsen]

Description

Addresses user feedback from #2166.

By default, we no longer download and install Chromium for our docs service.

Instead, this PR introduces a new experimental configuration, useChromiumForDocsCrawling, that is used to control installation.

If this config is set to true, and Chromium is not installed, we will attempt to install Chromium. If a user attempts to crawl a doc site using the default crawler and fails, we give them the option to install Chromium, set this config to true, and attemp to re-crawl the docs using Chromium.

Checklist

• The base branch of this PR is dev, rather than main
• The relevant docs, if any, have been updated or created

Testing

Installing Chromium

• rm -rf ~/.continue/.utils to remove any existing Chromium installation
• Add to config.json: "experimental": {"useChromiumForDocsCrawling": true}
• Open dev console, verify installation
• You should see logs that look like this:

[Extension Host] Installing Chromium (at console.<anonymous> (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:144:47723))
[Extension Host] [PCR] Puppeteer version: 23.1.1 (at console.<anonymous> (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:144:47723))
console.ts:137 [Extension Host] [PCR] Browser platform: mac (at console.<anonymous> (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:144:47723))
log.ts:419  INFO Extension host (LocalProcess pid: 61770) is responsive.
console.ts:137 [Extension Host] [PCR] Chromium revision: 1351534 (at console.<anonymous> (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:144:47723))
console.ts:137 [Extension Host] [PCR] Not found local chromium (at console.<anonymous> (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:144:47723))
console.ts:137 [Extension Host] [PCR] Head request: https://storage.googleapis.com/chromium-browser-snapshots/Mac/1351534/chrome-mac.zip - true (186ms) (at console.<anonymous> (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:144:47723))
console.ts:137 [Extension Host] [PCR] Downloading from: https://storage.googleapis.com ... (at console.<anonymous> (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:144:47723))

Disabling Chromium

• Add to config.json: "experimental": {"useChromiumForDocsCrawling": false}
• Add a new doc site to config.json: "docs": [{"startUrl": "[https://amplified.dev/","title](https://amplified.dev/%22,%22title)": "Amplified.dev"}]
• Open dev console, verify that the docs are crawled using Cheerio
• You should see logs that look like this:

[Extension Host] Indexing new doc: https://amplified.dev/ (at console.<anonymous> (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:144:47723))
console.ts:137 [Extension Host] [CheerioCrawler] Starting crawl from: https://amplified.dev/ - Max Depth: 3 (at console.<anonymous> (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:144:47723))

Attempt crawl using Cheerio, error, install Chromium, retry

• rm -rf ~/.continue/.utils to remove any existing Chromium installation
• Remove useChromiumForDocsCrawlingfrom config.json
• Update crawlPage() in CheerioCrawler to emulate an error being thrown while crawling, e.g. add a throw new Error("") to the top of the try/catch
• Attempt to add a new doc, e.g. https://amplified.dev/
• Confirm you get a toast asking you to install Chromium, accept
• Confirm you get a successful install toast
• Confirm you get a successful index toast