Connect hosted LLM #350

dimits-ts · 2024-12-10T11:14:25Z

Fixes #305

Allow open-source LLMs hosted on any server to be called as Gemini would.

Changes:

Abstracted LLM API type
- Created an interface to use any kind of API key
- Exported interface for LLM responses regardless of API
- API validity checks can now be extended for any kind of API
Added Ollama server support
- Supports most open source models (is not limited to LLaMa)
- Added frontend section to connect to Ollama server through HTTP URL
- Added sample Ollama server in the form of a docker container for testing
Included unit tests for most new functions

TODOs:

LLaMa server may need some kind of authentication
While Ollama supports almost all popular open-source LLMs, this patch only allows the use of llama3.2.
- We can include a text area or dropdown menu in Settings with all available models. The issue is that doing so would necessitate pulling and deploying all such models on the remote Ollama server first.
API key check only checks whether the key is not empty, not if it's valid. This also goes for Gemini keys.
Right now all moderators are either Gemini or LLaMa. Future versions may include mixing different kinds of models (see LLaMa integration design doc)

we can remove the tests later but they are essential for now

the ollama api necessitates keeping the model state active, so we need a class instead of a set of functions

on second thought it's too much trouble for what it's worth

Build automation

…erate-lab into build_automation

Standardize docker container creation for local/remote LLM hosting

mkbehr · 2024-12-13T18:49:12Z

frontend/api/gemini.api.ts

We probably don't want to duplicate the API libraries between the frontend and functions directories - too easy for the copies to get out of sync. I'm still getting familiar with this codebase, so I'm not sure of the best way to organize this. @cjqian or @vivtsai, any thoughts?

Agreed - I think only having API functions in one place would be best. @dimits-ts is there a reason to make LLM calls from the frontend? Our initial assumption when setting up the codebase was that backend calls would suffice/be preferred.

At quick glance it looks like the frontend/api is no longer used and could be deleted (since the actual ollama call uses functions/src/api)?

You are both correct, I have no idea how the files ended up getting duplicated. The actual API calls are made to the functions module, the frontend/api module is unused. I am removing it, sorry for the confusion.

mkbehr · 2024-12-13T18:52:30Z

functions/src/api/llama.api.ts

I'd recommend calling this ollama.api.ts, and consistently referring to this by "ollama" instead of "llama" (e.g. OLLAMA_CUSTOM_URL instead of LLAMA_CUSTOM_URL). Like you mention in the description, ollama can host plenty of non-llama models - but there are also other methods of hosting llama models that won't work with the ollama API.

You are correct, this was a holdout from when I wasn't sure which backend library was going to be used for hosting the OSS LLMs. I will correct it.

I'm still seeing some references to "llama", mostly the things defined in utils/src/experimenter.ts: llamaApiKey, LlamaServerConfig, LLAMA_CUSTOM_URL.

Ah right, I overlooked these. There should be no references to llama anywhere now. Apologies.

mkbehr · 2024-12-13T19:00:53Z

functions/src/api/llama.api.ts

+ * and is managed through the `ollama` framework (https://github.com/ollama/ollama).
+ * Example docker instance hosting an ollama server: https://github.com/dimits-ts/deliberate-lab-utils/tree/master/llm_server
+ * 
+ * Note: there already exists a client library for JavaScript, but not for Typescript.


I'm working on adding some more API support, and I'm planning to support Ollama either via the openai library and Ollama's openai API compatibility, or via multi-llm-ts. This looks good in the meantime, just a heads up, and I wouldn't recommend putting a lot more work into polishing this.

We may need to change the name in the future, to denote that this functionality is purely for local ollama support in that case? Admittedly, I'm not familiar with ollama's openai integration.

The openai API will work fine wherever ollama is hosted. That method won't touch openai's servers or anything, it's just a client library that we don't need to maintain ourselves.

Final touches

dimits-ts · 2024-12-17T12:28:35Z

New updates:

The user can now select which OSS model will be used through the Settings tab
Disambiguated "ollama server" from "llama" both on source files and HTML
Added some HTML text and links in the "Ollama Server" section for ease-of-use

I think this is enough for this PR feature-wise. The other issues outlined in the TODO section above may need extensive changes, which may impact how we internally handle agents and configurations in general.

P.S. I did encounter a bug where the exported index.ts files wouldn't get updated for some reason (which is one of the reasons these patches were delayed). I can't seem to be able to replicate this now, however. If you encounter any issues with outdated types, give me a heads-up.

cjqian · 2024-12-17T20:55:50Z

frontend/api/gemini.api.ts

+  }
+
+  // Log the response
+  console.log(response);


Consider removing these console messages (here and elsewhere) or adding more informative statements.

I removed most from the functions module and my own code in chat_triggers, but left the warnings and errors as console.warn and console.error respectively. I didn't touch the logs for the rest of the codebase, let me know if I should.

cjqian · 2024-12-18T22:41:18Z

frontend/api/llama.api.test.ts

@@ -0,0 +1,19 @@
+import { OllamaChat } from "./llama.api";


Amazing, thank you for adding tests!

cjqian · 2024-12-18T22:43:17Z

frontend/api/llama.api.ts

+ */
+type IncomingMessage = {
+    model: string,
+    created_at: Date,


Nit: Consider renaming to dateCreated for consistency with the rest of the codebase

Upon inspection, this type didn't end up being used anywhere, since we are only extracting the LLM's response. I removed it for now.

cjqian · 2024-12-18T22:44:18Z

frontend/api/llama.api.ts

+    model: string,
+    created_at: Date,
+    message: LlmMessage,
+    done_reason: string,


Nit: For this variable and others, rename using mixedCase (e.g. doneReason, modelType) instead of done_reason, model_type for consistency.

This was an outdated version of the file which I mistakenly left committed (see mike's comment above). Sorry for the confusion!

That said, I had indeed missed some, fixed!

cjqian · 2024-12-18T22:45:09Z

frontend/src/components/stages/chat_editor.ts

      apiCheck = html`
        <div class="warning">
-          <b>Note:</b> In order for LLM calls to work, you must add your Gemini
-          API key under Settings.
+          <b>Note:</b> In order for LLM calls to work, you must add your Gemini API key 


Let's make this more generic: "you must add an API key or server configuration under Experimenter Settings." (Here and in chat_panel.ts)

…ages to appropriate console calls

mkbehr · 2024-12-20T17:18:04Z

.vscode/settings.json

This probably shouldn't go in the repository.

Fixed and added to gitignore

mkbehr · 2024-12-20T17:27:27Z

functions/src/api/llama.api.ts

I'm still seeing some references to "llama", mostly the things defined in utils/src/experimenter.ts: llamaApiKey, LlamaServerConfig, LLAMA_CUSTOM_URL.

mkbehr · 2024-12-20T17:34:53Z

functions/src/api/llama.api.ts

+ * and is managed through the `ollama` framework (https://github.com/ollama/ollama).
+ * Example docker instance hosting an ollama server: https://github.com/dimits-ts/deliberate-lab-utils/tree/master/llm_server
+ * 
+ * Note: there already exists a client library for JavaScript, but not for Typescript.


The openai API will work fine wherever ollama is hosted. That method won't touch openai's servers or anything, it's just a client library that we don't need to maintain ourselves.

vivtsai · 2024-12-20T21:23:20Z

utils/src/experimenter.ts

  id: string;
-  apiKeys: APIKeyConfig;


@dimits-ts @mkbehr @cjqian The original idea here was to have APIKeyConfig contain all API key info (as ExperimenterData may eventually contain non-API key settings). What do you think about keeping APIKeyConfig and nesting all api settings under there? E.g.,

export interface ExperimenterData { id: string; email: string; // added recently in `main` branch commit apiConfig: apiKeyConfig; } export interface APIKeyConfig { geminiApiKey: string; llamaApiKey: LlamaServerConfig; activeApiKeyType: ApiKeyType; }

(we also may want geminiApiKey to point to a config, but I'm fine with waiting until we have something else to store there)

I agree that this seems like a clearer API. I refactored the types like your example.

vivtsai

Hi @dimits-ts - thanks for working on this! I left a comment about structuring ExperimenterData / API key configs.

cjqian · 2024-12-20T22:28:38Z

I took my final pass and pushed a small cosmetic change. I've also verified that Gemini keys continue to work following this change.

Dimitris, this looks phenomenal! Thank you for all of your hard work on this and this incredible contribution.

Leaving to @vivtsai to merge when she's done with her final pass.

dimits-ts · 2024-12-23T10:27:45Z

I have refactored the types as @vivtsai recommended, and took the opportunity to make the code in experimenter_data_editor.ts more DRY-like. The changes seem to be compatible with the new changes in the upstream.

dimits-ts and others added 30 commits October 31, 2024 15:42

sketch out code for llama server communication (not tested!)

2872c58

include configurations for tests

4ec0639

we can remove the tests later but they are essential for now

conform with project style

1f924c2

create first tests

6a43687

create test configuration

1209b10

add error detection

252b7b5

print response after integration test

cb9cd5c

refactor chat functions into class

59ad59e

the ollama api necessitates keeping the model state active, so we need a class instead of a set of functions

successfully connect class to server and support multiple messages

77c3a4f

make test more consistent

0a27cf4

add documentation

ce50224

Merge branch 'PAIR-code:main' into llama-support

f76c543

remove extra package configs

7395075

add deploy scripts (can be removed before merge)

0aa0ae3

Merge branch 'PAIR-code:main' into llama-support

a739909

solve dependency issues

435be72

rename deploy scripts to be clearer

f8c8198

remove unified build script

7bc134c

on second thought it's too much trouble for what it's worth

change directory name

de1add4

Merge pull request #1 from dimits-ts/build_automation

8aa4172

Build automation

fix local installation

c0cbe48

add preliminary llama server install and deploy scriptS

7a750dc

Merge branch 'build_automation' of https://github.com/dimits-ts/delib…

7c33c1c

…erate-lab into build_automation

fix jest/babel dependency

7f1d2c7

create deployed server dockerfile

bbb795e

revert ollama model pull script

648f05a

Merge pull request #2 from dimits-ts/build_automation

c3daa0d

Standardize docker container creation for local/remote LLM hosting

Merge branch 'PAIR-code:main' into main

8fccd6d

build UI elements for llama server settings

92c4c4c

include function to thoroughly check apikey details

0ed9777

mkbehr reviewed Dec 13, 2024

View reviewed changes

dimits-ts and others added 4 commits December 17, 2024 14:07

debug server settings

57b4afd

rename llama API file to 'ollama'

4aceb7a

pop outward html link to new window

e81aee1

Merge pull request #3 from dimits-ts/choose-ollama-llm

b526ee7

Final touches

remove resolved TODOs and standardize console.logs

71801e9

cjqian requested changes Dec 18, 2024

View reviewed changes

dimits-ts added 7 commits December 19, 2024 10:38

remove accidentally duplicated source files

be3a28f

Remove console logging from functions, and direct error and warn mess…

62d9333

…ages to appropriate console calls

convert local variables to camelCase

6bbbc44

make html messages about missing api keys more generic

2cffa35

remove unused type from ollama api

309a9fa

Merge upstream

553b35c

update function name from "mediator" to "agent"

8da7264

mkbehr reviewed Dec 20, 2024

View reviewed changes

vivtsai reviewed Dec 20, 2024

View reviewed changes

vivtsai requested changes Dec 20, 2024

View reviewed changes

mkbehr mentioned this pull request Dec 20, 2024

Add basic OpenAI API text completion support. #381

Open

Add a few cosmetic changes for consistency.

e40c6cc

dimits-ts added 8 commits December 23, 2024 10:44

Rename existing llama references to "ollama"

d05267f

update ExperimenterData type structure

e345070

refactor oldData updates on Settings page

1412a56

Merge branch 'main' of https://github.com/dimits-ts/deliberate-lab

52704f1

Merge remote-tracking branch 'upstream/main'

26875fe

merge upstream

6d0a9b8

fix ollama settings bug

d83d286

add vscode settings to gitignore

fd4511a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connect hosted LLM #350

Connect hosted LLM #350

dimits-ts commented Dec 10, 2024

mkbehr Dec 13, 2024

vivtsai Dec 18, 2024

vivtsai Dec 18, 2024

dimits-ts Dec 19, 2024

mkbehr Dec 13, 2024

dimits-ts Dec 17, 2024

mkbehr Dec 20, 2024

dimits-ts Dec 23, 2024

mkbehr Dec 13, 2024

dimits-ts Dec 17, 2024 •

edited

Loading

mkbehr Dec 20, 2024

dimits-ts commented Dec 17, 2024

cjqian Dec 17, 2024

dimits-ts Dec 19, 2024

cjqian Dec 18, 2024

cjqian Dec 18, 2024

dimits-ts Dec 19, 2024

cjqian Dec 18, 2024

dimits-ts Dec 19, 2024 •

edited

Loading

cjqian Dec 18, 2024

dimits-ts Dec 19, 2024

mkbehr Dec 20, 2024

cjqian Dec 20, 2024

dimits-ts Dec 23, 2024

mkbehr Dec 20, 2024

mkbehr Dec 20, 2024

vivtsai Dec 20, 2024 •

edited

Loading

dimits-ts Dec 23, 2024

vivtsai left a comment

cjqian commented Dec 20, 2024

dimits-ts commented Dec 23, 2024 •

edited

Loading

Connect hosted LLM #350

Are you sure you want to change the base?

Connect hosted LLM #350

Conversation

dimits-ts commented Dec 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimits-ts Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimits-ts commented Dec 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimits-ts Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vivtsai Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vivtsai left a comment

Choose a reason for hiding this comment

cjqian commented Dec 20, 2024

dimits-ts commented Dec 23, 2024 • edited Loading

dimits-ts Dec 17, 2024 •

edited

Loading

dimits-ts Dec 19, 2024 •

edited

Loading

vivtsai Dec 20, 2024 •

edited

Loading

dimits-ts commented Dec 23, 2024 •

edited

Loading