Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extension for local generators #17

Closed
psychemedia opened this issue Mar 16, 2023 · 4 comments
Closed

Extension for local generators #17

psychemedia opened this issue Mar 16, 2023 · 4 comments
Labels
enhancement New feature or request project:extensibility Extension points, routing, configuration

Comments

@psychemedia
Copy link

I'm not sure if this a a feature request or question about whether you might expect, or are hoping to see, support for other models being submitted as PRs to this repo or as plugin modules that can be added from third party installers.

Recent days have seen powerful hints that the days of running LLMs on the desktop have arrived sooner than we might have expected ( https://simonwillison.net/2023/Mar/11/llama/ ). Image generaators are also available for local use, either bundled as an an application or as a local web service. For example, automatic111/stable-diffusion-webui provides quite a complex UI which suggests their might be quite a rich API available that jupyter-ai could hook into.

So as a feature request, I guess my issue is: it would be useful to have modules for stable diffusion running locally and for one of the new "portable" LLMs running locally.

(I also wonder about jupyterlite in-browser exetnensions, eg wrapping something like https://xenova.github.io/transformers.js/ ?)

As a question about third party plugins, where eg there is a module shipped to support the stable--diffusion-web-ui shipped from the automatic111 org, how would that work?

Possibly related to this, I note that jupyterlite, which originally baked in pyolite kernel, now seems to be re-engineering so that all kernels are in their own repo, and jupyterlite is just an "empty" container. Would that be a more scaleable approach for this project?

@psychemedia psychemedia added the enhancement New feature or request label Mar 16, 2023
@dlqqq
Copy link
Member

dlqqq commented Mar 31, 2023

@psychemedia These are all very interesting questions! Sorry for the late reply, I've been super busy as of late.

it would be useful to have modules for stable diffusion running locally and for one of the new "portable" LLMs running locally.

We are in the process of integrating more deeply with LangChain. LangChain does a lot of interesting things with language models, but the most relevant feature here is that they provide a fairly comprehensive "bank" of language model providers. A model provider is any class that provides an interface to one or more models; this abstraction is necessary because many models share the same Python interface (e.g. AI21, Cohere, OpenAI). You can see the full list under langchain/llms.

LangChain also includes a self-hosted provider which can use models being run locally via the transformers library.

We fully plan to expose LangChain model providers out-of-the-box so users can get started quickly with the most popular and relevant language models. This would also allow for self-hosting.

(I also wonder about jupyterlite in-browser exetnensions, eg wrapping something like https://xenova.github.io/transformers.js/ ?)

LangChain also has a JavaScript library; so maybe this will be possible in the future. However, this isn't a priority for us, because there are some interesting applications of language models that require a server runtime and access to the ContentsManager. This would expand Jupyter AI's capabilities beyond simple tasks achievable via one-shot/few-shot prompting.

As a question about third party plugins, where eg there is a module shipped to support the stable--diffusion-web-ui shipped from the automatic111 org, how would that work?

New functionality to Jupyter AI is added via AI modules, which are just Python packages that automatically register new modular components. Modular components are anything extensible via AI modules, including model providers. Meaning here that if we wanted to support stable diffusion, the next step is to create a new AI module jupyter-ai-stablediffusion that exposes a StableDiffusionProvider class. To use this, users would then just run:

pip install jupyter-ai-stablediffusion

And automagically get access to the stable diffusion model. This is already what's happening with jupyter-ai-dalle. 😁

Possibly related to this, I note that jupyterlite, which originally baked in pyolite kernel, now seems to be re-engineering so that all kernels are in their own repo, and jupyterlite is just an "empty" container. Would that be a more scaleable approach for this project?

Not sure whether this is a question regarding the design of the backend or the design of the Git repo itself.

  1. For the backend design, we do intend support for LangChain providers out-of-the-box because we plan to make LangChain a core dependency of Jupyter AI, and because the most common use-case is interaction with common language models. Our team had a few discussions about this, and the idea of keeping Jupyter AI as an "empty box" seems like a bad implementation detail being exposed to users who just want to use Jupyter AI out-of-the-box. Keep in mind however that we do not install the dependencies out-of-the-box, so you will have to run pip install openai manually if you want to use the LangChain OpenAI provider. This limits wheel size growth to just the source code in LangChain, which is not problematic.

  2. For the repository design, we chose a monorepo because it's easier to keep intra-package dependencies synced. All AI modules are a downstream dependent on the core jupyter-ai package. If we were to maintain jupyter-ai in a separate repo and all AI modules in separate repos, we would have to do (N+1) releases, unless we want to forego version locks (which is silly in the context of production-readiness).

This is a common problem in the Jupyter ecosystem, where developers often have to undergo an incredibly time-consuming "release chain" (release an upstream dependency, then release all of the downstream dependents, and repeat this process). We want to encourage developers to adopt the monorepo convention for new projects that may involve multiple packages.

@ellisonbg ellisonbg changed the title Extrnsion for local generators Extension for local generators Apr 16, 2023
@JasonWeill
Copy link
Collaborator

@psychemedia Thanks again for opening this issue! Since @dlqqq 's comment in late March, we have adopted LangChain in this project. We're also working on adding support for registering LangChain chains for local usage: see #46.

@psychemedia
Copy link
Author

psychemedia commented May 9, 2023

@JasonWeill Thanks... having to write local chains is a bit beyond many people though? I was thinking something more like being able to say %ai -m gpt4all-lora-unfiltered-quantized.bin or %ai -m ./PATH/TO/gpt4all-lora-unfiltered-quantized.bin in a default chain?

@JasonWeill JasonWeill added the project:extensibility Extension points, routing, configuration label Jul 18, 2023
@JasonWeill
Copy link
Collaborator

Although this is an older issue, I'm closing this issue in favor of #190, where we're currently tracking work for local modules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request project:extensibility Extension points, routing, configuration
Projects
None yet
Development

No branches or pull requests

3 participants