Orca

Orca is a LLM Orchestration Framework written in Rust. It is designed to be a simple, easy-to-use, and easy-to-extend framework for creating LLM Orchestration. It is currently in development so it may contain bugs and its functionality is limited.

About Orca

Orca is currently in development. It is hard to say what the future of Orca looks like, as I am currently learning about LLM orchestrations and its extensive applications. These are some ideas I want to explore. Suggestions are welcome!

• WebAssembly to create simple, portable, yet powerful LLM applications that can run serverless across platforms.
• Taking advantage of Rust for fast memory-safe distributed LLM applications.
• Deploying LLMs to the edge (think IOT devices, mobile devices, etc.)

Set up

To set up Orca, you will need to install Rust. You can do this by following the instructions here. Once you have Rust installed, you can add Orca to your Cargo.toml file as a dependency:

[dependencies]
orca = { git = "https://github.com/scrippt-tech/orca", package = "orca-core" }

Features

• Prompt templating using handlebars-like syntax (see example below)
• Loading records (documents)
  • HTML from URLs or local files
  • PDF from bytes or local files
• Vector store support with Qdrant
• Current LLM support:
  • OpenAI Chat
  • Limited Bert support using the Candle ML framework
• Pipelines:
  • Simple pipelines
  • Sequential pipelines

Examples

Orca supports simple LLM pipelines and sequential pipelines. It also supports reading PDF and HTML records (documents).

OpenAI Chat

use orca::pipeline::simple::LLMPipeline;
use orca::pipeline::Pipeline;
use orca::llm::openai::OpenAI;
use orca::prompt::context::Context;
use serde::Serialize;

#[derive(Serialize)]
pub struct Data {
    country1: String,
    country2: String,
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = OpenAI::new();
    let prompt = r#"
            {{#chat}}
            {{#user}}
            What is the capital of {{country1}}?
            {{/user}}
            {{#assistant}}
            Paris
            {{/assistant}}
            {{#user}}
            What is the capital of {{country2}}?
            {{/user}}
            {{/chat}}
            "#;
    let pipeline = LLMPipeline::new(&client)
        .load_template("capitals", prompt)?
        .load_context(&Context::new(Data {
            country1: "France".to_string(),
            country2: "Germany".to_string(),
        })?)?;
    let res = pipeline.execute("capitals").await?.content();

    assert!(res.contains("Berlin") || res.contains("berlin"));
    Ok(())
}

Contributing

Contributors are welcome! If you would like to contribute, please open an issue or a pull request. If you would like to add a new feature, please open an issue first so we can discuss it.

Running locally

We use [cargo-make](https://github.com/sagiegurari/cargo-make) to run Orca locally. To install it run:

cargo install cargo-make

Once you have cargo-make installed, you can build or test Orca by running:

$ makers build # Build Orca
$ makers test # Test Orca