TrustyAI integration with Guardrails

[evaline-ju]

“Guardrails” is a stack of various components that enable standalone detections on text, as well as invoking detections on text generation server input (i.e. user prompts) and text generation server output (i.e. generated text).

Background

Components of the "Guardrails" stack include:

• Orchestrator - available open-sourced here / API here
  • This REST service is the main entry point for a user trying to get detection information, such as on their LLM inputs and/or outputs. It coordinates the calls to text generation servers (LLM servers), chunker server(s), and detector server(s), described below.
• Detectors / detector servers
  • These REST services provide detection information such as (but not limited to) classifications on text and are expected to conform to the detectors API.
• Chunkers / chunker servers
  • Chunkers can be thought of as servers that “chunk” the target modality. For text, this can look like tokenizers or sentence splitters. Currently to work with the orchestrator, these are expected to be GRPC servers, to support bidirectional streaming. Detectors can specify corresponding chunkers in configuration.
  • Proto for reference

We generally have assumed the text generation server(s) are separate from the “Guardrails” stack i.e. deployed and maintained separately. The grpc text generation provider can be specified through orchestrator configuration. Currently, the orchestrator is directly compatible and tested with:

1. TGIS grpc interface, proto
2. caikit-nlp text generation interface

Integration

This integration will allow for TrustyAI users to be able to get detections in their LLM applications in real-time and allow users to bring in other detectors (e.g. potentially toxic text detectors) into their applications.

NOTE: This particular TrustyAI integration discussion will be considered separately of being able to build and provide official images for the open-sourced orchestrator, along with any image maintenance issues such as security scanning, and thus is not contingent on Rust knowledge.

The integration will include:

• Standing up an open-sourced Guardrails stack
  • This will require open-sourced versions of at least one running detector and potentially chunker, depending on the detector of choice.
• Controller to be able to deploy / maintain a Guardrails stack via the TrustyAI operator
  • This will eventually allow RHOAI consumers access to the Guardrails stack. However, we want to make sure the features are toggle-able so a consumer who may want only metrics [through TrustyAIService today] or only detections are not bound to include other features.

Starting this discussion to make sure the points of intended integration are agreed upon, and if so, potentially the details of moving forward

[ruivieira]

@evaline-ju thank you for the detailed proposal, this looks great!

Can I suggest that this is also submitted as an ADR (could be verbatim) to the community repo? This way we can formalise the planning around this effort.

[evaline-ju]

sure, I can contribute an ADR

[evaline-ju]

ADR PR added: #41