Support prompt variants #14487

JonasHelming · 2024-11-20T10:19:27Z

What it does

Introduce the capability for prompt variants, i.e. variants of a main prompt to e.g. support specific LLMs.
There is always a main prompt (default) that can have multiple variants.
The currently selected variant is a setting.

Extended prompt service to manage variants
Extended AgentSettings to store if a variant for a main template is selected.
Prompt Variants can be edited as well
Add an empty prompt for the universal model (useful to talk to small models or just "diretcly" to an LLM)

How to test

Switch the prompt template for the universal agent and check in History view that the system prompt is adapted
Restart to check the the setting is persistent
Modify the variant
etc.

NOTE: There is an unrelated bug in the prompt watching. I can sometimes get into a state, where I change a prompt template, but the prompt customization service is not notified anymore. This usually happens during demos, but I have no reproducible path yet.

Review checklist

As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

As a reviewer, I agree to behave in accordance with the review guidelines

planger · 2024-11-20T11:14:42Z

Thanks, this enables a nice use case!

I'd be for option (b) or (c) -- tendency towards (b). I think the agent shouldn't really be aware of that at all when obtaining the prompt.

I tend to prefer keeping it as an agent setting too:
As far as I understand, the use case is a user switches to another LLM for an agent and as a consequence selects a variant of the prompt template that is optimized for the prompt. Both of these settings are currently scoped to the agent (at least practically).

The confusing thing is, that anyone can request this prompt template, and they would obtain the "agent-scoped" variant of the prompt when resolving it too. This is because prompt templates are per se indepenent of agents, but we only use it in an agent-specific way so far. However, I guess this is a rather theoretical problem, I can't see a lot of valid use cases where another agent than the agent who contributed the prompt template would consume it?
It feels like we should discourage that and rather say: create a copy of the template for your agent. WDYT?

fixed #14485 Signed-off-by: Jonas Helming <[email protected]>

eneufeld

I like the new feature, I left some small remarks

eneufeld · 2024-11-21T08:47:48Z

packages/ai-core/src/browser/ai-configuration/agent-configuration-widget.tsx

-                        template={template}
-                        promptCustomizationService={this.promptCustomizationService} />)}
+                {agent.promptTemplates
+                    ?.filter(template => !template.variantOf)


what if the agent does not provide a template without a variant? shouldn't we handle this? eg show 'no default template available'?

Great suggestions, see above

packages/ai-core/src/browser/ai-configuration/template-settings-renderer.tsx

eneufeld · 2024-11-21T08:53:11Z

packages/ai-core/src/common/agent-service.ts

@@ -99,7 +99,7 @@ export class AgentServiceImpl implements AgentService {
    registerAgent(agent: Agent): void {
        this._agents.push(agent);
        agent.promptTemplates.forEach(
-            template => this.promptService.storePrompt(template.id, template.template)
+            template => this.promptService.storePromptTemplate(template)


do we need the old method?

This would break the API, I don't know whether people are already using it. For conveniance, I would probably leave it.

mark it as deprecated? with a hint to use the new method instead?

Hmm, it is just a simpler version for all manditory values, I would keep it.

eneufeld · 2024-11-21T08:53:43Z

packages/ai-core/src/common/prompt-service.ts

@@ -68,6 +74,11 @@ export interface PromptService {
     * @param prompt the prompt template to store
     */
    storePrompt(id: string, prompt: string): void;


as above, do we need this anymore?

This would break the API, I don't know whether people are already using it. For conveniance, I would probably leave it.

eneufeld · 2024-11-21T08:56:24Z

packages/ai-core/src/browser/ai-configuration/template-settings-renderer.tsx

+        const selectedVariants = agentSettings?.selectedVariants || {};
+
+        const updatedVariants = { ...selectedVariants };
+        if (newVariant === DEFAULT_VARIANT) {


why do we need to handle that case special?

I dont understand the question. If the default template is selected for a template ID, we remove the entry from the selected variants?

yeah i see that, why not just store that the user selected the default variant?
I'm fine also with the current solution was just wondering, whether there is a special reason to do so

Hmm, I find it cleaner to only store if variants are actually selected.

eneufeld · 2024-11-21T09:53:55Z

Tested it, works great.

Signed-off-by: Jonas Helming <[email protected]>

JonasHelming · 2024-11-21T11:16:47Z

Adressed all comments (except the last question)

Signed-off-by: Jonas Helming <[email protected]>

eneufeld

Thank you. Works for me.
As follow-ups:

allow to add variants
if no default template is provided, the user should still be able to select/use a variant instead.

sdirix

LGTM! Thanks!

As discussed, preferably a single method for storing templates would simplify the interface.

Signed-off-by: Jonas Helming <[email protected]>

JonasHelming marked this pull request as draft November 20, 2024 10:19

JonasHelming force-pushed the GH-14485 branch from f3a3bf8 to e1b9102 Compare November 20, 2024 23:32

Support prompt variants

e770ba6

fixed #14485 Signed-off-by: Jonas Helming <[email protected]>

JonasHelming force-pushed the GH-14485 branch from e1b9102 to e770ba6 Compare November 20, 2024 23:44

JonasHelming marked this pull request as ready for review November 20, 2024 23:53

eneufeld reviewed Nov 21, 2024

View reviewed changes

Adress review comments

55e66af

Signed-off-by: Jonas Helming <[email protected]>

JonasHelming requested a review from eneufeld November 21, 2024 11:16

Fix linting

31db839

Signed-off-by: Jonas Helming <[email protected]>

eneufeld approved these changes Nov 21, 2024

View reviewed changes

sdirix approved these changes Nov 22, 2024

View reviewed changes

Remove deprecated function

de67625

Signed-off-by: Jonas Helming <[email protected]>

JonasHelming merged commit 2a0232d into master Nov 23, 2024
11 checks passed

github-actions bot added this to the 1.56.0 milestone Nov 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support prompt variants #14487

Support prompt variants #14487

JonasHelming commented Nov 20, 2024 •

edited

Loading

planger commented Nov 20, 2024

eneufeld left a comment

eneufeld Nov 21, 2024

JonasHelming Nov 21, 2024

JonasHelming Nov 21, 2024

eneufeld Nov 21, 2024

JonasHelming Nov 21, 2024

eneufeld Nov 21, 2024 •

edited

Loading

JonasHelming Nov 21, 2024

eneufeld Nov 21, 2024

JonasHelming Nov 21, 2024

eneufeld Nov 21, 2024

JonasHelming Nov 21, 2024

eneufeld Nov 21, 2024

JonasHelming Nov 21, 2024

eneufeld commented Nov 21, 2024

JonasHelming commented Nov 21, 2024

eneufeld left a comment

sdirix left a comment

Support prompt variants #14487

Support prompt variants #14487

Conversation

JonasHelming commented Nov 20, 2024 • edited Loading

What it does

How to test

Review checklist

Reminder for reviewers

planger commented Nov 20, 2024

eneufeld left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eneufeld Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eneufeld commented Nov 21, 2024

JonasHelming commented Nov 21, 2024

eneufeld left a comment

Choose a reason for hiding this comment

sdirix left a comment

Choose a reason for hiding this comment

JonasHelming commented Nov 20, 2024 •

edited

Loading

eneufeld Nov 21, 2024 •

edited

Loading