.Net: Bug: Conflict between Response Format property & function calling

[apg-developer]

Describe the bug
A HTTP 500 server_error is thrown when a ChatCompletion Agent uses function calling to retrieve data. The error only happens if Microsoft.SemanticKernel.Connectors.OpenAI.OpenAIPromptExecutionSettings.ResponseFormat property is enable on the settings of the agent. Otherwise error does not happen.

To Reproduce
Steps to reproduce the behavior:

1. Copy the sample code provided.
2. Use a model based in GPT4o version: 2024-08-06
3. Run the console project & type the word "chicken" to get the custom data default set in the plugin.
4. A HTTP 500 error is thrown on the agent's method: await chatCompletionService.GetChatMessageContentAsync()
5. Note that despite of the error SK adds to the history a correct tool entry with the data provided by the plugin.

Expected behavior
await chatCompletionService.GetChatMessageContentAsync() method should return without errors the data provided by the plugin using the structure provided in ResponseFormat property.

Screenshots

Platform

• OS: Windows
• IDE: Visual Studio
• Language: C#
• Source: [e.g. NuGet package version 0.1.0, pip package version 0.1.0, main branch of repository]

Additional context
I have tried several combinations to try to achieve the goal

1. Using the ResponseFormat property through the type-based schema (Like the example code below).
2. Using the ResponseFormat property through a JSON schema.
3. Returning a “RecipeOut” object from the plugin (As the provided example).
4. Returning a string from the plugin (Like the deserialization commented out in the plugin).

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.OpenAI;

namespace EntitiesAssistant.Recipes {
  public class RecipeCopilot {
    private const string _systemPrompt = "You are an expert recipe assistant." + 
"Your key objective is guide to the user to find the ingredients & steps necessary to prepare the dishes suggested by the user. " + 
"Friendly reminder: You have a repository with various recipes prepared earlier.";

    private ChatHistory _history = new();
    
    private Kernel BuildKernel() {
      var builder = Kernel.CreateBuilder().AddAzureOpenAIChatCompletion(
        deploymentName: ProjectSettings.DeploymentName, 
        apiKey: ProjectSettings.ApiKey, 
        serviceId: ProjectSettings.ServiceId, 
        endpoint: ProjectSettings.Endpoint);

      builder.Plugins.AddFromObject(new RecipePlugin(), pluginName: "recipe");
      return builder.Build();
    }

    public async Task ChatAsync() {
      Console.WriteLine("What would you like to cook today ?");
      string userInput = Console.ReadLine() ?? "I am not hungry";

      var kernel = BuildKernel();
      OpenAIPromptExecutionSettings settings = new() {
        ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions,
        ResponseFormat = typeof (RecipeOutput),
      };

      _history.AddSystemMessage(_systemPrompt);
      _history.AddUserMessage(userInput);

      try {
        var chatCompletionService = kernel.GetRequiredService < IChatCompletionService > (ProjectSettings.ServiceId);
        var response = await chatCompletionService.GetChatMessageContentAsync(_history, settings, kernel);
        Console.WriteLine(response);
      } catch (Exception e) {
        var toolResponse = _history.LastOrDefault(x => x.Role == AuthorRole.Tool)?.Content;
        if (toolResponse is not null) Console.WriteLine(toolResponse);
        Console.WriteLine($"{e.Message} ::: {e.StackTrace}");
        throw;
      }
    }
  }
}



using Microsoft.SemanticKernel;
using System.ComponentModel;

namespace EntitiesAssistant.Recipes {
    public class RecipePlugin {
      RecipeOutput _defaultRecipe = new() {
        Name = "Chicken Curry by APG", 
        Ingredients = [
            new Ingredient {Name = "Money", Quantity = "5", MeasureUnit = "Dollars"},
            ], 
        Instructions = "Step 1: Check your pocket. " + 
        "Step 2: Goes to the restaurant. " + 
        "Step 3: Place your order. " + 
        "Step 4: Pay & enjoy it. "
      };

      [KernelFunction("FindRecipe")][Description("Search for all internal recipes. This is a customized repository for those recipes prepared earlier.")] 
      public async Task < RecipeOutput > FindRecipeAsync() {
          await Task.Delay(1000);
          return _defaultRecipe; 
          //return System.Text.Json.JsonSerializer.Serialize(_defaultRecipe);        
          }    
    }
} 

[dmytrostruk]

@apg-developer Could you please share how the RecipeOutput model looks like? Thanks in advance!

[apg-developer]

@dmytrostruk sure, the definition is on the plugin but here you can see the class definition for RecipeOutput. Thank you so much for your help !!!

public class RecipeOutput {

  public string Name { get;    set;  }

  public string Description {    get;    set;  }

  public Ingredient[] Ingredients {    get;    set;  }

  public string Instructions {    get;    set;  }
}

public class Ingredient {

  public string Name {    get;    set;  }

  public string Quantity {    get;    set;  }

  public string MeasureUnit {    get;    set;  }

}

[dmytrostruk]

@apg-developer Thanks a lot! Could you please try to run the same example but also specify apiVersion parameter when you register Azure OpenAI chat completion service like in this example and see if that works?

semantic-kernel/dotnet/samples/Concepts/ChatCompletion/OpenAI_StructuredOutputs.cs

Lines 203 to 208 in f5facce

	Kernel kernel = Kernel.CreateBuilder()
	.AddAzureOpenAIChatCompletion(
	deploymentName: TestConfiguration.AzureOpenAI.ChatDeploymentName,
	endpoint: TestConfiguration.AzureOpenAI.Endpoint,
	credentials: new AzureCliCredential(),
	apiVersion: "2024-08-01-preview")

[apg-developer]

@dmytrostruk the result was the same. Note the following stack trace & the provided screenshot. Thanks for your help !

General stack trace

  at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.<RunRequestAsync>d__73`1.MoveNext()
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.<GetChatMessageContentsAsync>d__16.MoveNext()
   at Microsoft.SemanticKernel.ChatCompletion.ChatCompletionServiceExtensions.<GetChatMessageContentAsync>d__2.MoveNext()
   at EntitiesAssistant.Recipes.RecipeCopilot.<ChatAsync>d__3.MoveNext() in D:\Apg\SampleWithProcessFramework\EntitiesAssistant\Recipes\RecipeCopilot.cs:line 44
   at Program.<<Main>$>d__0.MoveNext() in ..\EntitiesAssistant\Program.cs:line 8

Inner exception stack trace

at Azure.AI.OpenAI.ClientPipelineExtensions.<ProcessMessageAsync>d__0.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Azure.AI.OpenAI.Chat.AzureChatClient.<CompleteChatAsync>d__14.MoveNext()
   at OpenAI.Chat.ChatClient.<CompleteChatAsync>d__8.MoveNext()
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.<RunRequestAsync>d__73`1.MoveNext()

Modified snippet code

var builder = Kernel.CreateBuilder()
       .AddAzureOpenAIChatCompletion(deploymentName: ProjectSettings.DeploymentName,
       apiKey: ProjectSettings.ApiKey,
       serviceId: ProjectSettings.ServiceId,
       endpoint: ProjectSettings.Endpoint,
       apiVersion: "2024-08-01-preview");

[deq3]

Hello, I am currently facing the same issue when returning data from a plugin function to a ChatCompletionAgent with the ResponseFormat property set. Other ChatCompletionAgents that have ResponseFormat set work fine. Only those that also use a plugin that returns data return a 500 error after calling the function and the results should be processed to the ResponseFormat by gpt-4o. I was surprised by this issue because I didn't change anything in my code and didn't update any dependencies, but it stopped working because of this.

Expected behaviour

agentKernel.ImportPluginFromObject(new SomePluginForDataRetrieval());

var agent = new ChatCompletionAgent()
{
    Name = "ToolAndResponseFormatAgent",
    Instructions = """
        A Prompt guiding how to use the plugin
    """,
    Kernel = agentKernel,
    Arguments = new KernelArguments(new OpenAIPromptExecutionSettings()
    {
        FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(),
        ResponseFormat = typeof(FormattedResponse)
    })
};

await agent.InvokeAsync(chatHistory);

Should not return a 500 error when using ResponseFormat and a plugin that returns data to the agent in the form of an object

Platform

• OS: Windows
• IDE: Visual Studio 2022
• Language: C# .NET 8
• Source:
  • Microsoft.SemanticKernel 1.28.0 (But also newer versions),
  • Microsoft.SemanticKernel.Agents.Core 1.28.0-alpha (Also newer versions)
• Model: gpt-4o version 2024-08-06 running on Azure OpenAI. I tried both eastus and westeurope for the Deployments, but both didn't work

Hope that this issue can be resolved soon, thanks for already pointing it out @apg-developer and for the support so far @dmytrostruk

[dmytrostruk]

@apg-developer @deq3 Thanks again for reporting this issue. It appeared that Structured Outputs feature in Azure OpenAI (i.e. Response Format as JSON Schema) doesn't work with parallel function calls:
https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs?tabs=python-secure#function-calling-with-structured-outputs

When you enable function calling by using old ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions approach or new FunctionChoiceBehavior = FunctionChoiceBehavior.Auto() approach with Azure OpenAI models, by default parallel calls are enabled, which is causing this error.

In order to avoid the error, you need to explicitly disable parallel calls with following syntax:

var executionSettings = new OpenAIPromptExecutionSettings
{
    ResponseFormat = typeof(FormattedResponse),
    FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(options: new() { AllowParallelCalls = false })
};

When you disable parallel calls explicitly, the issue should be resolved for you. Please let me know if that works. Thank you!

[deq3]

Thank you for the quick reply @dmytrostruk. Unfortunately, setting AllowParallelCalls to false did not fix it for me, I still get error 500.

I have removed specifics from my code, but this is how I currently have the agent configured and invoked. Could there be something else causing the problem?

var agent = new ChatCompletionAgent()
{
    Name = "ExampleAgent",
    Instructions = """
        Prompt that tells how to use the tool
    """,
    Kernel = agentKernel,
    Arguments = new KernelArguments(new OpenAIPromptExecutionSettings()
    {
        FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(options: new() { AllowParallelCalls = false }),
        ServiceId = serviceId,
        ResponseFormat = typeof(FormattedResponse)
    }),
};

var history = new ChatHistory();
history.AddUserMessage(message);

var res = agent.InvokeAsync(
    history,
    cancellationToken: cancellationToken
);

var responses = await res.ToListAsync(cancellationToken);
var messageContent = responses.FirstOrDefault();

I used this blog post when building my app and it worked at first, even though they are not using Azure OpenAI like I am. But somehow it stopped working https://devblogs.microsoft.com/semantic-kernel/using-json-schema-for-structured-output-in-net-for-openai-models/#structured-outputs-with-function-calling

Thanks again for your support @dmytrostruk

[apg-developer]

Hi @dmytrostruk, thank you for your answer. I keep getting the same error 500 despite of the provided configuration. Is there something else to consider or could it be a side-effect ?

Thank so much for your help !

[dmytrostruk]

@apg-developer @deq3 Thanks for provided information, I can reproduce the issue as well. As for now, I don't think this occurs because of the recent changes in Semantic Kernel, since I'm trying the newest as well as older SK versions and the issue is the same.

Taking into account that it's HTTP 500 error, it looks like it's something on Azure OpenAI side, related to response format together with function calling. Since there are no details in exception message, I would recommend following the link in exception message and report this issue together with request ID, so it can be reviewed by responsible team.

Meanwhile we will monitor this situation on our side as well. Thank you!

[deq3]

@dmytrostruk Thanks for reproducing the issue. I also didn't suspect Semantic Kernel was the problem since I didn't update my dependencies or touch the code when it stopped working. I will try to report the problem using the link provided, thanks for your help so far and for keeping an eye out.

Has anyone had a chance to try it with the OpenAI API and see if it still works as expected?