Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON schema response format does not set 'strict' parameter for OpenAI models #5808

Open
NiceWaffel opened this issue Jan 20, 2025 · 0 comments
Labels
area-AI enhancement This issue represents an ask for new feature or an enhancement to an existing one untriaged

Comments

@NiceWaffel
Copy link

Description

When using the OpenAIChatClient provided by the Microsoft.Extensions.AI.OpenAI package, the strict parameter of the chat response format is not set.

if (options.ResponseFormat is ChatResponseFormatText)
{
  result.ResponseFormat = OpenAI.Chat.ChatResponseFormat.CreateTextFormat();
}
else if (options.ResponseFormat is ChatResponseFormatJson jsonFormat)
{
  result.ResponseFormat = jsonFormat.Schema is { } jsonSchema
    ? OpenAI.Chat.ChatResponseFormat.CreateJsonSchemaFormat(
      jsonFormat.SchemaName ?? "json_schema",
      BinaryData.FromString(jsonSchema),
      jsonFormat.SchemaDescription)
    : OpenAI.Chat.ChatResponseFormat.CreateJsonObjectFormat();
}

Reproduction Steps

Try with the following setup. Here I am using a model on Azure.

var openAiClient = new AzureOpenAIClient(new Uri(ApiUrl), new ApiKeyCredential(ApiKey));

const string prompt = "Generate JSON according to the schema.";
const string schema = """
                      {
                        "$schema": "http://json-schema.org/draft-04/schema#",
                        "type": "object",
                        "additionalProperties": false,
                        "required": ["A"],
                        "properties": {
                          "A": {
                            "type": "integer",
                            "description": "Set this field to null."
                          }
                        }
                      }
                      """;

var chatClient = new OpenAIChatClient(openAiClient, "gpt-4o-structured-outputs");
var schemaResponseFormat = ChatResponseFormat.ForJsonSchema(JsonSerializer.Deserialize<JsonElement>(schema), "Schema");

var result = await chatClient.CompleteAsync(
  [new ChatMessage(ChatRole.User, prompt)],
  new ChatOptions { ResponseFormat = schemaResponseFormat });

Console.WriteLine(result.Choices[0].Contents[0].ToString());

Expected behavior

I'd expect there to either be a parameter for strict behavior, or strict behavior to be the default.

Actual behavior

The ChatResponseFormatJson does not have a property for strict, and the ForJsonSchema method does not have a parameter for it. Thus it just silently defaults to null.

Regression?

This did not work in any previous release.

Known Workarounds

You could implement your own version of the OpenAIChatClient. In there the strict parameter could be set to a different default value.

Configuration

Using version 9.1.0-preview.1.25064.3 of the package with .NET 9.

Other information

No response

@NiceWaffel NiceWaffel added bug This issue describes a behavior which is not expected - a bug. untriaged labels Jan 20, 2025
@stephentoub stephentoub added enhancement This issue represents an ask for new feature or an enhancement to an existing one area-AI and removed bug This issue describes a behavior which is not expected - a bug. labels Jan 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-AI enhancement This issue represents an ask for new feature or an enhancement to an existing one untriaged
Projects
None yet
Development

No branches or pull requests

2 participants