Deepseek R1 with JSON returns partial message text (a bunch of new lines)

[pdevito3]

Description

When making a curl call with deepseek r1 like this

curl --request POST \
  --url http://localhost:11434/api/generate \
  --header 'Content-Type: application/json; charset=utf-8' \
  --data '{
  "model": "deepseek-r1",
  "prompt": "[INST] Generate 5 product category names for an online retailer\n of high-tech outdoor adventure goods and related clothing/electronics/etc.\n Each category name is a single descriptive term, so it does not use the word '\''and'\''.\n Category names should be interesting and novel, e.g., \"Mountain Unicycles\", \"AI Boots\",\n or \"High-volume Water Filtration Plants\", not simply \"Tents\".\n This retailer sells relatively technical products.\n\n Each category has a list of up to 8 brand names that make products in that category. All brand names are\n purely fictional. Brand names are usually multiple words with spaces and/or special characters, e.g.\n \"Orange Gear\", \"Aqua Tech US\", \"Livewell\", \"E & K\", \"JAXⓇ\".\n Many brand names are used in multiple categories. Some categories have only 2 brands.\n \n The response should be in a JSON format like below with the exact batch count of objects. It is very important you make sure it is in this format and that it is valid JSON. \n { \"categories\": [{\"name\":\"Tents\", \"brands\":[\"Rosewood\", \"Summit Kings\"]}] } [/INST]",
  "format": "json",
  "response_format": {
    "type": "json_object"
  },
  "options": {
    "temperature": 0.9,
    "numPredict": 1750,
    "topP": 1
  },
  "raw": true,
  "stream": false,
  "stop": [
    "[/TOOL_CALLS]"
  ]
}'

you get a response like

{
  "model": "deepseek-r1",
  "created_at": "2025-01-26T17:26:53.933014Z",
  "response": "{\n \"categories\": [\n   {\n     \"name\": \"Mountain Unicycles\",\n     \"brands\": [\"Quark Waves\", \"Throttle Pro\"]\n   },\n   {\n     \"name\": \"AI Boots\",\n     \"brands\": [\"Nanite Steps\", \"RoboGuard\"]\n   },\n   {\n     \"name\": \"High-End Hiking Boots\",\n     \"brands\": [\"Pinnacle Trails\", \"Cobra X\"]\n   },\n   {\n     \"name\": \"Electric Kites\",\n     \"brands\": [\"EcoWings\", \"SkyDance\"]\n   },\n   {\n     \"name\": \"Smartwater Rides\",\n     \"brands\": [\"Voyager Pro\", \"Oasis Explore\"]\n   }\n ]\n}\n \n \n\n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n",
  "done": false
}

This is fine other than the weird new lines at the end -- they're causing a problem when interacting with MS AI though as they are the only part of the message text response.

Note

fwiw, i get that maybe this is a bug that needs to be resolved by them but wanted to at least call it out for y'all and other's awareness

Reproduction Steps

When doing a comparable call (at least i believe so) with MS AI:

var chatOptions = new ChatOptions
        {
            ResponseFormat = ChatResponseFormat.Json,
            // response_format neded for their json setup -- https://api-docs.deepseek.com/guides/json_mode
            AdditionalProperties = new AdditionalPropertiesDictionary
            {
                ["response_format"] = new
                {
                    type = "json_object"
                }
            }
        };

        var prompt = "Generate 5 product category names for an online retailer\n of high-tech outdoor adventure goods and related clothing/electronics/etc.\n Each category name is a single descriptive term, so it does not use the word 'and'.\n Category names should be interesting and novel, e.g., \"Mountain Unicycles\", \"AI Boots\",\n or \"High-volume Water Filtration Plants\", not simply \"Tents\".\n This retailer sells relatively technical products.\n\n Each category has a list of up to 8 brand names that make products in that category. All brand names are\n purely fictional. Brand names are usually multiple words with spaces and/or special characters, e.g.\n \"Orange Gear\", \"Aqua Tech US\", \"Livewell\", \"E & K\", \"JAXⓇ\".\n Many brand names are used in multiple categories. Some categories have only 2 brands.\n \n The response should be in a JSON format like below with the exact batch count of objects. It is very important you make sure it is in this format and that it is valid JSON. \n { \"categories\": [{\"name\":\"Tents\", \"brands\":[\"Rosewood\", \"Summit Kings\"]}] }";

        var chatCompletion = await chatClient.CompleteAsync(prompt, chatOptions);
        var parsedJson = JsonSerializer.Deserialize<CategoriesResponse>(chatCompletion.Message.Text, 
new(JsonSerializerDefaults.Web)
    {
        WriteIndented = true
    });

You just get the new lines as the message text response:

Expected behavior

get the full text response

Actual behavior

getting a partial response with just new lines

Regression?

Kind of if you're talking overall capabilities of parsing a message -- message text comes back fine for other models, but not deepseek, but could be classified as an enhancement to handle more scenarios like the deepseek one

Known Workarounds

None with MS AI that I know of, I'd need to manually make the http call like the curl call and parse the response from there.

Configuration

.NET 9
Mac Sequoia 15.2
AFAIK, not specific to this config

Other information

No response

[stephentoub]

What is chatClient here? How is it constructed? Please share a full repro.

[stephentoub]

(Separately, you might consider using CompleteAsync<T> instead of setting the response format and manually deserializing)

[pdevito3]

It's just an injected IChatClient from MS AI.

I'll try CompleteAsync and report back, but I don't think it'll matter for this, though it might simplify things which will be nice -- the issue is before deserialization to help with the actual message in the chat client.

Regardless, I'll give it a whirl and see 🙂

[pdevito3]

Will make a sample repo too

[stephentoub]

It's just an injected IChatClient from MS AI.

Right, but which concrete type. There are many different implementations of IChatClient; I'm wondering which you're using.

A simple repro would be most helpful.

[pdevito3]

gotcha -- here's a simple console app repro -- can reliably show successes with llama 3.2 and repro successes and failures with deepseek

[pdevito3]

I tried CompleteAsync<T> with TryGetResult() and it doesn't seem to help (if anything it actually makes the response parsing less reliable) -- i also updated it to just run all 4 permutations for easier comparison and debug

[stephentoub]

This appears to just be an issue with the model / Ollama's support for enforcing JSON object returns. I suspect deepseek's chain-of-thought reasonining makes it harder for it to produce just a JSON response object, as responses the model issues will generally be prefixed by a ... block.

@SteveSandersonMS, it might be interesting to think through what, if anything, we should do in CompleteAsync<T> to try to account for such things.