Add token usage tracking

[marcominerva]

Motivation and Context (Why the change? What's the scenario?)

This is the new PR for token usage. It replaces #872.

High level description (Approach, Design)

Following the suggestion of the previous PR, this new implementation uses the built-in token counter feature that is provided by models. For the moment, it supports TextGeneration using OpenAI and Azure OpenAI:

@dluc I have a weird behavior that duplicates the last character of the response:

II can't figure out the issue, so I have made the PR despite this. I hope you can help me finding the issue 😄

[dluc]

the last char (token) duplication is likely related to the new streaming logic. Here's another test, using streaming, the last 2 data blocks contain the same token

....

data: {"streamState":"append","noResult":false,"text":",","tokenUsages":[{"Timestamp":"2024-12-17T21:38:14.818472Z","ServiceType":null,"ModelType":"TextGeneration","ModelName":null,"tokenizer_tokens_in":15647,"tokenizer_tokens_out":0,"service_tokens_in":null,"service_tokens_out":null,"service_reasoning_tokens":null}]}

data: {"streamState":"append","noResult":false,"text":",","tokenUsages":[{"Timestamp":"2024-12-17T21:38:20.296029Z","ServiceType":"Azure OpenAI","ModelType":"TextGeneration","ModelName":"gpt-4o","tokenizer_tokens_in":15647,"tokenizer_tokens_out":0,"service_tokens_in":15654,"service_tokens_out":300,"service_reasoning_tokens":null}]}

data: [DONE]

[marcominerva]

@dluc I have applied your suggestions and I think I have also solved the duplicate token issue.

[dluc]

I've done a bit of refactoring to reduce the number of changes. There's something odd with streaming, e.g. the entire MemoryAnswer object, including the nested objects, should be additive, while instead the token usage is not, repeating the same data over each token.

I'll take a look at the streaming part as soon as I get some time, I also want to review the changes to SearchClient a bit better. The main job is done though, just a matter of few tweaks, thanks! :-)

[nurkmez2]

Hi @dluc, @marcominerva

Thanks for the updates on this topic.

When might this feature become available?

Happy new year

[dluc]

Hi @dluc, @marcominerva

Thanks for the updates on this topic.

When might this feature become available?

Happy new year

happy new year :-)

working on it, I'd like to merge this asap and release a new KM version with all the latest goodies - hopefully won't take long

[dluc]

@marcominerva I've made a few changes, mostly around how the token usage is streamed. Could you check if the code still works as you expected?

Other than that, I think the PR is ready to merge. Thanks!

[marcominerva]

@dluc I have just made a test using my environment and everything works correctly. Thank you for your support!

[nurkmez2]

Hi @dluc
When would it be possible to make this feature available?
Thanks

[dluc]

@nurkmez2 released! - see version 0.96.

[nurkmez2]

Hi @marcominerva, @dluc

Thanks for your help regarding the token usage.

Can you explain what you mean by the following fields?
Which values will provide the total tokens I consume for this request in the Azure model?

For example;

Answer.TokenUsage =
{
"serviceType": "Azure OpenAI",
"modelType": "TextGeneration",
"modelName": "gpt-4o",
"tokenizerTokensIn": 6512,
"tokenizerTokensOut": 454,
"serviceTokensIn": 6519,
"serviceTokensOut": 454,
"serviceReasoningTokens": 0
}

Total Consumed Token is "serviceTokensIn": 6519 + "serviceTokensOut": 454 = 6973 or not?

Thanks