Understanding the Reasoning Effort Setting on Azure Agents

I posted a blog the other day about my in-context RAG project, which is finally completed. After using it, I realized a few things regarding it's effectiveness, when it should be used and how specific platform settings can drastically alter your processing time

The Reasoning Effort setting on Azure Agent models will command the agent to dive deeper and logically fill in gaps when contextual information is not logically joined together. It also works well for figuring out multi-step processes and joining them together in a comprehensive fashion. It does this by allocating "thinking tokens" to your request.

The settings are none | low | medium | high. The higher you go, the more thinking tokens it uses, but also the more compute required and the longer your requests may take to complete. This feature will work with RAG pipelines or pre-built models, making no difference to its capability.

What does it do? Let's look at an example.

Let's say someone enters the following into a prompt and passes it on to the model:

'It's July 4th and the sun came up at daybreak and the moon rose at 9:45 PM.'

It sounds good, but there is a gap in the prompt between daybreak and moonrise. What happened in between? One must infer, and that is what Reasoning Effort will do - try to fill in the gaps.

Reasoning Effort may infer the following:

1. It was a clear daybreak, a clear evening, and probably a nice day, in general.
2. It was a long day, as summer days are.
3. The moon was probably full if it rose that late at night.

Where does this matter? The model will use these inferences for subsequent message exchanges between itself and the user. Let's say the user sends in another prompt:

"Are there any events tonight?"

The model may respond:

"Here is a list of events near you. I would suggest something outdoors, especially with a full moon overhead. It should be a special evening outside."


As you can see, the model inferred that it was likely a full moon outside without the user ever mentioning it.

Short-term Memory
The example above would only work in the context of that conversation. Even a pre-trained model will not retain that memory for a future conversation about the same day.

Changing Textual Summaries on the Fly
As stated before, the inferences the agent gains will be used in subsequent message exchanges in a prompt/response setting. But what if you are using RAG with pre-loaded data and some of your data has timeline gaps in it? You can instruct the model to change the text and add verbiage to tie two things together.

If you preload a prompt with, "The Bills took an early 7-0 lead in the 1st quarter and eventually won the game 21-17.", then the model can change that text and would likely come up with something like, "The Bills took an early 7-0 lead in the 1st quarter. The two teams slugged it out from there with each gaining small gains, but the Bills eventually won the game 21-17."

You just need to give the agent direct instructions to do this.

Final Thought
For chat-based applications, Reasoning Effort is a choice you will need to make. If you want to ensure that model responses intuitively cover gaps and infer missing logic in the user input, then you may want to consider using it. There are sure to be gaps in logic with a back-and-forth conversation with computers involved. Just remember: the higher you go, the more you pay for compute and the slower your responses will be.

For those running in-context RAG where you pre-load the prompt, it will really boil down to how thorough your text inputs are in terms of timelines within the text or systematic processes you are defining in the text. It makes no sense in paying for the extra tokens if there are no gaps.