For my specific sports data project, I am primarily interested in data validation and statistical correction. I have seen many cases where AI struggles with math and statistical calculations, whether using Copilot or public LLMs like Gemini and Claude. To ensure the utmost accuracy, I prefer implementing an automated audit. Code Interpreter utilizes a sandboxed Python environment packed with analytical libraries designed to clean up math and correct flawed statistical reasoning. Essentially, it performs an audit on the text you input.
Here is the catch: in a standard RAG pipeline, it will only alert you to anomalies; it will not auto-correct them within the text narrative for you. This is still an excellent feature if you want public-facing users to be alerted to data anomalies, or if you need an internal tool to flag statistical errors in documents before they are distributed. In those scenarios, your output would include inline disclaimers such as:
Note: The calculation of 5 / 10 = 5 is incorrect. The correct answer is 0.5.
However, I am building a pre-loaded prompt system where contextual summaries must be displayed directly, making a manual pre-screen before the text hits the screen impractical. Furthermore, I do not want disclaimers in my final output pointing out basic statistical errors that simply shouldn't be wrong in the first place - I want to fix them on the fly!
To clarify, I am less concerned about anomalies originating from my own input data (my site) and more concerned about anomalies slipping into the output from external sources like Bing Web Search. While building an internal review system is an option, my architecture is already heavily processing data. My core concern is catching minor anomalies in the text returned by the web search, meaning disclaimers within the text are out of the question.
So, what do you do if you want the output text to be dynamically updated with the correct math?
This is where things get a bit more complex, though not terribly complex. If you want to use Code Interpreter to find and fix statistical errors, you must perform an audit on the original output. You can achieve this by creating a lightweight, pre-built LLM agent, connecting it to the Code Interpreter tool, and giving it explicit instructions to rewrite the text while correcting the calculations.
The workflow looks like this:
- Step 1: Execute your primary RAG request.
- Step 2: Persist the raw RAG response to a database or Azure Blob Storage.
- Step 3: Pass that response to a dedicated, pre-built LLM agent configured strictly to run a statistical audit via Code Interpreter and return a modified version of the original text with all anomalies removed.
One note about Code Interpreter: it does not charge by CPU usage, but rather by the session. A standard session lasts for up to one hour. You will get a lot of mileage for your money if you can run all your requests within one session, but depending on the size of the requests, you may need to run them step by step or make smaller batches and run the batches step by step. It's all about orchestration in AI development!
I still have a lot of work ahead of me to scale this architecture out, but this approach is officially on the requirements list. After extensive research, this is the path forward I have chosen. I will be tackling this implementation over the next few days as I wrap up the in-context RAG project, so expect more updates on this topic soon!