For backdrop, I failed miserably at natural language prompting to generate dynamic SQL statements (those that require mathematical computations) with a GPT-5.1 model setup on Microsoft Foundry. I then tried my experiment at Hugging Face on a Qwen2.5-72B-Instruct model. Success! But not without a lot of work and a ton of instructions. So many instructions, it was no different than writing a stored procedure.
The dream of typing a simple question into a prompt - connected to a model - that will convert natural language into an SQL query is harder than you would think. Yet, it makes a lot of sense if you think about it.
Probabilistic vs. Deterministic: The Inherent Spirit of the Model
LLMs are probabilistic in nature. There is no way around this and it will never change. They excel at analyzing text as they play the game of "guess the next word." How do they guess the next word? By analyzing history, by reading what has already been written down.
While models will always remain probabilistic, they become extremely deterministic when they are grounded by historical data that is assumed to be factual and linear in nature. If you ask a model, "Who was the first President of the United States?" it will answer, "George Washington", because that is what it has in its vectors. Someone wrote it down.
If the model doesn't have the information stored, it has to search for it, meaning you must direct it to the source. That works well when you have a specific request and know exactly where the answer lives in your data. However, once we allow wildcard prompts, we can only tell the model where the data is. The model must then extract it independently. This is where the probabilistic nature of the model - combined with a few other factors - becomes a liability.
The Realities of Dynamic Text-to-SQL and Aggregations with LLMs
This could get complicated, but I am going to keep it simple with a handful of bullet points.
- An LLM cannot see your data source. If you are using a SQL database, the model lacks inherent knowledge of it. It knows nothing about its structure, fields, or data points.
- To connect a data source to an LLM, you have to use special protocols or libraries that assist in finding the database, writing the SQL query, and executing it - not to mention returning the data. In-between all that is a lot of fragility as these same protocols and tools also cannot see your data and also lack the inherent knowledge of your data source.
- Commercial LLMs will never return the exact same verbiage multiple times in a row. Those same models often never write the same SQL query two times in a row. Regardless of what language you want the model to spit out, it will never structure it the same way twice.
What About Simple Text Queries?
I must admit, I did not test many simple text queries. While they seemed fairly successful initially, I found they still required coaching. For example, prompting the model to "Find all records with the last name of Taylor" worked most of the time, but failed on occasion. Ultimately, I had to write the prompt using the exact dataset name and physical field names to get consistent results:
Find all the records in the [Users] table where [UserLastName] = "Taylor"
Not all that convenient.
I Wanna Hold Your Hand
I was able to successfully run aggregates using the Qwen2.5-72B-Instruct model on Hugging Face, but the sheer volume of required prompt instructions makes it unfeasible for real-world production. Not only did I have to hardcode a large portion of the SQL query directly into the system instructions, but I also had to set the temperature to zero to keep the query the same on multiple runs.
llm_engine = InferenceClientModel(
model_id="Qwen/Qwen2.5-72B-Instruct",
token="This isMyHFToken",
timeout=300,
temperature=0.0
)
You can click the image below to see an example of the all the prompt instructions I had to write for querying a sports database about the Sabres' record after scoring 5 or more goals in their previous game. It's just not practical.
Solutions?
Hopefully you found this blog before you pulled your hair out with this stuff. My radar is up to see if there is a method or technique out there that will bring us to the promised land. I doubt there ever will be - not through prompting alone. When we need to calculate a fact (deterministic), we are held back by the model and its counterparts (probabilistic). The fix isn't a better prompt. It's giving the model something deterministic to lean on.