have entered the world of computer science at a record pace. LLMs are powerful models able to effectively performing a wide selection of tasks. Nonetheless, LLM outputs are stochastic, making them unreliable. In this text, I discuss how you possibly can ensure reliability in your LLM applications by properly prompting the model and handling the output.
It’s also possible to read my articles on Attending NVIDIA GTC Paris 2025 and Creating Powerful Embeddings for Machine Learning.
Table of Contents
Motivation
My motivation for this text is that I’m consistently developing latest applications using LLMs. LLMs are generalized tools that might be applied to most text-dependent tasks equivalent to classification, summarization, information extraction, and rather more. Moreover, the rise of vision language models also enable us to handle images just like how we handle text.
I often encounter the issue that my LLM applications are inconsistent. Sometimes the LLM doesn’t respond in the specified format, or I’m unable to properly parse the LLM response. This can be a huge problem if you end up working in a production setting and are fully depending on consistency in your application. I’ll thus discuss the techniques I exploit to make sure reliability for my applications in a production setting.
Ensuring output consistency
Markup tags
To make sure output consistency, I exploit a method where my LLM answers in markup tags. I exploit a system prompt like:
prompt = f"""
Classify the text into "Cat" or "Dog"
Provide your response in tags
"""And the model will almost at all times respond with:
Cat
or
Dog You’ll be able to now easily parse out the response using the next code:
def _parse_response(response: str):
return response.split("")[1].split(" ")[0]The rationale using markup tags works so well is that that is how the model is trained to behave. When OpenAI, Qwen, Google, and others train these models, they use markup tags. The models are thus super effective at utilizing these tags and can, in just about all cases, adhere to the expected response format.
For instance, with reasoning models, which have been on the rise currently, the models first do their pondering enclosed in
Moreover, I also try to make use of as many markup tags as possible elsewhere in my prompts. For instance, if I’m providing just a few shot examples to my model, I’ll do something like:
prompt = f"""
Classify the text into "Cat" or "Dog"
Provide your response in tags
That is a picture showing a cat -> Cat
That is a picture showing a dog -> Dog
"""I do two things that help the model perform here:
- I provide examples in
tags. - In my examples, I ensure to stick to my very own expected response format, using the
Using markup tags, you possibly can thus ensure a high level of output consistency out of your LLM
Output validation
Pydantic is a tool you should use to make sure and validate the output of your LLMs. You’ll be able to define types and validate that the output of the model adheres to the kind we expect. For instance, you possibly can follow the instance below, based on this text:
from pydantic import BaseModel
from openai import OpenAI
client = OpenAI()
class Profile(BaseModel):
name: str
email: str
phone: str
resp = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": "Return the `name`, `email`, and `phone` of user {user} in a json object."
},
]
)
Profile.model_validate_json(resp.selections[0].message.content)As you possibly can see, we prompt GPT to reply with a JSON object, and we then run Pydantic to make sure the response is as we expect.
I’d also wish to note that sometimes it’s easier to easily create your individual output validation function. Within the last example, the one requirements for the response object are essentially that the response object incorporates the keys name, email, and phone, and that each one of those are of the string type. You’ll be able to validate this in Python with a function:
def validate_output(output: str):
assert "name" in output and isinstance(output["name"], str)
assert "email" in output and isinstance(output["email"], str)
assert "phone" in output and isinstance(output["phone"], str)With this, you would not have to put in any packages, and in a variety of cases, it is less complicated to establish.
Tweaking the system prompt
It’s also possible to make several other tweaks to your system prompt to make sure a more reliable output. I at all times recommend making your prompt as structured as possible, using:
- Markup tags as mentioned earlier
- Lists, equivalent to the one I’m writing in here
Basically, it’s best to also at all times ensure clear instructions. You should use the next to make sure the standard of your prompt
In the event you gave the prompt to a different human, that had never seen the duty before, and with no prior knowledge of the duty. Would the human find a way to perform the duty effectively?
In the event you cannot have a human do the duty, you often cannot expect an AI to do it (a minimum of for now).
Handling errors
Errors are inevitable when coping with LLMs. In the event you perform enough API calls, it is sort of certain that sometimes the response won’t be in your required format, or one other issue.
In these scenarios, it’s vital that you might have a strong application equipped to handle such errors. I exploit the next techniques to handle errors:
- Retry mechanism
- Increase the temperature
- Have backup LLMs
Now, let me elaborate on each point.
Exponential backoff retry mechanism
It’s vital to have a retry mechanism in place, considering a variety of issues can occur when making an API call. You may encounter issues equivalent to rate limiting, incorrect output format, or a slow response. In these scenarios, it’s essential to ensure to wrap the LLM call in a try-catch and retry. Normally, it’s also smart to make use of an exponential backoff, especially for rate-limiting errors. The rationale for that is to make sure you wait long enough to avoid further rate-limiting issues.
Temperature increase
I also sometimes recommend increasing the temperature a bit. In the event you set the temperature to 0, you tell the model to act deterministically. Nonetheless, sometimes this will have a negative effect.
For instance, if you might have an input example where the model didn’t respond in the right output format. In the event you retry this using a temperature of 0, you might be more likely to just experience the identical issue. I thus recommend you set the temperature to a bit higher, for instance 0.1, to make sure some stochasticness within the model, while also ensuring its outputs are relatively deterministic.
This is identical logic that a variety of agents use: the next temperature.
They should avoid being stuch in a loop. Having the next temperature may help them avoid repetitive errors.
Backup LLMs
One other powerful method to take care of errors is to have backup LLMs. I like to recommend using a sequence of LLM providers for all of your API calls. For instance, you first try OpenAI, if that fails, you employ Gemini, and if that fails, you should use Claude.
This ensures reliability within the event of provider-specific issues. These may very well be issues equivalent to:
- The server is down (for instance, if OpenAI’s API isn’t available for a time frame)
- Filtering (sometimes, an LLM provider will refuse to reply your request if it believes your request is in violation of jailbreak policies or content moderation)
Basically, it is solely good practice to not be fully depending on one provider.
Conclusion
In this text, I even have discussed how you possibly can ensure reliability in your LLM application. LLM applications are inherently stochastic because you can not directly control the output of an LLM. It’s thus vital to make sure you might have proper policies in place, each to attenuate the errors that occur and to handle the errors once they occur.
I even have discussed the next approaches to attenuate errors and handle errors:
- Markup tags
- Output validation
- Tweaking the system prompt
- Retry mechanism
- Increase the temperature
- Have backup LLMs
In the event you mix these techniques into your application, you possibly can achieve each a robust and robust LLM application.
👉 Follow me on socials:
🧑💻 Get in contact
🌐 Personal Blog
🔗 LinkedIn
🐦 X / Twitter
✍️ Medium
🧵 Threads
