No more unstructured data outputs; turn ChatGPT’s completions into structured JSON!

A couple of months ago, OpenAI released their API to most of the people, which excited many developers who wanted to utilize ChatGPT’s outputs in a scientific way. As exciting has this has been, it’s equally been a little bit of a nightmare since we programmers are likely to work within the realm of structured data types. We like integers, booleans, and lists. The unstructured string might be unwieldy to take care of, and with a view to get consistent results, a programmer is required to face their worst nightmare: developing a daily expression () for correct parsing. 🤢

After all, prompt engineering can actually help quite a bit here, nevertheless it’s still not perfect. For instance, if you need to have ChatGPT analyze the sentiment of a movie review for positivity or negativity, you may structure a prompt that appears like this:

prompt = f'''
Please perform a sentiment evaluation on the next movie review:
{MOVIE_REVIEW_TEXT}
Please output your response as a single word: either "Positive" or "Negative".
'''

This prompt truthfully does pretty decently, but the outcomes aren’t precisely consistent. For instance, I even have seen ChatGPT produce outputs that appear to be the next when it comes to the movie sentiment example:

• Positive
• positive
• Positive.

This won’t appear to be a giant deal, but on the planet of programming, those are NOT equal. Again, you’ll be able to get around an easier example like this with a little bit of , but beyond the indisputable fact that most individuals (including myself) are terrible at writing regular expressions, there are simply some instances where even can’t parse the data appropriately.

As you’ll be able to tell, programmers have been hoping that OpenAI would add functionality to support structured JSON outputs, and OpenAI has delivered in the shape of function calling. Function calling is strictly because it sounds: it allows ChatGPT to provide arguments that may interact with a custom function in a way that uses structured data types. Yup, no more fancy prompt engineering and to cross your fingers and hope you get the correct consequence. On this post, we’ll cover how one can make use of this recent functionality, but first, let’s start with an example of how we used to try and produce structured data outputs with prompt engineering and .

Before we jump into the majority of our post, please allow me to share a link to this Jupyter notebook in my GitHub. This notebook accommodates all of the code I might be running (and more) as a part of this blog post. Moreover, I’d encourage you to take a look at OpenAI’s official function calling documentation for anything that I’ll not cover here.

To reveal what we used to do within the “pre-function calling days”, I wrote a small little bit of text about myself, and we’ll be using the OpenAPI to extract bits of knowledge from this text. Here is the “About Me” text we’ll be working with:

Hello! My name is David Hundley. I’m a principal machine learning engineer at State Farm. I enjoy learning about AI and teaching what I learn back to others. I even have two daughters. I drive a Tesla Model 3, and my favorite video game series is The Legend of Zelda.

Let’s say I would like to extract the next bits of knowledge from that text:

• Name
• Job title
• Company
• Number of kids as an integer (This is essential!)
• Automobile make
• Automobile model
• Favorite video game series

Here’s how I’d engineer a few-shot prompt with a view to produce a structured JSON output:

# Engineering a prompt to extract as much information from "About Me" as a JSON object
about_me_prompt = f'''
Please extract information as a JSON object. Please search for the next pieces of knowledge.
Name
Job title
Company
Number of kids as a single integer
Automobile make
Automobile model
Favorite video game series
That is the body of text to extract the data from:
{about_me}
'''
# Getting the response back from ChatGPT (gpt-3.5-turbo)
openai_response = openai.ChatCompletion.create(
model = 'gpt-3.5-turbo',
messages = [{'role': 'user', 'content': about_me_prompt}]
)
# Loading the response as a JSON object
json_response = json.loads(openai_response['choices'][0]['message']['content'])
json_response

Let’s take a look at how ChatGPT returned this completion to me:

As you’ll be able to see, this actually isn’t bad. But it surely’s not ideal and will prove to be dangerous for the next reasons:

• We aren’t guaranteed that OpenAI’s response will provide a clean JSON output. It could have produced something like “Here is your JSON:” followed by the JSON output, meaning that with a view to use json.loads() to parse the string right into a JSON object, we’d first should strip out that little little bit of text that opens the response.
• We aren’t guaranteed that the keys within the key-value pairs of the JSON object might be consistent from API call to API call. Recall the instance from above of the three instances of the word Positive. That is precisely the identical risk you run attempting to have ChatGPT parse out keys through few-shot prompt engineering. The one way you might perhaps lock this down is with , which comes with its own baggage as we already discussed.
• We aren’t guaranteed to receive our responses in the right data type format. While our prompt engineering to extract number of kids did parse right into a proper integer, we’re on the mercy of crossing our fingers and hoping we get that consistent result for each API call.

We could summarize these issues right into a single statement: Without function calling, we aren’t guaranteed to get consistent results which might be necessary for the precision required for systematic implementation. It’s a nontrivial issue that might be very difficult to treatment through prompt engineering and regular expressions.

Now that we’ve built an intuition around why getting structured outputs from ChatGPT was formerly problematic, let’s move into the brand new function calling capability introduced by OpenAI.

Function calling is definitely a little bit of a misnomer. OpenAI just isn’t actually running your code in a real function call. Relatively, it’s simply establishing the structured arguments you’d have to execute your personal custom functions, and I’d argue that is preferred behavior. When you is likely to be pondering that it doesn’t make sense that the OpenAI API isn’t executing your custom function, consider that with a view to do this, you’d should pass that function code into ChatGPT. This function code probably accommodates proprietary information that you just would NOT want to show to anybody, hence why it’s good that you just don’t actually should pass this code to utilize OpenAI’s function calling.

Let’s jump into an example of how one can enable function calling with a single custom function. Using our “About Me” sample text from the previous section, let’s create a custom function called extract_person_info. This function needs just three bits of knowledge: person name, job title, and number of kids. (We’ll revisit extracting the remainder of the data in the following section; I just want to start out simpler for now.) This tradition function is intentionally quite simple and can simply take our arguments and print them together in a single string. Here’s the code for this:

def extract_person_info(name, job_title, num_children):
'''
Prints basic "About Me" information
Inputs:
- name (str): Name of the person
- job_title (str): Job title of the person
- num_chilren (int): The number of kids the parent has.
'''
print(f'This person's name is {name}. Their job title is {job_title}, they usually have {num_children} children.')

With a view to make use of function calling, we want to establish a JSON object in a selected way that notes the name of our custom function and what data elements we hope ChatGPT will extract from the body of the text. Due to specificity on how this JSON object should look, I’d encourage you reference OpenAI’s developer documentation if you need to know any details that I don’t cover here.

(Note: Within the OpenAI documentation, I noticed one element within the JSON object called required that seemingly indicates that a parameter should be present for ChatGPT to properly recognize the function. I attempted testing this out, and either this isn’t how this functionality works or I did something improper. Either way, I transparently do not know what this required parameter indicates. 😅)

Here is how we want to structure our JSON object to utilize our custom function:

my_custom_functions = [
{
'name': 'extract_person_info',
'description': 'Get "About Me" information from the body of the input text',
'parameters': {
'type': 'object',
'properties': {
'name': {
'type': 'string',
'description': 'Name of the person'
},
'job_title': {
'type': 'string',
'description': 'Job title of the person'
},
'num_children': {
'type': 'integer',
'description': 'Number of children the person is a parent to'
}
}
}
}
]

You’re probably already accustomed to JSON syntax, although let me draw attention for a moment to the info type associated to every property. In the event you are a Python developer like myself, bear in mind that the info typing for this JSON structure is NOT directly akin to how we define data structures in Python. Generally speaking, we will find equivalencies that work out alright, but when you need to know more in regards to the specific data types related to this JSON structure, take a look at this documentation.

Now we’re able to make our API call to get the outcomes! Using the Python client, you’ll notice the syntax could be very just like how we obtain completions typically. We’re just going so as to add some additional arguments into this call that represent our function calling:

# Getting the response back from ChatGPT (gpt-3.5-turbo)
openai_response = openai.ChatCompletion.create(
model = 'gpt-3.5-turbo',
messages = [{'role': 'user', 'content': about_me}],
functions = my_custom_functions,
function_call = 'auto'
)
print(openai_response)

As you’ll be able to see, we simply pass in our list of custom functions (or in our case for now, our singular custom function) because the functions parameter, and also you’ll also notice an extra parameter called function_call that we’ve set to auto. Don’t worry about this for now as we’ll revisit what this auto piece is doing in the following section.

Let’s run this code and check out the total API response from ChatGPT

For essentially the most part, this response looks similar to a non-function call response, but now there’s an extra field within the response called function_call, and nested under this dictionary are two additional items: name and arguments. name indicates the name of our custom function that we might be calling with ChatGPT’s output, and arguments accommodates a string that we will load using json.loads() to load our custom function arguments as a JSON object.

Notice now that we’re getting far more consistency than we were in our pre-function calling methodology. Now we might be guaranteed that the keys of the key-value pairs WILL be consistent, and the info types WILL be consistent. No need for fancy prompt engineering or regular expressions!

That’s the core of OpenAI’s function calling! After all, this was a really simplistic example to get you going, but you most likely have additional questions. Let’s cover those on this next section.

The previous section covered a quite simple example of how one can enable function calling, but in case you’re like me, you most likely have some additional questions beyond this point. Naturally, I can’t cover all these questions, but I do need to cover two big ones which might be barely more advanced than what we covered within the previous section.

What if the prompt I submit doesn’t contain the data I would like to extract per my custom function?

In our original example, our custom function sought to extract three very specific bits of knowledge, and we demonstrated that this worked successfully by passing in my custom “About Me” text as a prompt. But you is likely to be wondering, what happens in case you pass in some other prompt that doesn’t contain that information?

Recall that we set a parameter in our API client call called function_call that we set to auto. We’ll explore this even deeper in the following subsection, but what this parameter is basically doing is telling ChatGPT to make use of its best judgment in determining when to structure the output for certainly one of our custom functions.

So what happens once we submit a prompt that doesn’t match any of our custom functions? Simply put, it defaults to typical behavior as if function calling doesn’t exist. Let’s test this out with an arbitrary prompt: “How tall is the Eiffel Tower?”

As you’ll be able to see, we’re getting a typical “Completions” output though we passed in our custom function. Naturally, this is smart since this arbitrary Eiffel Towel prompt accommodates none of the precise information we’re searching for.

What if I would like to pass multiple custom functions and a few of them have overlapping parameters?

Briefly, ChatGPT intelligently handles this with out a problem. Where we previously passed in a single custom function as essentially a listing of Python dictionaries, we just have to keep adding additional Python dictionaries to this same list, each representing its own distinct function. Let’s add two recent functions: one called extract_vehicle_info and one other called extract_all_info. Here’s what our adjusted syntax looks like:

# Defining a function to extract only vehicle information
def extract_vehicle_info(vehicle_make, vehicle_model):
'''
Prints basic vehicle information
Inputs:
- vehicle_make (str): Make of the vehicle
- vehicle_model (str): Model of the vehicle
'''
print(f'Vehicle make: {vehicle_make}nVehicle model: {vehicle_model}')
# Defining a function to extract all information provided in the unique "About Me" prompt
def extract_vehicle_info(name, job_title, num_children, vehicle_make, vehicle_model, company_name, favorite_vg_series):
'''
Prints the total "About Me" information
Inputs:
- name (str): Name of the person
- job_title (str): Job title of the person
- num_chilren (int): The number of kids the parent has
- vehicle_make (str): Make of the vehicle
- vehicle_model (str): Model of the vehicle
- company_name (str): Name of the corporate the person works for
- favorite_vg_series (str): Person's favorite video game series.
'''
print(f'''
This person's name is {name}. Their job title is {job_title}, they usually have {num_children} children.
They drive a {vehicle_make} {vehicle_model}.
They work for {company_name}.
Their favorite video game series is {favorite_vg_series}.
''')
# Defining how we wish ChatGPT to call our custom functions
my_custom_functions = [
{
'name': 'extract_person_info',
'description': 'Get "About Me" information from the body of the input text',
'parameters': {
'type': 'object',
'properties': {
'name': {
'type': 'string',
'description': 'Name of the person'
},
'job_title': {
'type': 'string',
'description': 'Job title of the person'
},
'num_children': {
'type': 'integer',
'description': 'Number of children the person is a parent to'
}
}
}
},
{
'name': 'extract_car_info',
'description': 'Extract the make and model of the person's car',
'parameters': {
'type': 'object',
'properties': {
'vehicle_make': {
'type': 'string',
'description': 'Make of the person's vehicle'
},
'vehicle_model': {
'type': 'string',
'description': 'Model of the person's vehicle'
}
}
}
},
{
'name': 'extract_all_info',
'description': 'Extract all information about a person including their vehicle make and model',
'parameters': {
'type': 'object',
'properties': {
'name': {
'type': 'string',
'description': 'Name of the person'
},
'job_title': {
'type': 'string',
'description': 'Job title of the person'
},
'num_children': {
'type': 'integer',
'description': 'Number of children the person is a parent to'
},
'vehicle_make': {
'type': 'string',
'description': 'Make of the person's vehicle'
},
'vehicle_model': {
'type': 'string',
'description': 'Model of the person's vehicle'
},
'company_name': {
'type': 'string',
'description': 'Name of the company the person works for'
},
'favorite_vg_series': {
'type': 'string',
'description': 'Name of the person's favorite video game series'
}
}
}
}
]

Notice specifically how the extract_all_info covers a number of the same parameters as our original extract_person_info function, so how does ChatGPT know which one to pick? Simply put, ChatGPT looks for the perfect match. If we pass in a prompt that accommodates all of the arguments needed for the extract_all_info function, that’s the one it’ll select. But when we just pass in a prompt that accommodates either just easy details about me or a prompt about my vehicle, it’ll leverage the respective functions that do this. Let’s execute that in code here with a couple of samples:

• Sample 1: The unique “About Me” text. (See above.)
• Sample 2: “My name is David Hundley. I’m a principal machine learning engineer, and I even have two daughters.”
• Sample 3: “She drives a Kia Sportage.”

With each of the respective prompts, ChatGPT chosen the proper custom function, and we will specifically note that within the name value under function_call within the API’s response object. Along with this being a handy strategy to discover which function to make use of the arguments for, we will programmatically map our actual custom Python function to this value to run the proper code appropriately. If that doesn’t make sense, perhaps this in code would make this more clear:

# Iterating over the three samples
for i, sample in enumerate(samples):
print(f'Sample #{i + 1}'s results:')
# Getting the response back from ChatGPT (gpt-3.5-turbo)
openai_response = openai.ChatCompletion.create(
model = 'gpt-3.5-turbo',
messages = [{'role': 'user', 'content': sample}],
functions = my_custom_functions,
function_call = 'auto'
)['choices'][0]['message']
# Checking to see that a function call was invoked
if openai_response.get('function_call'):
# Checking to see which specific function call was invoked
function_called = openai_response['function_call']['name']
# Extracting the arguments of the function call
function_args = json.loads(openai_response['function_call']['arguments'])
# Invoking the right functions
if function_called == 'extract_person_info':
extract_person_info(*list(function_args.values()))
elif function_called == 'extract_vehicle_info':
extract_vehicle_info(*list(function_args.values()))
elif function_called == 'extract_all_info':
extract_all_info(*list(function_args.values()))

**Beware one thing**: Within the spirit of full transparency, I needed to run that code there multiple times to get it to provide like that. The difficulty is that since the extract_person_info and extract_all_info are more similar in nature, ChatGPT kept confusing those for each other. I suppose the lesson to be learned here is that your functions ought to be extracting distinct information. I also only tested using gpt-3.5-turbo, so it’s possible that a more powerful model like GPT-4 could have handled that higher.

I hope you’ll be able to see now why function calling might be so powerful! In relation to constructing applications that leverage Generative AI, this sort of function calling is a godsend for programmers. By not having to fret a lot now in regards to the output JSON structure, we will now focus our time on constructing out other parts of the applying. It’s an awesome time to be working on this space 🥳