I actually have began my evaluation by obtaining data from HuggingFace. The dataset is known as financial-reports-sec (This dataset has Apache License 2.0 and permits for industrial use), and in keeping with the dataset authors, it incorporates the annual reports of U.S. public corporations filing with the SEC EDGAR system from 1993–2020. Each annual report (10-K filing) is split into 20 sections.
Two relevant attributes of this data are useful for the present task:
- Sentence: Excerpts from the 10-K filing reports
- Section: Labels denoting the section of the 10-K filing that the sentence belongs to
I actually have focused on three sections:
- Business (Item 1): Describes the corporate’s business, including subsidiaries, markets, recent events, competition, regulations, and labor. Denoted by 0 in the information.
- Risk Aspects (Item 1A): Discusses risks that might impact the corporate, resembling external aspects, potential failures, and other disclosures to warn investors. Denoted by 1.
- Properties (Item 2): Details significant physical property assets. Doesn’t include mental or intangible assets. Denoted by 3.
For every label, I sampled 10 examples without alternative. The info is structured as follows:
Once the information is prepared, all I actually have to do is to make a classifier function that takes the sentence from the dataframe and predicts the label.
Role = '''
You're expert in SEC 10-K forms.
You might be presented by a text and you'll want to classify the text into either 'Item 1', 'Item 1A' or 'Item 2'.
The text only belongs to considered one of the mentioned categories so only return one category.
'''
def sec_classifier(text): response = openai.ChatCompletion.create(
model='gpt-4',
messages=[
{
"role": "system",
"content": Role},
{
"role": "user",
"content": text}],
temperature=0,
max_tokens=256,
top_p=1,
frequency_penalty=0,
presence_penalty=0)
return response['choices'][0]['message']['content']
I’m using GPT-4 here because it’s OpenAI’s most capable model up to now. I’ve also set the temperature to 0 simply to make certain the model doesn’t go off course. The really fun part is how I define the Role — that’s where I get to guide the model on what I need it to do. The Role tells it to remain focused and deliver the form of output I’m searching for. Defining a transparent role for the model helps it generate relevant, high-quality responses. The prompt on this function is:
You’re expert in SEC 10-K forms.
You might be presented by a text and you’ll want to classify the text into either ‘Item 1’, ‘Item 1A’ or ‘Item 2’.
The text only belongs to considered one of the mentioned categories so only return one category.
After applying the classification function across all data rows, I generated a classification report to judge model performance. The macro average F1 rating was 0.62, indicating reasonably strong predictive capabilities for this multi-class problem. For the reason that variety of examples was balanced across all 3 classes, the macro and weighted averages converged to the identical value. This baseline rating reflects the out-of-the-box accuracy of the pretrained model prior to any additional tuning or optimization.
precision recall f1-score supportItem 1 0.47 0.80 0.59 10
Item 1A 0.80 0.80 0.80 10
Item 2 1.00 0.30 0.46 10
accuracy 0.63 30
macro avg 0.76 0.63 0.62 30
weighted avg 0.76 0.63 0.62 30
As mentioned, few-shot learning is all about generalising the model with just a few good examples. To that end, I’ve modified my class by describing what Item 1, Item 1A and Item2 are (based on Wikipedia):
Role_fewshot = '''
You're expert in SEC 10-K forms.
You might be presented by a text and you'll want to classify the text into either 'Item 1', 'Item 1A' or 'Item 2'.
The text only belongs to considered one of the mentioned categories so only return one category.
In your classification take the next definitions into consideration: Item 1 (i.e. Business) describes the business of the corporate: who and what the corporate does, what subsidiaries it owns, and what markets it operates in.
It might also include recent events, competition, regulations, and labor issues. (Some industries are heavily regulated, have complex labor requirements, which have significant effects on the business.)
Other topics on this section may include special operating costs, seasonal aspects, or insurance matters.
Item 1A (i.e. Risk Aspects) is the section where the corporate lays anything that might go mistaken, likely external effects, possible future failures to satisfy obligations, and other risks disclosed to adequately warn investors and potential investors.
Item 2 (i.e. Properties) is the section that lays out the numerous properties, physical assets, of the corporate. This only includes physical kinds of property, not mental or intangible property.
Note: Only state the Item.
'''
def sec_classifier_fewshot(text):
response = openai.ChatCompletion.create(
model='gpt-4',
messages=[
{
"role": "system",
"content": Role_fewshot},
{
"role": "user",
"content": text}],
temperature=0,
max_tokens=256,
top_p=1,
frequency_penalty=0,
presence_penalty=0)
return response['choices'][0]['message']['content']
The prompt now reads:
You’re expert in SEC 10-K forms.
You might be presented by a text and you’ll want to classify the text into either ‘Item 1’, ‘Item 1A’ or ‘Item 2’.
The text only belongs to considered one of the mentioned categories so only return one category.
In your classification take the next definitions into consideration:Item 1 (i.e. Business) describes the business of the corporate: who and what the corporate does, what subsidiaries it owns, and what markets it operates in.
It might also include recent events, competition, regulations, and labor issues. (Some industries are heavily regulated, have complex labor requirements, which have significant effects on the business.)
Other topics on this section may include special operating costs, seasonal aspects, or insurance matters.Item 1A (i.e. Risk Aspects) is the section where the corporate lays anything that might go mistaken, likely external effects, possible future failures to satisfy obligations, and other risks disclosed to adequately warn investors and potential investors.
Item 2 (i.e. Properties) is the section that lays out the numerous properties, physical assets, of the corporate. This only includes physical kinds of property, not mental or intangible property.
If we run this on the texts we get the next performance:
precision recall f1-score supportItem 1 0.70 0.70 0.70 10
Item 1A 0.78 0.70 0.74 10
Item 2 0.91 1.00 0.95 10
accuracy 0.80 30
macro avg 0.80 0.80 0.80 30
weighted avg 0.80 0.80 0.80 30
The macro average F1 is now 0.80, that’s 29% improvement in our prediction, only by providing a very good description of every class.
Finally you possibly can see the complete dataset:
In actual fact the examples I provided gives the model concrete instances to learn from. Examples allow the model to infer patterns and features, by taking a look at multiple examples, the model can begin to notice commonalities and differences that characterise the general concept being learned. This helps the model form a more robust representation. Moreover, providing examples essentially acts as a weak type of supervision, guiding the model towards the specified behaviour in lieu of enormous labeled datasets.
Within the few-shot function, concrete examples help point the model to the kinds of information and patterns it should listen to. In summary, concrete examples are essential for few-shot learning as they supply anchor points for the model to construct an initial representation of a novel concept, which may then be refined over the few examples provided. The inductive learning from specific instances helps models develop nuanced representations of abstract concepts.
If you happen to’ve enjoyed reading this and need to keep up a correspondence, you will discover me on my LinkedIn or via my webpage: iliateimouri.com
Note: All images, unless otherwise noted, are by the creator.
