Getting Began with Sentiment Evaluation using Python

Sentiment evaluation is the automated strategy of tagging data in keeping with their sentiment, similar to positive, negative and neutral. Sentiment evaluation allows firms to investigate data at scale, detect insights and automate processes.

Previously, sentiment evaluation was limited to researchers, machine learning engineers or data scientists with experience in natural language processing. Nonetheless, the AI community has built awesome tools to democratize access to machine learning in recent times. Nowadays, you should utilize sentiment evaluation with a number of lines of code and no machine learning experience in any respect! 🤯

On this guide, you will learn the whole lot to start with sentiment evaluation using Python, including:

1. What’s sentiment evaluation?
2. Learn how to use pre-trained sentiment evaluation models with Python
3. Learn how to construct your personal sentiment evaluation model
4. Learn how to analyze tweets with sentiment evaluation

Let’s start! 🚀

1. What’s Sentiment Evaluation?

Sentiment evaluation is a natural language processing technique that identifies the polarity of a given text. There are different flavors of sentiment evaluation, but one of the vital widely used techniques labels data into positive, negative and neutral. For instance, let’s take a have a look at these tweets mentioning @VerizonSupport:

• “dear @verizonsupport your service is straight 💩 in dallas.. been with y’throughout a decade and that is all time low for y’all. i’m talking no web in any respect.” → Can be tagged as “Negative”.

• “@verizonsupport ive sent you a dm” → could be tagged as “Neutral”.

• “because of michelle et al at @verizonsupport who helped push my no-show-phone problem along. order canceled successfully and ordered this for pickup today on the apple store within the mall.” → could be tagged as “Positive”.

Sentiment evaluation allows processing data at scale and in real-time. For instance, do you should analyze 1000’s of tweets, product reviews or support tickets? As an alternative of sorting through this data manually, you should utilize sentiment evaluation to robotically understand how individuals are talking about a particular topic, get insights for data-driven decisions and automate business processes.

Sentiment evaluation is utilized in a wide selection of applications, for instance:

• Analyze social media mentions to grasp how individuals are talking about your brand vs your competitors.
• Analyze feedback from surveys and product reviews to quickly get insights into what your customers like and dislike about your product.
• Analyze incoming support tickets in real-time to detect offended customers and act accordingly to forestall churn.

2. Learn how to Use Pre-trained Sentiment Evaluation Models with Python

Now that we’ve covered what sentiment evaluation is, we’re able to play with some sentiment evaluation models! 🎉

On the Hugging Face Hub, we’re constructing the biggest collection of models and datasets publicly available so as to democratize machine learning 🚀. Within the Hub, yow will discover greater than 27,000 models shared by the AI community with state-of-the-art performances on tasks similar to sentiment evaluation, object detection, text generation, speech recognition and more. The Hub is free to make use of and most models have a widget that permits to check them directly in your browser!

There are greater than 215 sentiment evaluation models publicly available on the Hub and integrating them with Python just takes 5 lines of code:

pip install -q transformers
from transformers import pipeline
sentiment_pipeline = pipeline("sentiment-analysis")
data = ["I love you", "I hate you"]
sentiment_pipeline(data)

This code snippet uses the pipeline class to make predictions from models available within the Hub. It uses the default model for sentiment evaluation to investigate the list of texts data and it outputs the next results:

[{'label': 'POSITIVE', 'score': 0.9998},
 {'label': 'NEGATIVE', 'score': 0.9991}]

You should use a particular sentiment evaluation model that is best suited to your language or use case by providing the name of the model. For instance, for those who need a sentiment evaluation model for tweets, you’ll be able to specify the model id:

specific_model = pipeline(model="finiteautomata/bertweet-base-sentiment-analysis")
specific_model(data)

You possibly can test these models together with your own data using this Colab notebook:

The next are some popular models for sentiment evaluation models available on the Hub that we recommend trying out:

• Twitter-roberta-base-sentiment is a roBERTa model trained on ~58M tweets and fine-tuned for sentiment evaluation. Superb-tuning is the strategy of taking a pre-trained large language model (e.g. roBERTa on this case) after which tweaking it with additional training data to make it perform a second similar task (e.g. sentiment evaluation).
• Bert-base-multilingual-uncased-sentiment is a model fine-tuned for sentiment evaluation on product reviews in six languages: English, Dutch, German, French, Spanish and Italian.
• Distilbert-base-uncased-emotion is a model fine-tuned for detecting emotions in texts, including sadness, joy, love, anger, fear and surprise.

Are you concerned with doing sentiment evaluation in languages similar to Spanish, French, Italian or German? On the Hub, you will discover many models fine-tuned for various use cases and ~28 languages. You possibly can try the whole list of sentiment evaluation models here and filter on the left in keeping with the language of your interest.

3. Constructing Your Own Sentiment Evaluation Model

Using pre-trained models publicly available on the Hub is a fantastic strategy to start straight away with sentiment evaluation. These models use deep learning architectures similar to transformers that achieve state-of-the-art performance on sentiment evaluation and other machine learning tasks. Nonetheless, you’ll be able to fine-tune a model together with your own data to further improve the sentiment evaluation results and get an additional boost of accuracy in your particular use case.

On this section, we’ll go over two approaches on find out how to fine-tune a model for sentiment evaluation together with your own data and criteria. The primary approach uses the Trainer API from the 🤗Transformers, an open source library with 50K stars and 1K+ contributors and requires a bit more coding and experience. The second approach is a bit easier and more straightforward, it uses AutoNLP, a tool to robotically train, evaluate and deploy state-of-the-art NLP models without code or ML experience.

Let’s dive in!

a. Superb-tuning model with Python

On this tutorial, you will use the IMDB dataset to fine-tune a DistilBERT model for sentiment evaluation.

The IMDB dataset incorporates 25,000 movie reviews labeled by sentiment for training a model and 25,000 movie reviews for testing it. DistilBERT is a smaller, faster and cheaper version of BERT. It’s 40% smaller than BERT and runs 60% faster while preserving over 95% of BERT’s performance. You may use the IMDB dataset to fine-tune a DistilBERT model that’s in a position to classify whether a movie review is positive or negative. When you train the model, you’ll use it to investigate recent data! ⚡️

We now have created this notebook so you should utilize it through this tutorial in Google Colab.

1. Activate GPU and Install Dependencies

As a primary step, let’s arrange Google Colab to make use of a GPU (as an alternative of CPU) to coach the model much faster. You possibly can do that by going to the menu, clicking on ‘Runtime’ > ‘Change runtime type’, and choosing ‘GPU’ because the Hardware accelerator. When you do that, it is best to check if GPU is on the market on our notebook by running the next code:

import torch
torch.cuda.is_available()

Then, install the libraries you will likely be using on this tutorial:

!pip install datasets transformers huggingface_hub

It’s best to also install git-lfs to make use of git in our model repository:

!apt-get install git-lfs

2. Preprocess data

You wish data to fine-tune DistilBERT for sentiment evaluation. So, let’s use 🤗Datasets library to download and preprocess the IMDB dataset so you’ll be able to then use this data for training your model:

from datasets import load_dataset
imdb = load_dataset("imdb")

IMDB is a big dataset, so let’s create smaller datasets to enable faster training and testing:

small_train_dataset = imdb["train"].shuffle(seed=42).select([i for i in list(range(3000))])
small_test_dataset = imdb["test"].shuffle(seed=42).select([i for i in list(range(300))])

To preprocess our data, you’ll use DistilBERT tokenizer:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

Next, you’ll prepare the text inputs for the model for each splits of our dataset (training and test) through the use of the map method:

def preprocess_function(examples):
   return tokenizer(examples["text"], truncation=True)
 
tokenized_train = small_train_dataset.map(preprocess_function, batched=True)
tokenized_test = small_test_dataset.map(preprocess_function, batched=True)

To hurry up training, let’s use a data_collator to convert your training samples to PyTorch tensors and concatenate them with the correct quantity of padding:

from transformers import DataCollatorWithPadding
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

3. Training the model

Now that the preprocessing is finished, you’ll be able to go ahead and train your model 🚀

You will likely be throwing away the pretraining head of the DistilBERT model and replacing it with a classification head fine-tuned for sentiment evaluation. This lets you transfer the knowledge from DistilBERT to your custom model 🔥

For training, you will likely be using the Trainer API, which is optimized for fine-tuning Transformers🤗 models similar to DistilBERT, BERT and RoBERTa.

First, let’s define DistilBERT as your base model:

from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=2)

Then, let’s define the metrics you will likely be using to guage how good your fine-tuned model is (accuracy and f1 rating):

import numpy as np
from datasets import load_metric
 
def compute_metrics(eval_pred):
   load_accuracy = load_metric("accuracy")
   load_f1 = load_metric("f1")
  
   logits, labels = eval_pred
   predictions = np.argmax(logits, axis=-1)
   accuracy = load_accuracy.compute(predictions=predictions, references=labels)["accuracy"]
   f1 = load_f1.compute(predictions=predictions, references=labels)["f1"]
   return {"accuracy": accuracy, "f1": f1}

Next, let’s login to your Hugging Face account so you’ll be able to manage your model repositories. notebook_login will launch a widget in your notebook where you will need so as to add your Hugging Face token:

from huggingface_hub import notebook_login
notebook_login()

You’re almost there! Before training our model, you should define the training arguments and define a Trainer with all of the objects you constructed up up to now:

from transformers import TrainingArguments, Trainer
 
repo_name = "finetuning-sentiment-model-3000-samples"
 
training_args = TrainingArguments(
   output_dir=repo_name,
   learning_rate=2e-5,
   per_device_train_batch_size=16,
   per_device_eval_batch_size=16,
   num_train_epochs=2,
   weight_decay=0.01,
   save_strategy="epoch",
   push_to_hub=True,
)
 
trainer = Trainer(
   model=model,
   args=training_args,
   train_dataset=tokenized_train,
   eval_dataset=tokenized_test,
   tokenizer=tokenizer,
   data_collator=data_collator,
   compute_metrics=compute_metrics,
)

Now, it is time to fine-tune the model on the sentiment evaluation dataset! 🙌 You simply must call the train() approach to your Trainer:

trainer.train()

And voila! You fine-tuned a DistilBERT model for sentiment evaluation! 🎉

Training time relies on the hardware you utilize and the variety of samples within the dataset. In our case, it took almost 10 minutes using a GPU and fine-tuning the model with 3,000 samples. The more samples you utilize for training your model, the more accurate it’ll be but training could possibly be significantly slower.

Next, let’s compute the evaluation metrics to see how good your model is:

trainer.evaluate()

In our case, we got 88% accuracy and 89% f1 rating. Quite good for a sentiment evaluation model just trained with 3,000 samples!

4. Analyzing recent data with the model

Now that you’ve gotten trained a model for sentiment evaluation, let’s use it to investigate recent data and get 🤖 predictions! This unlocks the ability of machine learning; using a model to robotically analyze data at scale, in real-time ⚡️

First, let’s upload the model to the Hub:

trainer.push_to_hub()

Now that you’ve gotten pushed the model to the Hub, you should utilize it pipeline class to investigate two recent movie reviews and see how your model predicts its sentiment with just two lines of code 🤯:

from transformers import pipeline
 
sentiment_model = pipeline(model="federicopascual/finetuning-sentiment-model-3000-samples")
sentiment_model(["I love this movie", "This movie sucks!"])

These are the predictions from our model:

[{'label': 'LABEL_1', 'score': 0.9558},
 {'label': 'LABEL_0', 'score': 0.9413}]

Within the IMDB dataset, Label 1 means positive and Label 0 is negative. Quite good! 🔥

b. Training a sentiment model with AutoNLP

AutoNLP is a tool to coach state-of-the-art machine learning models without code. It provides a friendly and easy-to-use user interface, where you’ll be able to train custom models by simply uploading your data. AutoNLP will robotically fine-tune various pre-trained models together with your data, deal with the hyperparameter tuning and find one of the best model in your use case. All models trained with AutoNLP are deployed and prepared for production.

Training a sentiment evaluation model using AutoNLP is super easy and it just takes a number of clicks 🤯. Let’s give it a try!

As a primary step, let’s get some data! You may use Sentiment140, a well-liked sentiment evaluation dataset that consists of Twitter messages labeled with 3 sentiments: 0 (negative), 2 (neutral), and 4 (positive). The dataset is sort of big; it incorporates 1,600,000 tweets. As you do not need this amount of knowledge to get your feet wet with AutoNLP and train your first models, we’ve prepared a smaller version of the Sentiment140 dataset with 3,000 samples that you would be able to download from here. That is how the dataset looks like:

Next, let’s create a recent project on AutoNLP to coach 5 candidate models:

Then, upload the dataset and map the text column and goal columns:

When you add your dataset, go to the “Trainings” tab and accept the pricing to start out training your models. AutoNLP pricing could be as little as $10 per model:

After a number of minutes, AutoNLP has trained all models, showing the performance metrics for all of them:

One of the best model has 77.87% accuracy 🔥 Pretty good for a sentiment evaluation model for tweets trained with just 3,000 samples!

All these models are robotically uploaded to the Hub and deployed for production. You should use any of those models to start out analyzing recent data straight away through the use of the pipeline class as shown in previous sections of this post.

4. Analyzing Tweets with Sentiment Evaluation and Python

On this last section, you will take what you’ve gotten learned to this point on this post and put it into practice with a fun little project: analyzing tweets about NFTs with sentiment evaluation!

First, you will use Tweepy, an easy-to-use Python library for getting tweets mentioning #NFTs using the Twitter API. Then, you’ll use a sentiment evaluation model from the 🤗Hub to investigate these tweets. Finally, you’ll create some visualizations to explore the outcomes and find some interesting insights.

You should use this notebook to follow this tutorial. Let’s jump into it!

1. Install dependencies

First, let’s install all of the libraries you’ll use on this tutorial:

!pip install -q transformers tweepy wordcloud matplotlib

2. Arrange Twitter API credentials

Next, you’ll arrange the credentials for interacting with the Twitter API. First, you will need to join a developer account on Twitter. Then, you’ve gotten to create a brand new project and connect an app to get an API key and token. You possibly can follow this step-by-step guide to get your credentials.

Once you’ve gotten the API key and token, let’s create a wrapper with Tweepy for interacting with the Twitter API:

import tweepy
 

consumer_key = "XXXXXX"
consumer_secret = "XXXXXX"
 

auth = tweepy.AppAuthHandler(consumer_key, consumer_secret)
 

api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)

3. Seek for tweets using Tweepy

At this point, you’re ready to start out using the Twitter API to gather tweets 🎉. You’ll use Tweepy Cursor to extract 1,000 tweets mentioning #NFTs:

def limit_handled(cursor):
   while True:
       try:
           yield cursor.next()
       except tweepy.RateLimitError:
           print('Reached rate limite. Sleeping for >quarter-hour')
           time.sleep(15 * 61)
       except StopIteration:
           break
 

query = '#NFTs'
query = query + ' -filter:retweets'
 

count = 1000
 

search = limit_handled(tweepy.Cursor(api.search,
                       q=query,
                       tweet_mode='prolonged',
                       lang='en',
                       result_type="recent").items(count))

4. Run sentiment evaluation on the tweets

Now you’ll be able to put our recent skills to work and run sentiment evaluation in your data! 🎉

You’ll use one among the models available on the Hub fine-tuned for sentiment evaluation of tweets. Like in other sections of this post, you’ll use the pipeline class to make the predictions with this model:

from transformers import pipeline
 

sentiment_analysis = pipeline(model="finiteautomata/bertweet-base-sentiment-analysis")
 

tweets = []
for tweet in search:
   try:
     content = tweet.full_text
     sentiment = sentiment_analysis(content)
     tweets.append({'tweet': content, 'sentiment': sentiment[0]['label']})
 
   except:
     pass

5. Explore the outcomes of sentiment evaluation

How are people talking about NFTs on Twitter? Are they talking mostly positively or negatively? Let’s explore the outcomes of the sentiment evaluation to seek out out!

First, let’s load the outcomes on a dataframe and see examples of tweets that were labeled for every sentiment:

import pandas as pd
 

df = pd.DataFrame(tweets)
pd.set_option('display.max_colwidth', None)
 

display(df[df["sentiment"] == 'POS'].head(1))
display(df[df["sentiment"] == 'NEU'].head(1))
display(df[df["sentiment"] == 'NEG'].head(1))

Output:

Tweet: @NFTGalIery Warm, exquisite and chic palette of charming beauty Its price is 2401 ETH. nhttps://t.co/Ej3BfVOAqcn#NFTs #NFTartists #art #Bitcoin #Crypto #OpenSeaNFT #Ethereum #BTC	Sentiment: POS

Tweet: How much our followers made on #Crypto in December:n#DAPPRadar airdrop — $200nFree #VPAD tokens — $800n#GasDAO airdrop — as much as $1000nStarSharks_SSS IDO — $3500nCeloLaunch IDO — $3000n12 Binance XMas #NFTs — $360 nTOTAL PROFIT: $8500+nnJoin and earn with us https://t.co/fS30uj6SYx	Sentiment: NEU

Tweet: Silly guy #2nhttps://t.co/8yKzYjCYIlnn#NFT #NFTs #nftcollector #rarible https://t.co/O4V19gMmVk		Sentiment: NEG

Then, let’s have a look at what number of tweets you bought for every sentiment and visualize these results:

sentiment_counts = df.groupby(['sentiment']).size()
print(sentiment_counts)


fig = plt.figure(figsize=(6,6), dpi=100)
ax = plt.subplot(111)
sentiment_counts.plot.pie(ax=ax, autopct='%1.1f%%', startangle=270, fontsize=12, label="")

Interestingly, many of the tweets about NFTs are positive (56.1%) and almost none are negative
(2.0%):

Finally, let’s have a look at what words stand out for every sentiment by making a word cloud:

from wordcloud import WordCloud
from wordcloud import STOPWORDS
 

positive_tweets = df['tweet'][df["sentiment"] == 'POS']
stop_words = ["https", "co", "RT"] + list(STOPWORDS)
positive_wordcloud = WordCloud(max_font_size=50, max_words=100, background_color="white", stopwords = stop_words).generate(str(positive_tweets))
plt.figure()
plt.title("Positive Tweets - Wordcloud")
plt.imshow(positive_wordcloud, interpolation="bilinear")
plt.axis("off")
plt.show()
 

negative_tweets = df['tweet'][df["sentiment"] == 'NEG']
stop_words = ["https", "co", "RT"] + list(STOPWORDS)
negative_wordcloud = WordCloud(max_font_size=50, max_words=100, background_color="white", stopwords = stop_words).generate(str(negative_tweets))
plt.figure()
plt.title("Negative Tweets - Wordcloud")
plt.imshow(negative_wordcloud, interpolation="bilinear")
plt.axis("off")
plt.show()

A few of the words related to positive tweets include Discord, Ethereum, Join, Mars4 and Shroom:

In contrast, words related to negative tweets include: cookies chaos, Solana, and OpenseaNFT:

And that’s it! With just a number of lines of python code, you were in a position to collect tweets, analyze them with sentiment evaluation and create some cool visualizations to investigate the outcomes! Pretty cool, huh?

5. Wrapping up

Sentiment evaluation with Python has never been easier! Tools similar to 🤗Transformers and the 🤗Hub makes sentiment evaluation accessible to all developers. You should use open source, pre-trained models for sentiment evaluation in only a number of lines of code 🔥

Do you should train a custom model for sentiment evaluation together with your own data? Easy peasy! You possibly can fine-tune a model using Trainer API to construct on top of huge language models and get state-of-the-art results. When you want something even easier, you should utilize AutoNLP to coach custom machine learning models by simply uploading data.

If you’ve gotten questions, the Hugging Face community can assist answer and/or profit from, please ask them within the Hugging Face forum. Also, join our discord server to speak with us and with the Hugging Face community.