Multilingual language modeling within the Natural Language space

We humans use words, sounds, gestures and symbols to convey complex concepts and abstracts to one another in numerous forms equivalent to speech, writing and signs. With the arrival of computers and with the intention to make the most of these powerful machines, we needed to provide you with ways for computers to grasp human communications and the present corpus of information. Hence, the Natural Language Processing (NLP), Understanding (NLU) and Generation (NLG) branches of Artificial Intelligence were born. Boundaries of those three areas are sometimes unclear and the general Natural Language space encompasses various applications in today’s Computer and Data Science world. Probably probably the most common of such applications are (I) Sentiment Evaluation, (II) Machine Translation and (III) Named-Entity Recognition (NER), which we are going to define and implement on this post.

To be able to implement these three tasks, we are going to leverage existing “Pre-Trained Language Models”. Subsequently, let’s first understand what language modeling is. I do recommend glancing through the “Language Modeling” section but in the event you are mainly fascinated by the appliance and/or implementation, be happy to skip it.

Language modeling encompasses various methods that use statistics and probability, combined with machine learning architectures (equivalent to and particularly deep neural networks) to find out the likelihood of a sequence of words occurring in a string, equivalent to a sentence. Based on the calculated probability, certain decisions may be made — for instance, a model can generate a string/response to a user-provided prompt (equivalent to ChatGPT), perform a text classification to find out whether a word in query is a noun, a verb, etc. Because of the massive corpora of textual data available throughout us nowadays, such language models are often trained on vast amounts of textual data. Consequently, such models are also known as Large Language Models. At this point chances are you’ll be wondering how that is relevant to our post — we are only getting there. These pre-trained language models then may be further trained (i.e. fine-tuned) to perform specific tasks, equivalent to sentiment evaluation, machine translation and named-entity recognition, which we are going to explore in further detail today. Going deeper into the architecture and training strategies of language models is beyond the intent of this post, but in the event you are fascinated by that topic, be happy to go to the post below:

Now that we’re aware of what Natural Language space and Language Modeling are, let’s go to the fun a part of using these models!

Task of identifying the sentiment of a bit of text, equivalent to whether it’s positive, negative, or neutral, known as Sentiment Evaluation. It’s utilized in applications equivalent to social media monitoring, customer feedback evaluation, and product review evaluation. As you possibly can imagine, this one is kind of useful for loads of firms. For instance, a big online retail company wouldn’t find a way to dedicate the human resources required to manually read all of the comments about various products. As a substitute, they will run a Sentiment Evaluation model on the reviews and analyze the outcomes, saving money and time in the method. Next, let’s see how we are able to implement this.

1.1. Sentiment Evaluation — Implementation

In this instance, we first load a pre-trained langugae model from the Transformers library. Then, we use the model to generate sentiment from the input sentence. Finally, we test it on two different sentences, one positive and one negative, to confirm model’s performance. Below are the 2 sentences that we can be using:

• I loved this movie!, which we expect to be classified as a “Positive” sentiment by the model
• I didn't like this movie., which we expect to be classified as a “Negative” sentiment by the model

Let’s see how it really works!

# Import libraries
from transformers import pipeline
# Load the pre-trained model
nlp = pipeline('sentiment-analysis', model='distilbert-base-uncased-finetuned-sst-2-english')
# Define the function to perform sentiment evaluation
def sentiment_analyzer(input_text):
# Perform sentiment evaluation
result = nlp(input_text)[0]
# Return results
return f"'{input_text}' has a {result['label']} sentiment, with a rating of {round(result['score'], 4)}!n"
# Define example sentences
sentence_1 = "I loved this movie!"
sentence_2 = "I didn't like this movie."
sentence_list = [sentence_1, sentence_2]
# Analyze the sentiment of every sentence
for sentence in sentence_list:
print(sentiment_analyzer(sentence))

Results:

Results look pretty good and as we expected. If you happen to are fascinated by a more in-depth have a look at Sentiment Evaluation, the next post is for you:

Let’s speak about Machine Translation next.

Task of using computers to translate text from one language to a different known as Machine Translation. Probably the most well-known example for many users is Google Translate — what Google Translate does known as Machine Translation! Applications are plentiful. For instance, one can read and understand information in other languages.

2.1. Machine Translation — Implementation

To implement Machine Translation, we’re going to use mBART-50, developed by Facebook AI, from the Transformers library, which is a pre-trained language model for Machine Translation. The steps are very much like what we did within the Sentiment Evaluation and are as follows:

1. Install Transformers as follows: pip install transformers
2. Import the library
3. Load the pre-trained model
4. Run example sentences through the model and return the outcomes

What’s interesting about mBART-50 is that it’s a multilingual Machine Translation model, meaning it could possibly translate to and from different languages. Let’s test this capability in motion, where we translate one sentence to six different languages, using one model!

I emphasized the multilingual nature of this language model because historically, Neural Machine Translation models used to find a way to translate only from one specific language to a different specific language. For instance, we would want to coach one specific model for translation from English to French and train a separate model to translate from English to German. Then again, these multilingual machine translation models can translate from many languages to many languages using one single model, which is kind of impressive!

In the next code block, we import the libraries, load the models after which create the translator function, which accepts three arguments: (1) A sentence (source_sentence), (2) The language of the provided sentence (source_language) and (3) The language that we would love the sentence to be translated to (target_language). Then translator returns the interpretation as instructed.

# Import library
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast
# Load model and tokenizer
model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
def translator(source_sentence, source_language, target_language):
# Encode sentence
tokenizer.src_lang = source_language
input_ids = tokenizer(source_sentence, return_tensors="pt").input_ids
# Translate sentence
output_ids = model.generate(input_ids, forced_bos_token_id=tokenizer.lang_code_to_id[target_language])
translation = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
# return translation
return translation

Next, let’s test our function by translating Multilingual machine translation is impressive! to French, Spanish, Italian, German, Simplified Chinese and Japanese.

# Define sentence to be translated
original_sentence = 'Multilingual machine translation is impressive!'
# Define source language
english = "en_XX"
# Define goal languages
french = "fr_XX"
spanish = "es_XX"
italian = "it_IT"
german = "de_DE"
simplified_chinese = "zh_CN"
japanese = "ja_XX"
# Create an inventory of goal languages
target_list = [french, spanish, italian, german, simplified_chinese, japanese]
# Create a prompt list of lists
prompt_list = []
for goal in target_list:
prompt_list.append([original_sentence, english, target])
# Create translations
print(f"Generating machine translations for: n'{original_sentence}'n")
for i in enumerate(prompt_list):
translation = translator(source_sentence=i[1][0], source_language=i[1][1], target_language=i[1][2])
print(f"{i[1][2]}:")
print(f"{translation}n")

Results:

And we see the translations within the goal languages in the outcomes! I don’t personally speak any of those languages but I verified them using Google Translate and the translations seem accurate!

Last but not least, let’s have a look at Named-Entity Recognition.

Task of identifying and categorizing named entities in text and categorizing them into pre-defined classes known as Named-Entity Recognition or NER for brief. Some examples of those categories are: person names, locations, dates, organizations, numbers, etc. It’s possible you’ll be wondering why we would want such a model. NER has many applications within the Natural Language space. For instance, Visa, American Express, Amazon, etc. can use NER to discover and black-out sensitive information in a customer communication to guard customers’ sensitive information, equivalent to date of birth and bank card information. One other application for social media firms equivalent to Meta is identifying locations and individual names in comments/posts and using them for content advice.

Now that we understand what NER is, let’s implement it and have a look at the outcomes.

3.1. NER — Implementation

In this instance, we’re going to use spaCy pre-trained model for NER. The implementation is pretty straightforward. We are going to follow these steps:

1. Install and download the required data
2. Import the library
3. Load the pre-trained tasks, inclusive of NER
4. Run an example sentence through the model and return the outcomes

If you’ll want to install spaCy and download the information, you should use the next commands (source):

pip3 install spacy
python -m spacy download en_core_web_sm

I ran the installation above using the Command Line Interface. It is so simple as (I) opening the Terminal after which (II) running the above two lines. If you happen to need a more detailed Command Line tutorial, be happy to examine out the next:

Next, let’s implement and apply NER to the next sentence:Farzad wrote this Medium article in March 2023, using an Apple laptop, on a Jupyter notebook!

# Import library
import spacy
# Load English tokenizer, tagger, parser and NER
nlp = spacy.load("en_core_web_sm")
# Define example sentence
sentence = "Farzad wrote this Medium article in March 2023, using an Apple laptop, on a Jupyter notebook!"
# Apply NER
doc = nlp(sentence)
# Analyze syntax
print(f"Noun phrases:{[chunk.text for chunk in doc.noun_chunks]}n")
print("Verbs:", [token.lemma_ for token in doc if token.pos_ == "VERB"])
print("")
# Find named entities, phrases and ideas
for entity in doc.ents:
print(entity.text, entity.label_)

Results:

That’s quite interesting! Let’s talk in regards to the results. The primary line identified the nouns — I don’t personally agree with all of them but still, that is kind of impressive! The second line has appropriately identified “write” and “use” as verbs and the third block has identified “Medium” as a location, “March 2023” as a date, “Apple” as a corporation (this one is interesting since apple may be the name of a fruit however the model recognized the corporate name, presumably based on the context of the sentence) and “Jupyter” as an individual (this one needs some improvement). There are methods to further train these pre-trained models to make sure NER works more accurately for each use case but the purpose we desired to articulate here was to showcase how these pre-trained language models may be used to perform complicated tasks equivalent to NER with an inexpensive level of accuracy.

On this post, we briefly walked through the world of Natural Language Processing (NLP), Understanding (NLU) and Generation (NLG) and tried to grasp their importance by introducing and implementing a number of the commonest tasks throughout the Natural Language space, using language modeling. We then moved on to the introduction and language model implementation of (I) Sentiment Evaluation, (II) Machine Translation and (III) Named-Entity Recognition (NER) and checked out the impressive results of those powerful pre-trained language models in multiple languages.

If you happen to found this post helpful, please follow me on Medium and subscribe to receive my latest posts!

(All images, unless otherwise noted, are by the writer.)