When starting work with a brand new dataset, it’s all the time a great idea to begin with some **exploratory data evaluation** (EDA). Taking the time to know your data before training any fancy models can enable you to understand the structure of the dataset, discover any obvious issues, and apply domain-specific knowledge.

You see EDA in various forms with the whole lot from house prices to advanced applications in the information science industry. But I still haven’t seen it for the most popular recent dataset: **word embeddings**, the idea of our greatest large language models. So why not try it?

In this text, we’ll **apply EDA to GloVe word embeddings**, using techniques like **covariance matrices, clustering, PCA, and vector math**. This can help us understand the structure of word embeddings, giving us a useful place to begin for constructing more powerful models with this data. As we discover this structure, we’ll find that it’s not all the time what it seems, and a few surprising biases are hidden within the corpus.

You have to:

- Basic understanding of linear algebra, statistics, and vector mathematics
- Python packages:
`numpy`

,`sklearn`

, and`matplotlib`

- About 3 GB of spare disk space

To start, download the dataset at huggingface.co/stanfordnlp/glove/resolve/most important/glove.6B.zip[1]. This incorporates three text files, each containing an inventory of words together with their vector representations. We are going to use the **300-dimensional representations** (glove.6B.300d.txt).

A fast note on where this dataset comes from: essentially, it is a list of word embeddings derived from 6 billion tokens’ price of **co-occurrence data** from Wikipedia and various news sources. A useful side effect of using co-occurrence is that **words that mean similar things are inclined to be close together**. For instance, since “the *red* bird” and “the *blue* bird” are each valid sentences, we would expect the vectors for “red” and “blue” to be close to one another. For more technical information, you possibly can check the unique GloVe paper[1].

To be clear, these are **not** word embeddings trained for the aim of huge language models. They’re a completely unsupervised technique based on a big corpus. But they display quite a lot of similar properties to language model embeddings, and are interesting in their very own right.

Each line of this text file consists of a word, followed by all 300 vector components of the associated embedding separated by spaces. We are able to load that in with Python. (To cut back noise and speed things up, I’m using the highest 10% of the complete dataset here with the `//10`

, but you possibly can change that when you’d like.)

`import numpy as np`embeddings = {}

with open(f"glove.6B/glove.6B.300d.txt", "r") as f:

glove_content = f.read().split('n')

for i in range(len(glove_content)//10):

line = glove_content[i].strip().split(' ')

if line[0] == '':

proceed

word = line[0]

embedding = np.array(list(map(float, line[1:])))

embeddings[word] = embedding

print(len(embeddings))

That leaves us with 40,000 embeddings loaded in.

One natural query we would ask is: **are vectors generally near other vectors with similar meaning?** And as a follow-up query, how will we quantify this?

There are two most important ways we are going to quantify similarity between vectors: one is **Euclidean distance**, which is solely the natural Pythagorean theorem distance we’re acquainted with. The opposite is **cosine similarity**, which measures the cosine of the *angle* between two vectors. A vector has a cosine similarity of 1 with itself, -1 with an opposite vector, and 0 with an orthogonal vector.

Let’s implement these in NumPy:

`def cos_sim(a, b):`

return np.dot(a,b)/(np.linalg.norm(a) * np.linalg.norm(b))

def euc_dist(a, b):

return np.sum(np.square(a - b)) # no need for square root since we are only rating distances

Now we are able to find all of the closest vectors to a given word or embedding vector! We’ll do that in increasing order.

`def get_sims(to_word=None, to_e=None, metric=cos_sim):`

# list all similarities to the word to_word, OR the embedding vector to_e

assert (to_word is just not None) ^ (to_e is just not None) # find similarity to a word or a vector, not each

sims = []

if to_e is None:

to_e = embeddings[to_word] # get the embedding for the word we're

for word in embeddings:

if word == to_word:

proceed

word_e = embeddings[word]

sim = metric(word_e, to_e)

sims.append((sim, word))

sims.sort()

return sims

Now we are able to write a function to display the ten most similar words. It’s going to be useful to incorporate a reverse option as well, so we are able to display the *least* similar words.

`def display_sims(to_word=None, to_e=None, n=10, metric=cos_sim, reverse=False, label=None):`

assert (to_word is just not None) ^ (to_e is just not None)

sims = get_sims(to_word=to_word, to_e=to_e, metric=metric)

display = lambda sim: f'{sim[1]}: {sim[0]:.5f}'

if label is None:

label = to_word.upper() if to_word is just not None else ''

print(label) # a heading so we all know what these similarities are for

if reverse:

sims.reverse()

for i, sim in enumerate(reversed(sims[-n:])):

print(i+1, display(sim))

return sims

Finally, we are able to test it!

`display_sims(to_word='red')`

# yellow, blue, pink, green, white, purple, black, coloured, sox, shiny

Looks just like the Boston Red Sox made an unexpected appearance here. But aside from that, that is about what we might expect.

Possibly we are able to try some verbs, and not only nouns and adjectives? How a few nice and type verb like “share”?

`display_sims(to_word='share')`

# shares, stock, profit, percent, shared, earnings, profits, price, gain, cents

I suppose “share” isn’t often used as a verb on this dataset. Oh well.

We are able to try some more conventional examples as well:

`display_sims(to_word='cat')`

# dog, cats, pet, dogs, feline, monkey, horse, pets, rabbit, leopard

display_sims(to_word='frog')

# toad, frogs, snake, monkey, squirrel, species, rodent, parrot, spider, rat

display_sims(to_word='queen')

# elizabeth, princess, king, monarch, royal, majesty, victoria, throne, lady, crown

Considered one of the fascinating properties about word embeddings is that **analogy is inbuilt** using vector math. The instance from the GloVe paper is *king – queen = man – woman.* In other words, rearranging the equation, we expect *king = man – woman + queen*. Is that this true?

`display_sims(to_e=embeddings['man'] - embeddings['woman'] + embeddings['queen'], label='king-queen analogy')`

# queen, king, ii, majesty, monarch, prince...

Not quite: the closest vector to *man – woman + queen* seems to be *queen* (cosine similarity 0.78), followed somewhat distantly by *king* (cosine similarity 0.66). Inspired by this excellent 3Blue1Brown video, we would try *aunt* and *uncle* as an alternative:

`display_sims(to_e=embeddings['aunt'] - embeddings['woman'] + embeddings['man'], label='aunt-uncle analogy')`

# aunt, uncle, brother, grandfather, grandmother, cousin, uncles, grandpa, dad, father

This is healthier (cosine similarity 0.7348 vs 0.7344), but still doesn’t work perfectly. But we are able to try switching to Euclidean distance. Now we’d like to set `reverse=True`

, because a *higher* Euclidean distance is definitely a *lower* similarity.

`display_sims(to_e=embeddings['aunt'] - embeddings['woman'] + embeddings['man'], metric=euc_dist, reverse=True, label='aunt-uncle analogy')`

# uncle, aunt, grandfather, brother, cousin, grandmother, newphew, dad, grandpa, cousins

Now we got it. Nevertheless it looks like the analogy math may not be as perfect as we hoped, at the least within the naïve way that we’re doing it here.

Cosine similarity is all concerning the *angles* between vectors. But is the **magnitude** of a vector also necessary?

We are able to reuse our existing code by expressing magnitude because the Euclidean distance from the zero vector. Let’s see which words have the biggest and smallest magnitudes:

`zero_vec = np.zeros_like(embeddings['the'])`

display_sims(to_e=zero_vec, metric=euc_dist, label='largest magnitude')

# republish, nonsubscribers, hushen, tael, www.star, stoxx, 202-383-7824, resend, non-families, 225-issue

display_sims(to_e=zero_vec, metric=euc_dist, reverse=True, label='smallest magnitude')

# likewise, lastly, interestingly, sarcastically, incidentally, furthermore, conversely, moreover, aforementioned, wherein

It doesn’t seem like there’s much of a pattern to the meaning of the big magnitude vectors, but all of them appear to have very specific (and sometimes confusing) meanings. Then again, the smallest magnitude vectors are inclined to be quite common words that could be present in quite a lot of contexts.

There’s a **huge range between magnitudes**: from about 2.6 for the smallest vector all of the technique to about 17 for the biggest. What does this distribution seem like? We are able to plot a **histogram** to get a greater picture of this.

`import matplotlib.pyplot as plt`def plot_magnitudes():

words = [w for w in embeddings]

magnitude = lambda word: np.linalg.norm(embeddings[word])

magnitudes = list(map(magnitude, words))

plt.hist(magnitudes, bins=40)

plt.show()

plot_magnitudes()

This distribution looks roughly normal. If we desired to test this further, we could use a Q-Q plot. But for our purposes at once, that is high-quality.

It seems that directions and subspaces in vector embeddings can encode various sorts of concepts, often in biased ways. This paper[2] studied how this works for **gender bias**.

We are able to replicate this idea in our GloVe embeddings, too. First, let’s find the direction of the concept of “masculinity”. We are able to accomplish this by taking the typical of **differences between vectors** like *he* and *she*, *man* and *woman*, and so forth:

`gender_pairs = [('man', 'woman'), ('men', 'women'), ('brother', 'sister'), ('he', 'she'),`

('uncle', 'aunt'), ('grandfather', 'grandmother'), ('boy', 'girl'),

('son', 'daughter')]

masc_v = zero_vec

for pair in gender_pairs:

masc_v += embeddings[pair[0]]

masc_v -= embeddings[pair[1]]

Now we are able to find the “most masculine” and “most feminine” vectors, as judged by the embedding space.

`display_sims(to_e=masc_v, metric=cos_sim, label='masculine vecs')`

# brother, colonel, himself, uncle, gen., nephew, brig., brothers, son, sir

display_sims(to_e=masc_v, metric=cos_sim, reverse=True, label='feminine vecs')

# actress, herself, businesswoman, chairwoman, pregnant, she, her, sister, actresses, woman

Now, we are able to run a simple test to detect bias within the dataset: compute the similarity between *nurse* and every of *man* and *woman*. Theoretically, these needs to be about equal: nurse is just not a gendered word. Is that this true?

`print("nurse - man", cos_sim(embeddings['nurse'], embeddings['man'])) # 0.24`

print("nurse - woman", cos_sim(embeddings['nurse'], embeddings['woman'])) # 0.45

That’s a reasonably large difference! (Remember cosine similarity runs from -1 to 1, with positive associations within the range 0 to 1.) For reference, 0.45 can be near the cosine similarity between *cat* and *leopard*.

Let’s see if we are able to **cluster words with similar meaning** using ** k-means clustering**. This is straightforward to do with the package

`scikit-learn`

. We’re going to use 300 clusters, which appears like loads, but trust me: almost all the clusters are so interesting, you may write a complete article just interpreting them!`from sklearn.cluster import KMeans`def get_kmeans(n=300):

kmeans = KMeans(n_clusters=n, n_init=1)

X = np.array([embeddings[w] for w in embeddings])

kmeans.fit(X)

return kmeans

def display_kmeans(kmeans):

# print all clusters and 5 associated words for every

words = np.array([w for w in embeddings])

X = np.array([embeddings[w] for w in embeddings])

y = kmeans.predict(X) # get the cluster for every word

for cluster in range(kmeans.cluster_centers_.shape[0]):

print(f'KMeans {cluster}')

cluster_words = words[y == cluster] # get all words in each cluster

for i, w in enumerate(cluster_words[:5]):

print(i+1, w)

kmeans = get_kmeans()

display_kmeans(kmeans)

There’s loads to take a look at here. We’ve clusters for things as diverse as Recent York City (`manhattan, n.y., brooklyn, hudson, borough`

), molecular biology (`protein, proteins, enzyme, beta, molecules`

), and Indian names (`singh, ram, gandhi, kumar, rao`

).

But **sometimes these clusters should not what they appear**. Let’s write code to display all words of a cluster containing a given word, together with the closest and farthest cluster.

`def get_kmeans_cluster(kmeans, word=None, cluster=None):`

# given a word, find the cluster of that word. (or start with a cluster index.)

# then, get all words of that cluster.

assert (word is None) ^ (cluster is None)

if cluster is None:

cluster = kmeans.predict([embeddings[word]])[0]

words = np.array([w for w in embeddings])

X = np.array([embeddings[w] for w in embeddings])

y = kmeans.predict(X)

cluster_words = words[y == cluster]

return cluster, cluster_wordsdef display_cluster(kmeans, word):

cluster, cluster_words = get_kmeans_cluster(kmeans, word=word)

# print all words within the cluster

print(f"Full KMeans ({word}, cluster {cluster})")

for i, w in enumerate(cluster_words):

print(i+1, w)

# rank all clusters (excluding this one) by Euclidean distance of their centers from this cluster's center

distances = np.concatenate([kmeans.cluster_centers_[:cluster], kmeans.cluster_centers_[cluster+1:]], axis=0)

distances = np.sum(np.square(distances - kmeans.cluster_centers_[cluster]), axis=1)

nearest = np.argmin(distances, axis=0)

_, nearest_words = get_kmeans_cluster(kmeans, cluster=nearest)

print(f"Nearest cluster: {nearest}")

for i, w in enumerate(nearest_words[:5]):

print(i+1, w)

farthest = np.argmax(distances, axis=0)

print(f"Farthest cluster: {farthest}")

_, farthest_words = get_kmeans_cluster(kmeans, cluster=farthest)

for i, w in enumerate(farthest_words[:5]):

print(i+1, w)

Now let’s check out this code.

`display_cluster(kmeans, 'animal')`

# species, fish, wild, dog, bear, males, birds...

display_cluster(kmeans, 'dog')

# same as 'animal'

display_cluster(kmeans, 'birds')

# same again

display_cluster(kmeans, 'bird')

# spread, bird, flu, virus, tested, humans, outbreak, infected, sars....?

You would possibly not get exactly this result each time: the clustering algorithm is non-deterministic. But much of the time, “birds” is related to disease words fairly than animal words. It seems the unique dataset tends to make use of the word “bird” within the context of disease vectors.

There are actually a whole bunch more clusters so that you can explore the contents of. Another clusters I discovered interesting are “Illinois” and “Genghis”.

**Principal Component Evaluation (PCA)** is a tool we are able to use to seek out the directions in vector space related to probably the most variance in our dataset. Let’s try it. Like clustering, `sklearn`

makes this easy.

`from sklearn.decomposition import PCA`def get_pca_vecs(n=10): # get the primary 10 principal components

pca = PCA()

X = np.array([embeddings[w] for w in embeddings])

pca.fit(X)

principal_components = list(pca.components_[:n, :])

return pca, principal_components

pca, pca_vecs = get_pca_vecs()

for i, vec in enumerate(pca_vecs):

# display the words with the very best and lowest values for every principal component

display_sims(to_e=vec, metric=cos_sim, label=f'PCA {i+1}')

display_sims(to_e=vec, metric=cos_sim, label=f'PCA {i+1} negative', reverse=True)

Like our *k*-means experiment, quite a lot of these PCA vectors are really interesting. For instance, let’s take a take a look at principal component 9:

` PCA 9`

1 featuring: 0.38193

2 hindi: 0.37217

3 arabic: 0.36029

4 sung: 0.35130

5 che: 0.34819

6 malaysian: 0.34474

7 ka: 0.33820

8 video: 0.33549

9 bollywood: 0.33347

10 counterpart: 0.33343

PCA 9 negative

1 suffolk: -0.31999

2 cumberland: -0.31697

3 northumberland: -0.31449

4 hampshire: -0.30857

5 missouri: -0.30771

6 calhoun: -0.30749

7 erie: -0.30345

8 massachusetts: -0.30133

9 counties: -0.29710

10 wyoming: -0.29613

It looks like positive values for component 9 are related to Middle Eastern, South Asian and Southeast Asian terms, while negative values are related to North American and British terms.

One other interesting one is component 3. All of the positive terms are decimal numbers, apparently quite a salient feature for this model. Component 8 also shows an analogous pattern.

` PCA 3`

1 1.8: 0.57993

2 1.6: 0.57851

3 1.2: 0.57841

4 1.4: 0.57294

5 2.3: 0.57019

6 2.6: 0.56993

7 2.8: 0.56966

8 3.7: 0.56660

9 1.9: 0.56424

10 2.2: 0.56063

## Dimensionality Reduction

Considered one of the most important advantages of PCA is that it allows us to **take a really high-dimensional dataset** (300-dimensional on this case) and **plot it in only two or three dimensions** by projecting onto the primary components. Let’s try a two-dimensional plot and see if there may be any information we are able to gather from it. We’ll also include color-coding by cluster using *k*-means.

`def plot_pca(pca_vecs, kmeans):`

words = [w for w in embeddings]

x_vec = pca_vecs[0]

y_vec = pca_vecs[1]

X = np.array([np.dot(x_vec, embeddings[w]) for w in words])

Y = np.array([np.dot(y_vec, embeddings[w]) for w in words])

colours = kmeans.predict([embeddings[w] for w in words])

plt.scatter(X, Y, c=colours, cmap='spring') # color by cluster

for i in np.random.alternative(len(words), size=100, replace=False):

# annotate 100 randomly chosen words on the graph

plt.annotate(words[i], (X[i], Y[i]), weight='daring')

plt.show()plot_pca(pca_vecs, kmeans)

Unfortunately, this plot is a complete mess! It’s difficult to learn much from it. It looks like just two dimensions in isolation should not very easy to interpret amongst 300 total dimensions, at the least within the case of this dataset.

There are two exceptions. First, we see that names are inclined to cluster near the highest of this graph. Second, there may be a bit section that stands proud like a sore thumb at the underside left. This area appears to be related to numbers, particularly decimal numbers.

It is usually helpful to get an idea of the **covariance between input features**. On this case, our input features are only abstract vector directions which are difficult to interpret. Still, a covariance matrix can tell us how much of this information is definitely getting used. If we see high covariance, it means some dimensions are strongly correlated, and possibly we could get away with reducing the dimensionality a bit bit.

`def display_covariance():`

X = np.array([embeddings[w] for w in embeddings]).T # rows are variables (components), columns are observations (words)

cov = np.cov(X)

cov_range = np.maximum(np.max(cov), np.abs(np.min(cov))) # be certain the colorbar is balanced, with 0 in the center

plt.imshow(cov, cmap='bwr', interpolation='nearest', vmin=-cov_range, vmax=cov_range)

plt.colorbar()

plt.show()display_covariance()

After all, there’s a giant line down the main diagonal, representing that every component is strongly correlated with itself. Apart from that, this isn’t a really interesting graph. The whole lot looks mostly blank, which is a great sign.

Should you look closely, there’s one exception: components 9 and 276 seem somewhat strongly related (covariance of 0.308).

Let’s investigate this further by printing the vectors which are most related to components 9 and 276. That is reminiscent of cosine similarity to a basis vector of all zeros, aside from a one within the relevant component.

`e9 = np.zeros_like(zero_vec)`

e9[9] = 1.0

e276 = np.zeros_like(zero_vec)

e276[276] = 1.0

display_sims(to_e=e9, metric=cos_sim, label='e9')

# grizzlies, supersonics, notables, posey, bobcats, wannabe, hoosiers...

display_sims(to_e=e276, metric=cos_sim, label='e276')

# pehr, zetsche, steadied, 202-887-8307, bernice, goldie, edelman, kr...

These results are strange, and never very informative.

But wait: we may also have a positive covariance in these components if words with a really *negative* value in a single are inclined to even be very negative in the opposite. Let’s try reversing the direction of similarity.

`display_sims(to_e=e9, metric=cos_sim, label='e9', reverse=True)`

# subsequently, that, it, which, government, because, furthermore, fact, thus, very

display_sims(to_e=e276, metric=cos_sim, label='e276', reverse=True)

# they, as an alternative, those, a whole bunch, addition, dozens, others, dozen, only, outside

It looks like each of those components are related to basic function words and numbers that could be present in many alternative contexts. This helps explain the covariance between them, at the least more so than the positive case did.

In this text we applied quite a lot of **exploratory data evaluation (EDA)** techniques to a 300-dimensional dataset of **GloVe word embeddings**. We used cosine similarity to measure the similarity between the meaning of words, clustering to group words into related groups, and principal component evaluation (PCA) to discover the directions in vector space which are most vital to the embedding model.

We visually observed overall minimal covariance between the input features using principal component evaluation. We tried using PCA to plot all of our 300-dimensional data in only two dimensions, but this was still a bit messy.

We also tested assumptions and biases in our dataset. We identified gender bias in our dataset by comparing the cosine similarity of *nurse* with each of *man* and *woman*. We tried using vector math to represent analogies (like “king” is to “queen” as “man” is to “woman”), with some success. By subtracting various examples of vectors referring to men and women, we were capable of discover a vector direction related to gender, and display the “most masculine” and “most feminine” vectors within the dataset.

There’s loads more EDA you may try on a dataset of word embeddings, but I hope this was a great place to begin to know each some techniques of EDA usually and the structure of word embeddings particularly. If you wish to see the complete code related to this text, plus some additional examples, you possibly can take a look at my GitHub at crackalamoo/glove-embeddings-eda. Thanks for reading!

## References

[1] J. Pennington, R. Socher and C.Manning, GloVe: Global Vectors for Word Representation (2014), Stanford NLP (Public Domain Dataset)

[2] T. Bolukbasi, K. Chang, J. Zou, V. Saligrama and A. Kalai, Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (2016), Microsoft Research Recent England

*All images created by the writer using Matplotlib.*