tl;dr: We made a tool you should use online to construct, measure, and compare datasets.
Click to access the 🤗 Data Measurements Tool here.
As developers of a fast-growing unified repository for Machine Learning datasets (Lhoest et al. 2021), the 🤗 Hugging Face team has been working on supporting good practices for dataset documentation (McMillan-Major et al., 2021). While static (if evolving) documentation represents a obligatory first step on this direction, getting a very good sense of what is definitely in a dataset requires well-motivated measurements and the flexibility to interact with it, dynamically visualizing different features of interest.
To this end, we introduce an open-source Python library and no-code interface called the 🤗 Data Measurements Tool, using our Dataset and Spaces Hubs paired with the good Streamlit tool. This could be used to assist understand, construct, curate, and compare datasets.
What’s the 🤗 Data Measurements Tool?
The Data Measurements Tool (DMT) is an interactive interface and open-source library that lets dataset creators and users mechanically calculate metrics which can be meaningful and useful for responsible data development.
Why have we created this tool?
Thoughtful curation and evaluation of Machine Learning datasets is commonly ignored in AI development. Current norms for “big data” in AI (Luccioni et al., 2021, Dodge et al., 2021) include using data scraped from various web sites, with little or no attention paid to concrete measurements of what the various data sources represent, nor the nitty-gritty details of how they could influence what a model learns. Although dataset annotation approaches may also help to curate datasets which can be more in keeping with a developer’s goals, the methods for “measuring” different features of those datasets are fairly limited (Sambasivan et al., 2021).
A brand new wave of research in AI has called for a fundamental paradigm shift in how the sphere approaches ML datasets (Paullada et al., 2020, Denton et al., 2021). This includes defining fine-grained requirements for dataset creation from the beginning (Hutchinson et al., 2021), curating datasets in light of problematic content and bias concerns (Yang et al., 2020, Prabhu and Birhane, 2020), and making explicit the values inherent in dataset construction and maintenance (Scheuerman et al., 2021, Birhane et al., 2021). Although there may be general agreement that dataset development is a task that individuals from many various disciplines should have the ability to tell, in practice there is commonly a bottleneck in interfacing with the raw data itself, which tends to require complex coding skills with the intention to analyze and query the dataset.
Despite this, there are few tools openly available to the general public to enable people from different disciplines to measure, interrogate, and compare datasets. We aim to assist fill this gap. We learn and construct from recent tools reminiscent of Know Your Data and Data Quality for AI, in addition to research proposals for dataset documentation reminiscent of Vision and Language Datasets (Ferraro et al., 2015), Datasheets for Datasets (Gebru et al, 2018), and Data Statements (Bender & Friedman 2019). The result’s an open-source library for dataset measurements, and an accompanying no-code interface for detailed dataset evaluation.
When can I exploit the 🤗 Data Measurements Tool?
The 🤗 Data Measurements Tool could be used iteratively for exploring a number of existing NLP datasets, and can soon support iterative development of datasets from scratch. It provides actionable insights informed by research on datasets and responsible dataset development, allowing users to hone in on each high-level information and specific items.
What can I learn using the 🤗 Data Measurements Tool?
Dataset Basics
For a high-level overview of the dataset
This begins to reply questions like “What is that this dataset? Does it have missing items?”. You should utilize this as “sanity checks” that the dataset you’re working with is as you expect it to be.
Descriptive Statistics
To take a look at the surface characteristics of the dataset
This begins to reply questions like “What form of language is on this dataset? How diverse is it?”
-
The dataset vocabulary size and word distribution, for each open- and closed-class words.
-
The dataset label distribution and knowledge about class (im)balance.
-
The mean, median, range, and distribution of instance lengths.
-
The variety of duplicates within the dataset and the way over and over they’re repeated.
You should utilize these widgets to envision whether what’s most and least represented within the dataset make sense for the goals of the dataset. These measurements are intended to tell whether the dataset could be useful in capturing quite a lot of contexts or if what it captures is more limited, and to measure how ”balanced” the labels and instance lengths are. You can too use these widgets to discover outliers and duplicates you might wish to remove.
Distributional Statistics
To measure the language patterns within the dataset
This begins to reply questions like “How does the language behave on this dataset?”
- Adherence to Zipf’s law, which provides measurements of how closely the distribution over words within the dataset matches to the expected distribution of words in natural language.
You should utilize this to work out whether your dataset represents language because it tends to behave within the natural world or if there are things which can be more unnatural about it. For those who’re someone who enjoys optimization, then you definately can view the alpha value this widget calculates as a worth to get as close as possible to 1 during dataset development. Further details on alpha values following Zipf’s law in several languages is on the market here.
Typically, an alpha greater than 2 or a minimum rank greater than 10 (take with a grain of salt) signifies that your distribution is comparatively unnatural for natural language. This is usually a sign of mixed artefacts within the dataset, reminiscent of HTML markup. You should utilize this information to scrub up your dataset or to guide you in determining how further language you add to the dataset needs to be distributed.
Comparison statistics
This begins to reply questions like “What sorts of topics, biases, and associations are on this dataset?”
-
Embedding clusters to pinpoint any clusters of comparable language within the dataset.
Taking in the range of text represented in a dataset could be difficult when it’s made up of a whole bunch to a whole bunch of 1000’s of sentences. Grouping these text items based on a measure of similarity may also help users gain some insights into their distribution. We show a hierarchical clustering of the text fields within the dataset based on a Sentence-Transformer model and a maximum dot product single-linkage criterion. To explore the clusters, you possibly can:- hover over a node to see the 5 most representative examples (deduplicated)
- enter an example within the text box to see which leaf clusters it’s most just like
- select a cluster by ID to indicate all of its examples
-
The normalized pointwise mutual information (nPMI) between word pairs within the dataset, which could also be used to discover problematic stereotypes.
You should utilize this as a tool in coping with dataset “bias”, where here the term “bias” refers to stereotypes and prejudices for identity groups along the axes of gender and sexual orientation. We are going to add further terms within the near future.
What’s the status of 🤗 Data Measurements Tool development?
We currently present the alpha version (v0) of the tool, demonstrating its usefulness on a handful of popular English-language datasets (e.g. SQuAD, imdb, C4, …) available on the Dataset Hub, with the functionalities described above. The words that we chosen for nPMI visualization are a subset of identity terms that got here up continuously within the datasets that we were working with.
In coming weeks and months, we can be extending the tool to:
- Cover more languages and datasets present within the 🤗 Datasets library.
- Provide support for user-provided datasets and iterative dataset constructing.
- Add more features and functionalities to the tool itself. For instance, we’ll make it possible so as to add your personal terms for the nPMI visualization so you possibly can pick the words that matter most to you.
Acknowledgements
Thanks to Thomas Wolf for initiating this work, in addition to other members of the 🤗 team (Quentin, Lewis, Sylvain, Nate, Julien C., Julien S., Clément, Omar, and plenty of others!) for his or her help and support.
