Model Cards

Model cards are a vital documentation framework for understanding, sharing, and improving machine learning models. When done well, a model card can function a boundary object, a single artefact that’s accessible to individuals with different backgrounds and goals in understanding models – including developers, students, policymakers, ethicists, and people impacted by machine learning models.

Today, we launch a model card creation tool and a model card Guide Book, which details how one can fill out model cards, user studies, and cutting-edge in ML documentation. This work, constructing from many other people and organizations, focuses on the inclusion of individuals with different backgrounds and roles. We hope it serves as a stepping stone in the trail toward improved ML documentation.

In sum, today we announce the discharge of:

1. A Model Card Creator Tool, to ease card creation while not having to program, and to assist teams share the work of various sections.

2. An updated model card template, released in the huggingface_hub library, drawing together model card work in academia and throughout the industry.

3. An Annotated Model Card Template, which details how one can fill the cardboard out.

4. A User Study on model card usage at Hugging Face.

5. A Landscape Evaluation and Literature Review of the cutting-edge in model documentation.

Model Cards To-Date

Since Model Cards were proposed by Mitchell et al. (2018), inspired by the main documentation framework efforts of Data Statements for Natural Language Processing (Bender & Friedman, 2018) and Datasheets for Datasets (Gebru et al., 2018), the landscape of machine learning documentation has expanded and evolved. A plethora of documentation tools and templates for data, models, and ML systems have been proposed and developed – reflecting the incredible work of a whole lot of researchers, impacted community members, advocates, and other stakeholders. Essential discussions in regards to the relationship between ML documentation and theories of change in responsible AI have also shaped these developments within the ML documentation ecosystem.

Work to-date on documentation inside ML has provided for various audiences. We bring a lot of these ideas together within the work we share today.

Our Work

Our work presents a view of where model cards stand straight away and where they may go in the long run. We conducted a broad evaluation of the growing landscape of ML documentation tools and conducted user interviews inside Hugging Face to complement our understanding of the varied opinions about model cards. We also created or updated dozens of model cards for ML models on the Hugging Face Hub, and informed by all of those experiences, we propose a brand new template for model cards.

Standardising Model Card Structure

Through our background research and user studies, that are discussed further within the Guide Book, we aimed to determine a brand new standard of “model cards” as understood by most of the people.

Informed by these findings, we created a brand new model card template that not only standardized the structure and content of HF model cards but in addition provided default prompt text. This text aimed to aide with writing model card sections, with a selected give attention to the Bias, Risks and Limitations section.

Accessibility and Inclusion

With the intention to lower barriers to entry for creating model cards, we designed the model card writing tool, a tool with a graphical user interface (GUI) to enable people and teams with different skill sets and roles to simply collaborate and create model cards, while not having to code or use markdown.

The writing tool encourages those that have yet to write down model cards to create them more easily. For individuals who have previously written model cards, this approach invites them so as to add to the prompted information — while centering the moral components of model documentation.

As ML continues to be more intertwined with different domains, collaborative and open-source ML processes that center accessibility, ethics and inclusion are a critical a part of the machine learning lifecycle and a stepping stone in ML documentation.

Today’s release sits inside a bigger ecosystem of ML documentation work: Data and model documentation have been taken up by many tech corporations, including Hugging Face 🤗. We have prioritized “Repository Cards” for each dataset cards and model cards, specializing in multidisciplinarity. Continuing on this line of labor, the model card creation UI tool
focuses on inclusivity, providing guidance on formatting and prompting to help card creation for individuals with different backgrounds.

Call to motion

Let’s look ahead

This work is a “snapshot” of the present state of model cards, informed by a landscape evaluation of the various ways ML documentation artefacts have been instantiated. The model book and these findings represent one perspective amongst multiple about each the present state and more aspirational visions of model cards.

• The Hugging Face ecosystem will proceed to advance methods that streamline Model Card creation through code and user interfaces, including constructing more features directly into the repos and product.
• As we further develop model tools akin to Evaluate on the Hub, we are going to integrate their usage inside the model card development workflow. For instance, as mechanically evaluating model performance across disaggregated aspects becomes easier, these results might be possible to import into the model card.
• There may be further study to be done to advance the pairing of research models and model cards, akin to constructing out a research paper → to model documentation pipeline, making it make it trivial to go from paper to model card creation. This is able to allow for greater cross-domain reach and further standardisation of model documentation.

We proceed to learn more about how model cards are created and used, and the effect of cards on model usage. Based on these learnings, we are going to further update the model card template, instructions, and Hub integrations.

As we try to include more voices and stakeholders’ use cases for model cards, bookmark our model cards writing tool and provides it a try!

We’re excited to know your thoughts on model cards, our model card writing GUI, and the way AI documentation can empower your domain.🤗

Acknowledgements

This release wouldn’t have been possible without the extensive contributions of Omar Sanseviero, Lucain Pouget, Julien Chaumond, Nazneen Rajani, and Nate Raw.