Home Artificial Intelligence A research agenda for assessing the economic impacts of code generation models

A research agenda for assessing the economic impacts of code generation models

2
A research agenda for assessing the economic impacts of code generation models

OpenAI is developing a research program to evaluate the economic impacts of code generation models and is inviting collaboration with external researchers. Rapid advances within the capabilities of enormous language models (LLMs) trained on code have made it increasingly necessary to review their economic impacts on individuals, firms, and society. Codex – an LLM developed by OpenAI by fine-tuning GPT-3 on billions of lines of publicly available code from GitHub – has been shown to generate functionally correct code 28.8% of the time on a sample of evaluation problems (Chen et al. 2021). This may occasionally have necessary implications for the longer term of coding and the economics of the industries that rely on it. On this document, we lay out a research agenda to evaluate the results of Codex on economic aspects of interest to policymakers, firms, and the general public. We make a case for this research agenda by highlighting the doubtless broad applicability of code generation models to software development, the potential for other LLMs to create significant social and economic impact as model capabilities advance, and the worth of using Codex to generate evidence and establish methodologies which may be applicable to research on the economic impacts of future models. We propose that academic and policy research give attention to studying code generation models and other LLMs in order that evidence on their economic impacts might be used to tell decision-making in three key areas: Deployment policy, AI system design, and public policy. To assist guide this research, we outline six priority final result areas throughout the realm of economic impacts that we intend to make use of Codex to review: Productivity, Employment, Skill Development, Inter-firm Competition, Consumer Prices, and Economic Inequality. For every area, we briefly discuss previous literature on the impacts of artificial intelligence on each of those outcomes, describe questions that we imagine to be key inputs to the three decision-making areas mentioned above, and supply examples of research that might be conducted with Codex. To catalyze work that builds off of this initial research agenda, we’re announcing a Call for Expressions of Interest from external researchers to collaborate with OpenAI researchers and customers to higher measure the economic impacts of code generation models and other LLMs.

2 COMMENTS

LEAVE A REPLY

Please enter your comment!
Please enter your name here