Natural language conveys ideas, actions, information, and intent through context and syntax; further, there are volumes of it contained in databases. This makes it a superb source of information to coach machine-learning systems on. Two master’s of engineering students within the 6A MEng Thesis Program at MIT, Irene Terpstra ’23 and Rujul Gandhi ’22, are working with mentors within the MIT-IBM Watson AI Lab to make use of this power of natural language to construct AI systems.
As computing is becoming more advanced, researchers want to improve the hardware that they run on; this implies innovating to create recent computer chips. And, since there may be literature already available on modifications that may be made to attain certain parameters and performance, Terpstra and her mentors and advisors Anantha Chandrakasan, MIT School of Engineering dean and the Vannevar Bush Professor of Electrical Engineering and Computer Science, and IBM’s researcher Xin Zhang, are developing an AI algorithm that assists in chip design.
“I’m making a workflow to systematically analyze how these language models can assist the circuit design process. What reasoning powers have they got, and the way can it’s integrated into the chip design process?” says Terpstra. “After which on the opposite side, if that proves to be useful enough, [we’ll] see in the event that they can robotically design the chips themselves, attaching it to a reinforcement learning algorithm.”
To do that, Terpstra’s team is creating an AI system that may iterate on different designs. It means experimenting with various pre-trained large language models (like ChatGPT, Llama 2, and Bard), using an open-source circuit simulator language called NGspice, which has the parameters of the chip in code form, and a reinforcement learning algorithm. With text prompts, researchers will have the option to question how the physical chip ought to be modified to attain a certain goal within the language model and produced guidance for adjustments. That is then transferred right into a reinforcement learning algorithm that updates the circuit design and outputs recent physical parameters of the chip.
“The ultimate goal could be to mix the reasoning powers and the knowledge base that’s baked into these large language models and mix that with the optimization power of the reinforcement learning algorithms and have that design the chip itself,” says Terpstra.
Rujul Gandhi works with the raw language itself. As an undergraduate at MIT, Gandhi explored linguistics and computer sciences, putting them together in her MEng work. “I’ve been eager about communication, each between just humans and between humans and computers,” Gandhi says.
Robots or other interactive AI systems are one area where communication must be understood by each humans and machines. Researchers often write instructions for robots using formal logic. This helps be certain that commands are being followed safely and as intended, but formal logic may be difficult for users to grasp, while natural language comes easily. To make sure this smooth communication, Gandhi and her advisors Yang Zhang of IBM and MIT assistant professor Chuchu Fan are constructing a parser that converts natural language instructions right into a machine-friendly form. Leveraging the linguistic structure encoded by the pre-trained encoder-decoder model T5, and a dataset of annotated, basic English commands for performing certain tasks, Gandhi’s system identifies the smallest logical units, or atomic propositions, that are present in a given instruction.
“When you’ve given your instruction, the model identifies all of the smaller sub-tasks you wish it to perform,” Gandhi says. “Then, using a big language model, each sub-task may be compared against the available actions and objects within the robot’s world, and if any sub-task can’t be carried out because a certain object shouldn’t be recognized, or an motion shouldn’t be possible, the system can stop right there to ask the user for help.”
This approach of breaking instructions into sub-tasks also allows her system to grasp logical dependencies expressed in English, like, “do task X until event Y happens.” Gandhi uses a dataset of step-by-step instructions across robot task domains like navigation and manipulation, with a deal with household tasks. Using data which are written just the way in which humans would discuss with one another has many benefits, she says, since it means a user may be more flexible about how they phrase their instructions.
One other of Gandhi’s projects involves developing speech models. Within the context of speech recognition, some languages are considered “low resource” since they may not have numerous transcribed speech available, or won’t have a written form in any respect. “One in every of the explanations I applied to this internship on the MIT-IBM Watson AI Lab was an interest in language processing for low-resource languages,” she says. “Numerous language models today are very data-driven, and when it’s not that easy to amass all of that data, that’s when you have to use the limited data efficiently.”
Speech is only a stream of sound waves, but humans having a conversation can easily work out where words and thoughts start and end. In speech processing, each humans and language models use their existing vocabulary to acknowledge word boundaries and understand the meaning. In low- or no-resource languages, a written vocabulary won’t exist in any respect, so researchers can’t provide one to the model. As an alternative, the model could make note of what sound sequences occur together more ceaselessly than others, and infer that those is perhaps individual words or concepts. In Gandhi’s research group, these inferred words are then collected right into a pseudo-vocabulary that serves as a labeling method for the low-resource language, creating labeled data for further applications.
The applications for language technology are “just about in all places,” Gandhi says. “You could possibly imagine people with the ability to interact with software and devices of their native language, their native dialect. You could possibly imagine improving all of the voice assistants that we use. You could possibly imagine it getting used for translation or interpretation.”