“It’s cool work,” says Matthew Gudzial, an AI researcher on the University of Alberta, who developed a similar game generator just a few years ago. 

Genie was trained on 30,000 hours of video of a whole bunch of 2D platform games taken from the web. Others have taken that approach before, says Gudzial. His own game generator learned from videos to create abstract platformers. Nivida used video data to coach a model called GameGAN, which could produce clones of games like Pac-Man.

But all of those examples trained the model with input actions, button presses on a games controller, in addition to video footage: a video frame showing Mario jumping was paired with the “jump” motion, and so forth. Tagging video footage with input actions takes a whole lot of work, nonetheless. This has limited the quantity of coaching data available. 

In contrast, Genie was trained on video footage alone. It then learned which of eight possible actions would cause the sport character in a video to alter its position. This turned countless hours of existing online video into potential training data. 

Genie generates each recent frame of the sport on the fly depending on the motion the player takes. Press jump and Genie updates the present image to indicate the sport character jumping; press left and the image changes to indicate the character moved to the left. The sport ticks along motion by motion, each recent frame generated from scratch because the player plays. 

Future versions of Genie could run faster. “There is no such thing as a fundamental limitation that forestalls us from reaching 30 frames per second,” says Tim Rocktäschel, a research scientist at Google DeepMind who leads the team behind the work. “Genie uses lots of the same technologies as contemporary large language models, where there was significant progress in improving inference speed.” 

Genie learned some common visual quirks present in platformers. Many games of this sort use parallax, where the foreground moves sideways faster than the background. Genie often adds this effect to the games it generates.  

While Genie is an in-house research project and won’t be released, Gudzial notes that the Google DeepMind team says it could sooner or later be become a game-making tool—something he’s working on too. “I’m definitely interested to see what they construct,” he says.