Welcome to AI for Game Development! On this series, we’ll be using AI tools to create a completely functional farming game in only 5 days. By the tip of this series, you’ll have learned how you may incorporate quite a lot of AI tools into your game development workflow. I’ll show you the way you should utilize AI tools for:
- Art Style
- Game Design
- 3D Assets
- 2D Assets
- Story
Want the fast video version? You possibly can watch it here. Otherwise, when you want the technical details, keep reading!
Note: This tutorial is meant for readers who’re aware of Unity development and C#. If you happen to’re recent to those technologies, take a look at the Unity for Beginners series before continuing.
Day 3: 3D Assets
In Part 2 of this tutorial series, we used AI for Game Design. More specifically, we used ChatGPT to brainstorm the design for our game.
On this part, we’ll discuss how you should utilize AI to generate 3D Assets. The short answer is: you may’t. That is because text-to-3D is not at the purpose it could possibly be practically applied to game development, yet. Nevertheless, that is changing in a short time. Keep reading to find out about The Current State of Text-to-3D, Why It Is not Useful (yet), and The Way forward for Text-to-3D.
The Current State of Text-to-3D
As discussed in Part 1, text-to-image tools corresponding to Stable Diffusion are incredibly useful in the sport development workflow. Nevertheless, what about text-to-3D, or generating 3D models from text descriptions? There have been many very recent developments on this area:
Lots of these approaches, excluding CLIPMatrix and CLIP-Mesh-SMPLX, are based on view synthesis, or generating novel views of a subject, as opposed to standard 3D rendering. That is the thought behind NeRFs or Neural Radiance Fields, which use neural networks for view synthesis.
What does all of this mean when you’re a game developer? Currently, nothing. This technology hasn’t reached the purpose that it’s useful in game development yet. Let’s discuss why.
Why It Is not Useful (yet)
Note: This section is meant for readers who’re aware of conventional 3D rendering techniques, corresponding to meshes, UV mapping and photogrammetry.
While view synthesis is impressive, the world of 3D runs on meshes, which will not be the identical as NeRFs. There may be, nonetheless, ongoing work on converting NeRFs to meshes. In practice, that is reminiscient of photogrammetry, where multiple photos of real-world objects are combined to creator 3D assets.
The sensible use of assets generated using the text-to-NeRF-to-mesh pipeline is proscribed in an analogous solution to assets produced using photogrammetry. That’s, the resulting mesh isn’t immediately game-ready, and requires significant work and expertise to change into a game-ready asset. On this sense, NeRF-to-mesh could also be a great tool as-is, but doesn’t yet reach the transformative potential of text-to-3D.
Since NeRF-to-mesh, like photogrammetry, is currently most suited to creating ultra-high-fidelity assets with significant manual post-processing, it doesn’t really make sense for making a farming game in 5 days. Wherein case, I made a decision to simply use cubes of various colours to represent the crops in the sport.
Things are changing rapidly on this area, though, and there could also be a viable solution within the near future. Next, I’ll discuss a number of the directions text-to-3D could also be going.
The Way forward for Text-to-3D
While text-to-3D has come a good distance recently, there remains to be a major gap between where we at the moment are and what could have an effect along the lines of text-to-image. I can only speculate on how this gap shall be closed. There are two possible directions which can be most apparent:
- Improvements in NeRF-to-mesh and mesh generation. As we have seen, current generation models are just like photogrammetry in that they require quite a lot of work to supply game-ready assets. While this is beneficial in some scenarios, like creating realistic high-fidelity assets, it’s still more time-consuming than making low-poly assets from scratch, especially when you’re like me and use an ultra-low-poly art style.
- Latest rendering techniques that allow NeRFs to be rendered directly in-engine. While there have been no official announcements, one could speculate that NVIDIA and Google, amongst others, could also be working on this.
In fact, only time will tell. If you should sustain with advancements as they arrive, be at liberty to follow me on Twitter. If there are recent developments I’ve missed, be at liberty to achieve out!
Click here to read Part 4, where we use AI for 2D Assets.
Attribution
Because of Poli @multimodalart for providing info on the most recent open source text-to-3D.
