AI is Changing Computer Graphics
In the past few years, advancements in AI and machine learning research have facilitated the emergence of astonishing and invaluable technologies. AI research is especially thriving in the world of computer graphics: new techniques have streamlined the production of computer-generated imagery and have enhanced both the development process and the enjoyability of video games. I'd like to highlight some cutting-edge computer graphics research by discussing the data science concepts utilized to accomplish feats that nearly seem impossible.
Game engines and 3D computer graphics software utilize many mathematical functions and algorithms to simulate how a virtual scene should look. When trying to accurately represent realistic lighting conditions on complex surfaces and geometries, these functions and algorithms can easily become computationally expensive, taking a long time to render at high resolutions. Historically, this has led to a tradeoff between visual quality and computation speeds. However, recent research has focused on using deep learning and AI techniques to lighten computational loads while minimizing the compromise of visual acuity. One of the predominant approaches to enhancing computational efficiency involves rendering scenes at a reduced resolution or with less precise simulations and then feeding them into specialized deep-learning algorithms designed to infer and restore any lost details.
If these concepts still seem vague, check out this Jupyter Notebook, where I thoroughly explain a technique called render scaling to explain the intuition of this process. The notebook also provides additional commentary on basic computer graphics concepts and an analogy of the problem above via a simple neural network learning a non-linear function.
Approaching this Problem With Machine Learning
A fundamental premise of machine learning, and specifically deep learning, is that any unknown function can be learned and approximated with sufficient data. If an unknown function is deterministic or has a relatively low variance, then a fully trained neural network should be able to make reasonable predictions. Therefore, if pixels are a function of nearby pixels or low-resolution artifacts have a consistent pattern, it is possible to train a neural network to extrapolate missing detail from a low-resolution frame and then approximate what the frame should look like if it were rendered in full resolution.
Despite being conceptually feasible, implementing a valid model to solve this problem can still be quite challenging. First, you need to devise a robust pipeline to extract a lot of high-quality data to train the model. And perhaps more importantly in this context, you need to find an appropriate model architecture that can support the overarching goal: accurately extrapolating missing detail faster than rendering the scene in full resolution. With these concepts in mind, let’s examine Nvidia’s DLSS, a family of AI technologies showcasing some impressive innovations in computer graphics research.
Nvidia DLSS
Deep learning super sampling (DLSS) uses a multifunctional deep learning algorithm that boosts video game performance by improving frame rates while delivering image quality that matches or surpasses native high-resolution rendering. To do so, DLSS renders the game on a reduced resolution to conserve computational resources, tracks game engine data such as motion vectors and depth, and calculates an optical flow field. It then takes all of this data for the current and prior game frames and feeds it into the deep learning network to predict how the next frames should look in a higher resolution. The latest version, DLSS 3.5, incorporates additional data from the game engine to enable ray reconstruction. This revolutionary technique circumvents the computationally expensive task of ray tracing, while still achieving stunning and realistic lighting effects.
Diagram of DLSS 3.5 Pipeline from Nvidia's Blog Post
Benefitting From These Technologies and Future Research
As impressive as DLSS seems, it’s still a nascent and imperfect technology with plenty of room for improvement. As of December 2023, DLSS 3.5 exists as an experimental, albeit mind-blowing, proof of concept, as it’s only available in two games (Cyberpunk 2077 and Alan Wake 2). However, its sheer existence is a testament to the trend of powerful AI technologies becoming increasingly accessible. Unreal Engine 5, an industry-leading game engine, has been continuously implementing AI-assisted tools throughout 2023, streamlining the development of both video games and virtual productions. These technologies are helping creatives craft captivating stories while simultaneously enhancing the experience of consumers - encouraging further research and development.
With the current momentum and trajectory of AI research, many exciting and experimental techniques that presently only exist in research papers will soon work their way into the lives of many. This wave of enthusiasm and excitement in AI research is ushering in a wave of discoveries, and there is a significant chance that these new ideas will revolutionize how we interact with and benefit from technology.
Grad School Update and Additional Resources:
The Fall 2023 semester has concluded, and I am officially done with 80% of my Master’s in Data Science program. I took two courses this semester: ‘Natural Language Processing (NLP)’ and ‘Design Principles & Casual Inference.’ The main theme of this semester was undoubtedly literature review; in fact, the final project for my NLP class was a mock NeurIPS research paper discussing methods to improve the performance of textual entailment models.
The constant exposure to many research papers was a huge inspiration for me to write this post and helped me hone my ability to make sense of unfamiliar and complex concepts. Specifically, my NLP course made me incredibly enthusiastic about the direction of AI research, and as a result, I have tried my best to keep up with the most fascinating updates in the field. Research papers can often be inaccessible from both a readability and availability standpoint, but I’ve found some excellent resources that have been assets in my learning journey. If you found this post interesting and would like to learn more about AI, computer graphics, and the technologies that are changing the world, I highly recommend the following:
Watching Neural Networks Learn by Emergent Garden
Additional Information about DLSS:
