Deep learning-based super resolution technology has been adopted in NVidia's latest GPU under the name DLSS (deep learning super sampling) and has become a technology that is actually serviced to consumers. It serves to change to 4K. Of course, there are points on whether the 4K made in this way can be called real 4K, but it is definitely better than 2K, so it seems that it can be said to move to some point between 2K and 4K based on subjective quality standards. (Not a physical resolution, but a quality-based resolution perspective)
Pixar used this technique to keep costs down during the animation rendering pipeline. According to the link, it takes 50 CPU hours to render 1 frame in 2K. Assuming 24 frames per second for a 90-minute movie, it is about 6.5 million CPU hours, but in the case of 4K rendering, it is said that more than twice as much is needed here. Therefore, the cost of the rendering farm was lowered by increasing the resolution after rendering in 2K or 1K. (Specifically, in the case of 1K -> 2K application, it is said that the cost of 50-75% is reduced)
Among the super resolution technologies, Pixar's technology is a joint research between ETH Zurich and Disney called “A Fully Progressive Approach to Single-Image Super-Resolution,” which seems to have used a GAN-applied version of the results of this study. Below is a link to the paper and github of this study.
However, in applying the initial research results in practice, solutions to various problems have been added, such as obtaining production-quality, considering HDR, processing alpha channels, and adapting to various lighting. Of course, from a data point of view, compared to the initial research papers using Div2K (800 pages), Pixar used the actual rendering results of Coco, Incredibles 2, and Toy Story 4 as training data to develop practical application techniques. (I'm envious of this part ^^) The result of the study was not applied directly to the movie, but it was applied to the auxiliary video recorded on the UHD disk of Onword currently being screened, and it is expected to be expanded in the future.
Personally, I am very interested in cases where deep learning-based technologies have proven their value by coming to the real world, not just in the lab. Compared to the numerous research achievements, application cases are still insufficient, but we look forward to innovation in this direction in the future.
