News & Press

On June 13, a panel at the HPA NET (Networking Entertainment Technology) lunch took a deep dive into the best way to maintain the artistic intent of material acquired in HD/2K and Standard Dynamic Range (SDR) and face the new distribution requirements of 4K and High Dynamic Range (HDR).

DigitalFilm Tree senior colorist Dan Judy, VideoGorillas chief executive Jason Brahms, and Miaoqi Zhu, Ph.D., research and development software engineer at Sony Pictures Entertainment revealed the different ways that their companies are approaching this complicated topic.

The need to migrate content from one format to another is intrinsic to the history of the entertainment industry, most recently experienced with the tumultuous transition from NTSC’s standard definition to High Definition. Then, the conversation centered on not just maintaining the artistic integrity of the original content – but on who was going to pay for the crucial up-rezzing of material.

The conversation over upgrading material from HD/2K to 4K and SDR to HDR is similar. Except now, new technologies, including machine learning/artificial intelligence, offer the possibility to make the process faster, better and more economical.

At VideoGorillas, Brahms says his company is doing just that. Its Bigfoot Super Resolution (R&D) relies on deep convolutional Generative Adversarial Networks (GANs) to upscale content. “GANs are a class of artificial intelligence algorithms used in unsupervised machine learning, implemented by a system of two neural networks contesting with each other in a zero-sum game framework,” explains Brahms. “One network generates candidates (generative) and the other evaluates them (discriminative). The generative network’s training objective is to increase the error rate of the discriminative network, that is “fool” the discriminator network by producing novel synthesized instances that appear to have come from the true data distribution.”

He adds that the “pillars of Bigfoot super resolution” are to “produce not only accurate and perceptual high-quality images but also a temporally consistent sequence of these images.” “It’s no easy task,” he says. He also stressed that pixel interpolation is not used in Bigfoot Super Resolution. “Bigfoot is a new category of upscaling,” he explains. “The process results in the synthesis of an entirely new image of higher resolution, as opposed to the interpolation of unknown pixels. Our approach is data driven and the result is a perceptual image quality that is significantly higher than the results from traditional algorithmic solutions.” He adds that, “Bigfoot Super Resolution networks understand what they’re superresing.  “For example, tree leaves are superresed in a different way than grass, despite both having similar looking pixels.” Brahms reports that VideoGorillas has constructed a pipeline for unsupervised image Super-Resolution with “no ground truth hi-res images and unknown low-res artifacts.” “It is still in the research stage but works better than other unsupervised approaches,” he says. “We have very promising results attacking all kinds of analog to digital conversion noise. Our current solution works with color in 32-bit float values. Provided we have enough HDR training material, the network will be able to output Super Resolution/HDR frames. We think these two tasks can easily be combined into one model.”

DigitalFilm Tree has developed two services to upgrade SDR material to HDR: HDR Express and HDR Mastering. The idea first took root when chief executive/founder Ramy Katrib realized that, since the debut of the ARRI Alexa camera, all his Hollywood clients, including ABC, CBS, Fox, Turner, HBO, and Lifetime, had acquired content in ProRes 1080p. Meanwhile, some companies, notably Netflix, began to capture content in 4K HDR. Katrib says he thought, what about the massive library across the industry in SDR?” He began to focus on delivering an SDR-to-HDR solution that would maintain the artistic integrity of the original master and be economical and quick enough for the owners of large content libraries to use.

About 18 months later, the company came out with a proprietary automated post pipeline that leverages industry tools such as Blackmagic Design Resolve. “Using the client’s original Color Timed Material (CTM) we take that master material and put it through our process. HDR Express,” says colorist Judy. “We then give them one IMF File that’s Dolby Preferred and streaming service friendly, with Dolby’s L1/L2 process.” At the NET lunch, July demonstrated HDR Express with TBS’ TV show Wrecked, as driven by the colorist.

DigitalFilm Tree also offers HDR Mastering, which does use the CTM (where color decisions are baked in), but rather the Video Assembled Master (VAM). “The goal of this method is to take full advantage of HDR from the RAW SDR materials and deliver an Optimized image seeking to see detail in the extended Dynamic Range available to us,” says July. “This is a complete recolor and by default a more involved process.

Finally, Sony Pictures Entertainment’s Zhu presented some of the R&D work he has been doing focused on colorization. He has provided a portion of his presentation here.

The conclusion? As the need to upgrade library content to 4K HDR becomes more pressing, some new tech-enabled solutions are on the cusp of making that process a possibility at scale.