News & Press

Today’s tentpole VFX pipelines are labor-intensive, time-consuming and expensive, while AI promises to produce the same but much cheaper and quicker with a smaller team.

Barbara Ford Grant, Media Advisor, Consultant, and pioneering VFX and creative technology executive set out to prove the merits and weakness in this argument.

“Studios working on $100 million or above productions have been in a really sweet spot, but now they’re taxed with having to make something that is a substantially better experience than their competitors because they want to get audience into theatres, but they have to do it a lot cheaper. They must get the cost down and the premium up.”

Ford Grant told attendees at the recent HPA NET Roundtable in March, “Cinematic storytelling is still a human creative process of writing, visualizing, planning, shooting, and executing, and then doing post, but you can weave AI tools throughout that entire process to create huge efficiencies.”

With the 2023 strikes, Ford Grant found herself with extra time.  Taking advantage of that unusual circumstance, she decided to make a short film to explore the possibilities and limitations of AI filmmaking.

“I’d been working on machine learning R&D for about 15 years, but once generative video tools like Midjourney came out, I wanted to play around with them unencumbered by the studio system.”

Under the banner of BFG Productions, she developed, wrote and produced a 22-minute film, Unhoused, on a shoestring $40,000 budget.  The majority of the budget was used to shoot the production traditionally with real actors on location with a union crew.

“Then we used AI tools to see how far above our weight we could punch in post and VFX.”

Turns out, pretty far indeed.

With Daniel Kramer, VFX Supervisor, FX Artist, Head of CG, her partner on the project, they shot for two days, then used Gaussian Splatting and AI to create a third day’s shoot on an LED stage at Lux Machina.

Of the 60 VFX shots, only four were entirely AI-generated. The rest used a real-time pipeline and conventional software like Houdini, Nuke, Maya and Adobe Creative Suite augmented with AI techniques for animation, FX, fluids and so-on from Adobe Firefly, Midjourney and Runway ML among others. Kramer used PostShot and LumaAI to create 3D Gaussian splats from multiple still photos captured with a DSLR and iPhone.  Those splats were loaded into Unreal and displayed in realtime on an LED wall for a virtual production shoot.

Some of these tools saved them time; others did not, due to the lack of maturity and early iteration of the new technology. “We set out to ask a number of key questions,” says Ford Grant. “What level of production value could we achieve? What size team did we need (how low can you go…) and how fast could we go?

Ford Grant and Kramer each have 30 years of production experience but haven’t been at the coalface of creating shots for some time. Aside from some complex composites involving HDR and 3D tracking, they found that they were able to complete 80% of the work on Unhoused themselves.

 

AI Challenges

Overall, it is “mind-blowing what can be done,” she says, “but there are many gaps where human expertise will be required.” A key one is that productions need control over every aspect in isolation.

“Most tools are limited in resolution and quality,” Grant says. “For example, often they aren’t 24fps and only 8-bit color. Many tools rely on prompts which are imprecise for control. Multiple takes with the same prompt produced very different results. Reference images, depth maps, roto mattes and 3D models are better inputs to guide AI more directly.”

Many of these shortcomings are actively being worked on and new models are released daily to give better control and consistency to the outputs.

“In fact, there are a couple shots I’m going to go back and redo because the tools have advanced in this short space of time and now I do have more control over the outcome.”

The power and accessibility of AI tools promises to democratize the entire creative process meaning, that anyone can potentially create content with A-list production values. On the other hand, studios could also use these tools and simply cut the artists out of the loop.

Ford doesn’t think either extreme is true. “It still takes a lot of understanding, taste and expertise by humans to get the best outcomes. Fully AI-generated material may look ‘okay’ superficially, but it wouldn’t make it past a review at a studio. You have to know what ‘good’ looks like. You have to know what you’re looking for and, when you get painted into a corner, how to solve it another way.”

She believes anyone with experience of production, particularly those with an animation or VFX background, should excel with AI tools.

“They’re going to go a long way into production with a lot less people.”

Studios, however, will need to employ more creative talent, not less, if they want to succeed with AI.

“Studios are likely to hire emerging AI studios (such as Secret Level, Asteria and the Russo Brothers’ AGBO) which are embracing these tools in order to negotiate productions at a lower cost.

“AI should be a golden time for authoring, directing and imagining content. That said, the further away you are from being the person who decides if a shot or a piece of content is good enough the more likely it is that your job is at risk.”

“I encourage people to learn these tools as quickly as possible. AI can be a really great collaboration tool. Use them to create novel and inventive things.”

Rights management will become even more integral to the creative and business process. Ford advises artists to digitally fingerprint their work to control access and verify a breach of their IP.

“Artists need control over their data,” she says. “Depending on how extensive your library is you may not need third party LLMs. If you’re a creator with 40 years of your own beautiful material to train on, you can arrange your own rules and regulations [enshrined on the blockchain] for allowing others to collaborate and build on those characters and models. If you’re a studio with a rich IP library, you might choose to do that as well. Whichever the case, people need to get control of their assets quickly.”

Looking ahead, Ford Grant predicts that AI democratization will see small production teams become viable again. “Artist-driven boutiques will return as quality becomes achievable at lower costs.”

Ford Grant emphasizes the positives in the industry adoption of AI and believes the future will be creator-led. “The further you industrialise your media business away from the creative process – away from the artistry of it all – the further you are from what’s happening now and the more you will miss out.”

For anyone who thinks an AI is going to generate a masterpiece on its own, Ford Grant offers an analogy with the creation of movie classics of the past. Wim Wenders’ Paris, Texas (1984) left an impression on audiences, Ford Grant included, because it continually upended our expectations of what narrative should look like.

“Paris, Texas is a prime example of how a group of people came together and made art that moved people because of the combined creative alchemy. That’s something that you never want to go away. Just as you could never make that film from a single point of view – it required Sam Shepherd, Robby Müller, Harry Dean Stanton, Ry Cooder and more – you could never rely on one AI model or AI alone to make something as fresh.”