By Kimberley Clementi
While the glitz and glam of the Academy Awards attract an international audience each year, off the red carpet and a few left-brain turns from the famed Hollywood stars an unlikely consortium is gaining recognition within the industry.
Computer scientists, including those from UC San Diego’s Computer Science and Engineering Department, are combining their brainpower with artists, creatives and storytellers to tackle some of filmmaking’s most complex, technological challenges.
Using techniques such as stochastic progressive photon mapping and advances in appearance modeling, computer scientists are working behind the scenes to make memorable animated characters and scenes more life-like on the big screen. Their work illuminates the juncture between mathematical theory and the serious science of film magic.
UC San Diego’s CSE fits right into this picture. Both faculty and alumni have contributed to blockbuster hits such as The Lord of the Rings trilogy, the Harry Potter series, Finding Nemo, Coco, Toy Story 4, Luca and, most recently, Turning Red.
Notably, most of these films have been nominated for an Academy Award. They have netted a CSE professor an Oscar, another professor a nomination and given an alumnus a crucial role in animating characters larger-than-life.
Turning Red: A Frontrunner for 2023 Oscars
Most recently, Pixar’s Turning Red was a contender for the Best Animated Feature Oscar for 2023. The lovable star in this coming-of-age film is 13-year-old Mei Lee, a typical, overachieving teen one minute and a giant panda the next. Each time Mei wrestles with supersized emotional triggers – feelings of anxiety, anger or excitement – she transforms into a supersized red panda.
This switch from teen to panda and back to teen presented a bevy of technical challenges. Animating a bright-eyed girl with colorful clothing, flowing hair and life-like movement created one layer of difficulty. Animating a red panda with fluffy fur added another. Animating the transition back and forth between the two required next-level skill and cutting-edge technology.
Lyon Liew (B.S. ‘03) was the master-mind responsible for this captivating cinematic magic. Liew, now in his fifth year at Pixar Animation Studios, was the film’s lead technical director for simulation. With film credits like Coco, Luca and Incredibles 2 already under his belt, Liew faced his greatest simulation challenge to date in Turning Red.
Ironically, this complex and supersized transition was affectionately dubbed with a light-hearted and diminutive moniker: the “poof.” And the “poof” was the biggest animation hurdle Liew and his team needed to address.
“We had a giant, furry red panda that poofs back and forth from a teenage girl and a group of friends with diverse and unique hairstyles and garments. Plus, they hug all the time,” said Liew, noting that this interaction was nearly as difficult to simulate as the “poof.”
Liew started his career in filmmaking with an internship at the San Diego Supercomputer Center, a hub of research and innovation located on the UC San Diego campus. That was followed by a stint at the Vancouver Film School, where he earned a diploma in 3D animation and visual special effects. But it was an undergraduate CSE course – Intro to Computer Graphics – that helped set Liew on this path.
“I took computer science because I really liked coding, but I never stopped thinking about movie-making,” said Liew. “In my junior year, I took a class with Professor Mike Bailey. It let me see that there is an option to mesh coding with movies.”
Oscar Nomination for Visual Effects
While Panda Mei with her rippling, red pelt is the latest Oscar hopeful with CSE ties, she stands on the computer-generated, furry shoulders of a tribe of apes. More specifically, she’s the animated descendant of the 2017 science fiction film, The War for the Planet of the Apes.
With explosive war scenes and an army of motion-captured, talking apes, The War for the Planet of the Apes was nominated for an Oscar in Best Visual Effects. Once again, computer scientists, including CSE professors Ravi Ramamoorthi and Henrik Wann Jensen, had a hand in making the effects – especially the rendering of the apes – more dramatic and realistic.
Ramamoorthi, the director of the Center for Visual Computing at UC San Diego, collaborated with Jensen, now an emeritus professor, and researchers from UC Berkeley to develop a method that dramatically improves the way computers simulate fur. Their paper, Physically-Accurate Fur Reflectance: Modeling, Measurement and Rendering, became the basis for rendering Caesar and his tribe of apes in The War for the Planet of the Apes.
Unlike previous models that were designed to create computer-generated hair, this model studied the passage of light through fur fibers. They looked at the medulla in fur, which is larger than in human hair, and at the way light bounces within an animal’s pelt.
The team’s algorithm rendered more realistic fur and offered a method that could be applied to everything from computer-generated special effects and computer-animated movies to video games
And the Oscar for Technical Achievement Goes to…
Before Panda Mei and Caesar, before new algorithms brought lifelike movement to computer-generated fur, moviegoers were enthralled by Steven Spielberg’s E.T. The Extra-Terrestrial.
Created using electric and mechanical models, puppets, costumes and a hand-mime, E.T. became an overnight and enduring sensation. For this, the film earned nine Oscars, including one for Best Visual Effects. That was 1983.
In 2003, Jensen, then a relatively new professor in computer science at UC San Diego, was building on work he had started in 1999. His 2001 paper on translucent materials explored techniques to graphically simulate the appearance of materials in the physical world. At the time, computer-generated images (CGI) tended to appear hard and distinctly un-lifelike. None of them could touch Spielberg’s E.T.
Jensen and his team theorized that the problem had to do with light. Existing CGI models assumed that light scatters at one surface point, an approximation that allowed computers to accurately simulate metals. In contrast, translucent materials exhibit significant light transport below the surface. To photo-realistically simulate translucent materials, new models needed to consider subsurface scattering. Just picture the light glowing through E.T.’s finger.
Jensen’s dipole diffusion model provided the answer. The model employed a rapid image-based measurement technique to determine the optical properties of translucent materials. It also enabled effects such as color bleeding within materials and light diffusion across shadow boundaries. These advances allowed the new model to simulate a variety of translucent materials with properties that replicated the biological world – items such as milk, fish and snow. And human skin.
Computer-generated human skin had immediate applications for visual effects movies. Jensen’s diffusion model was first used on Gollum in The Lord of the Rings trilogy. Later, it brought Dobby to life in the famed Harry Potter series. Jensen’s 2005 research on multi-layered translucent materials helped create the poignant characters in Avatar.
The Lord of the Rings: The Return of the King was nominated for eleven Oscars. It won them all. For his role in rendering Gollum’s skin, Jensen was awarded an Academy Award for Technical Achievement, together with Stephen R. Marschner and Pat Hanrahan, co-authors of A Practical Model for Subsurface Light Transport.
While Jensen’s pioneering research on computer-generated skin brought him into the Academy’s spotlight, it also laid the groundwork for Jensen and Ramamoorthi’s breakthroughs rendering hair and realistic fur. Jensen’s early hair rendering research was used on King Kong and subsequently paved the way for all movies featuring computer generated hair.
“I collaborated with Disney to create the artistic hair model for the movie Tangled, which was used throughout the movie on all the characters with hair,” said Jensen. “It also inspired the Disney material model that is used today for most materials in the movie industry.”
Throughout his years at UC San Diego, Jensen has had many students collaborate with him on his research and go on to Pixar, Disney, ILM and Weta. His first master’s student, Cyrus Jam (M.S. ’04), went to Industrial Light & Magic where he contributing to a variety of movies, including Star Wars.
Breakthroughs over the past two decades in rendering skin, hair and fur have made CGI’s more lifelike in their physical appearance. In turn, these human characteristics have made them more relatable, which could be the reason Gollum,
Dobby, Caesar, Panda Mei and other CGIs have captured the hearts of moviegoers everywhere.
copyright 2020 – Computer Science & Engineering – University of California San Diego
