Lighting Design and Technology student explores xR extended reality production
Jeremy Mayo uses hi-tech to create virtual reality on stage, similar to Disney's "The Mandalorian"
The Lighting Design and Technology program at the University of Cincinnati’s College-Conservatory of Music is teaching students how to create extended reality productions that use digital media as virtual sets for actors on stage.
By now, you may have seen The Mandalorian on Disney+. You know, the Star Wars spin-off featuring “Baby Yoda” (no spoilers on his real name) and out-of-this-world visual effects. The series was one of the first to use a groundbreaking extended reality (xR) production method that is now being explored at CCM.
Extended reality (xR) is a production technique for the stage and screen where virtual scenery is rendered around an actor or subject in real-time from the perspective of the video camera capturing it. The technique is similar to working with a green screen that uses digital environments, but unlike a green screen, xR environments are actually placed in front of the camera lens.
Lighting Design students work with the virtual set created for Jeremy Mayo's "Project Holodeck." Photo/Taylor Durantini.
“When first picturing this concept, it is easy to think about this workflow in terms of a backdrop,” says MFA Lighting Design and Technology student Jeremy Mayo, who is exploring xR production for his master’s thesis project. “Instead of painting a backdrop for actors to stand in front of, we can replace that backdrop with a video screen of sorts, which is able to change to show any content that is needed.”
The digital scene is programmed onto screens that make up the set behind the actor. Directors and performers in an xR production can see the digital environment in real time instead of imagining it. Additionally, the xR digital sets create their own light that matches the environment, whereas green screen productions require significant lighting and post-production editing in order to create realistic lighting effects.
“Since the computer systems needed for xR are extremely powerful, it is possible to rapidly recreate digital worlds,” Mayo adds. “Artists working with extended reality setups are able to rebuild scenes in a few minutes or even a matter of seconds depending on the change needed. This is an incredible tool as directors can request changes to the shots and environments they are filming without wasting any additional production time.”
Mayo first explored xR production techniques over the summer while researching The Mandalorian. He was immediately captivated by the technology and wanted to explore how it could be used in the performing arts. He set out to create three virtual scenes for his “Project Holodeck”: a graveyard, a firework display in the woods and a studio production set.
Lighting Design students work with the virtual set created for Jeremy Mayo's "Project Holodeck." Photo/Taylor Durantini.
“When the pandemic started, it completely decimated the entertainment industry (and it still is),” Mayo says. “Typically, in a normal timeline, I would design a large-scale production as a thesis project. However, the pandemic prevented CCM from staging [large-scale productions] this year. After discovering xR via Disney's The Mandalorian, I immediately knew xR could and would have a massive effect on how we create art, particularly during a pandemic, so I chose to see if I could learn and produce a production using an xR stage that I built myself.”
Last spring, CCM approved a major media software and hardware acquisition for the lighting department to make sure students have access to professional-quality tools of the industry. This made it possible for Mayo to explore xR at CCM this fall.
To create a proof of concept functional xR stage, Mayo used UnReal Engine, a 3D video game creation tool and management software. He also used the Disguise Media Server, which is a software/hardware solution for managing complex video and projection systems. Additionally, Mayo used an HTC Vive Tracker so the virtual environment on stage and the video camera shared positional data with each other — this made the set responsive so when the camera moved the whole 3D world moved on stage.
“This was my first time working with many of the tools and working specifically with xR,” Mayo says. “Though there is some crossover into traditional lighting and projection design. I'd say my experience as an entertainment designer allowed me to approach this project more easily.”
CCM Professor of Lighting Design and Technology Sharon Huizinga says it’s important that students are familiar with this technology because it’s being used in a myriad of ways throughout the industry. Recent examples include Katy Perry’s performance on American Idol, the 2020 MTV Video Music Awards and Billie Eilish’s “Where do we go? The Livesteam” concert. “Although it’s existed before, now it’s sort of blown up because it’s the way forward if you want to do a major festival or live concert or awards show,” Huizinga says.
Lighting Design students work with the virtual set created for Jeremy Mayo's "Project Holodeck." Photo/Taylor Durantini.
“I look at my program in terms of maximum employability for my students and to me, it would be unconscionable to ignore the media aspect of it,” she adds. “There’s a million jobs in lighting, but there is also a million jobs in projections. If someone isn’t into extended reality, you can still make content for rock shows, or maybe you want to be a Broadway projection designer, or maybe you don’t like any of that and just want to get into the equipment.”
She believes in partnerships with manufacturers so that students have unique access to the latest technology available in the industry. She has a partnership with Disguise to give CCM student access to the necessary hardware and software needed to explore xR production techniques. Huizinga has also hosted master classes with guest artists from Disguise to connect students with a professional network.
“I would like to create a framework where students have maximum access, and they have pathways to develop relationships with designers, experts, specialists, manufacturers and rental houses,” she adds. “I’m just saying ‘hey you need to know that, I’m going to bake it into one of my classes so that you have exposure to it.’ It feels like a great way forward.”
Mayo plans to pursue xR jobs as one option after he graduates, thanks to the opportunity to work with the production technique at CCM under Huizinga’s guidance. “A good professor makes you a better student, a great professor makes you a better person, the best does both,” he adds. “That’s Sharon.”
“This is one major success story of ‘wow, here’s a student who is using this equipment that no more than a tiny handful of people on an academic level in the U.S. have access to,” Huizinga says of Mayo’s work. “Students are walking out with a big degree of confidence that is uncommon. The field of choice just got massive — do you want to go into Broadway or dance performances, live concerts, architecture lighting or award shows, or light art festivals? I feel like we just doubled the job opportunities for every student in the lighting department.”
About CCM Lighting Design and Technology
CCM’s Department of Theatre Design and Production (TD&P) offers both a bachelor of fine arts (BFA) and master of fine arts (MFA) with specialization in Lighting Design and Technology. The production and the actors are inexpressive unless audiences can see them on stage. Good lighting design goes beyond the need for basic visibility into the realm of shadows, angles and colors as well as the subtleties of timing and rhythm. Lighting designers work closely with stage directors to bring out the intent of the production through mood atmosphere, composition and focus. Technicians must work with the sensitivity of an artist so that onstage dramatic effect and timing are achieved and maintained.
At CCM, a hands-on, experiential philosophy guides the development of all degree programs. Theoretical knowledge forms a solid base, but is only the beginning of the education process. A well-rounded student experience includes real production opportunities. Students in the four-year undergraduate program take studio and production courses in all areas, beginning with introductory and basic courses and progressing to advanced sequences in a major area. Additionally, students study theatre history, script analysis, dramatic literature, English, world history and arts history, as well as other liberal arts and electives. Graduate students follow a similar, more specialized structure in a two-year (or three-year) program.
Learn more by visiting ccm.uc.edu
Featured image at the top: Lighting Design students work with the virtual set created for Jeremy Mayo's "Project Holodeck." Photo/Taylor Durantini.
Related Stories
University of Cincinnati celebrates DAAP's class of 2024:...
April 27, 2024
Discover the achievements of the University of Cincinnati's College of Design, Architecture, Art, and Planning's Class of 2024 as they graduate into the alumni family, showcasing exceptional talent and innovation. From prestigious awards to prominent job offers, these graduates exemplify the transformative power of creativity and dedication in shaping tomorrow's leaders.
Ancient Maya blessed their ballcourts
April 26, 2024
Using environmental DNA analysis, researchers identified a collection of plants used in ceremonial rituals in the ancient Maya city of Yaxnohcah. The plants, known for their religious associations and medicinal properties, were discovered beneath a plaza floor upon which a ballcourt was built, suggesting the building might have been blessed or consecrated during construction.
UC celebrates record graduating class at commencement
April 26, 2024
UC celebrated its doctoral hooding and master's recognition ceremony at Fifth Third Arena as part of its three-day commencement for the largest graduating class in university history.