By RAY KELLY

Charles Foster Kane’s decaying pleasure palace is once again entertaining guests — courtesy of the wizardry of virtual reality.

Citizen Kane VR was launched at the University of Michigan in Ann Arbor, home to a massive collection of Orson Welles’ personal papers.  University faculty have re-created one of the Citizen Kane sets, Susan Alexander Kane’s bedroom, in virtual reality for use as a teaching and learning tool. (It was shortlisted for the 2020 International Conference of the Immersive Learning Research Network for innovative use of XR technology and instructional design.)

Using set design and continuity photographs from the Welles collections in the Special Collections Library, the Emerging Technologies Group at the university’s Duderstadt Center has created a realistic, three-dimensional grayscale virtual environment of the Xanadu bedroom set. Then, they used motion capture technology and a professional motion capture actor to reenact the action of the iconic scene in a studio.

The nearly three minute project reel was helmed by principal investigators Vincent Longo and Matthew Solomon, who worked with a VR visualization specialist, 3D artist, programmers and motion capture technicians.

“We chose Susan’s Xanadu bedroom for the pilot set of VR Citizen Kane because we had nearly a 360-degree view of the room from continuity photos taken during shooting. This allowed us to create a high fidelity model with minimal guessing,” Longo said. “The continuity photos reveal much of the set pieces/decor that are difficult to see when viewing the film alone.”

He added, “Currently this is the only set/scene available to users to explore and reshoot. Eventually our plan is to offer others, and we are currently replicating Mrs. Kane’s Boarding House.”

Users can experience the virtual scene with a VR headset (Oculus, Vive) or a desktop computer interface (either Mac or PC), pause and replay the action, and move the virtual camera, Longo said.

“The technology/platform we have developed could be used to replicate a scene from any movie,” Longo said. “We just need enough visual information to be able to create the whole set — not just what is pictured in the film.”

Ted Hall, a VR visualization specialist at Emerging Technologies Group, noted that low-mass wireframe props were used in place of heavier objects to stay on budget.

“This was accomplished on a miniscule budget compared to what Hollywood or the game industry could invest in such an endeavor,” Hall said.  “With an adequate budget, this could be as realistic as any contemporary VR game.”

According to lead 3D artist Stephanie O’Malley, one of the “unique processes” the team went through was trying to figure out the true shape and dimensions of Susan’s bedroom.

“No blueprints of this set exist to my knowledge, so we had to rely largely on improvised measuring techniques to determine the scale. Things like counting how many steps Orson Welles takes to cross the room, or figuring out a door’s height based on Orson’s height standing near it, all helped us to get a sense of how big the set would be,” O’Malley said. “The scale of the computer generated set is hugely important because ultimately, the size we determine the 3D set to be also impacts the area we have to work with when recording motion capture of an actor doing all of the motions Orson does throughout the scene.”

The team took the estimated measurements and created a 3D environment to those dimensions.

“Then we took those same measurements, and in a large video studio that we are fortunate to have access to on campus, we taped the blueprint of that room to the floor with gaffer’s tape. This gave us a real-world outline of the set that we could surround with motion capture cameras without obstructing them, and where we could place cardboard props at strategic locations for our motion capture actor to interact with,” O’Malley said. “Any object that Orson picks up and throws in this particular scene in the movie has a 3D modeled equivalent created and also a real-world cardboard version of the prop. With motion capture, the infrared cameras surrounding our taped outline of the set only pick up a series of reflective markers that the actor is wearing to track their movement — so the real-world props don’t have to actually look like the item from the movie, they just have to be roughly the correct dimensions so the motion capture actor knows where to move as they are re-enacting the scene, and so they know what they’re supposed to be throwing.”

She added, “If the scene requires Orson to chuck a suitcase across the room, the cardboard ‘suitcase’ prop may just be a box of about the same dimensions. It is up to the actor to make the box seem as heavy as a suitcase when they portray this part of the scene. It is also up to a skilled director — in our case, this was Vincent — to keep track of the various takes and ensure the timing is correct so the re-enactment aligns closely to the film.”

Motion capture actor Matthew Henerson flew in from California to be the avatar of Charles Foster Kane.

“Our motion capture actor was fairly close in resemblance to Orson Welles, so we actually used a process of 3d scanning called ‘photogrammetry’ to record his body using photographs and then a computer algorithm analyzes these photos to construct his likeness as a 3D avatar,” O’Malley said. “This is another modern Hollywood process often used to re-create CGI versions of deceased actors whose roles have continued on despite their death (Peter Cushing or Carrie Fischer in Star Wars: Rogue One) or to make actors younger/older. Big Hollywood production studios have hundreds of dedicated DSLR cameras for this purpose that are all positioned 360 degrees around the actor, and this allows them to zoom in and preserve all sorts of fine details in the actor’s likeness. Here at the university, our setup uses just three DSLR cameras that are all automated, and with some clever engineering of a turntable the actor stands on, we are able to get a decently comparable result as long as the subject can stay still for about a minute while the full rotation completes.”

She added, “In future versions of the app, we will have Susan as an option the students can select when determining which character they want to see re-enacting the scene.”

“All of this combines to give us a CGI re-creation of the live action film. The CGI avatar of Charles Foster Kane destroys Susan’s room over and over again on a loop, and students interacting with the application have the option to fast forward, rewind, pause, etc as this is happening, so they can reposition their virtual camera in new angles, adjust the focus or modify the frame rate, and begin filming,” O’Malley said. “All footage captured in the application is then exported as mp4 video files, which can be edited just like any other footage in their video editing software of choice.”

Susan’s bedroom at Xanadu and the Kane Boarding House may be only the beginning of Citizen Kane VR.