Primary Challenge

AnImage

Champions

Leaders

Description

Distributed human communication and simultaneous interaction with synthetic media continue to expand into new collaborative applications, including surgical planning, simulation training, and entertainment. In the context of these demanding human activities, we are critical of the capabilities of the current state of the art technologies. Higher-end videoconferencing systems have recognized the importance of image size and the presentation of a common visual background to enhance the participants\' sense of telepresence. Nevertheless, these only render a single perspective view that lacks support for gaze awareness, impose a perceptible delay that stilts natural dialogue, and limit the mobility of participants. As a result, these systems often prove inadequate for supporting group discussion or highly collaborative activity, especially between more than two sites. The problems are not solely ones of bandwidth or video quality, but equally of the necessary display and software architecture suited to such applications.

Similarly, the integration of remote avatars or virtual actors into shared spaces often imposes constraints on the realism, believability and artistic control of rendered content, while real-time data acquired during interaction and performance is not fully exploited in current post-production systems. These issues affect the realms of cinematic, theatre, and opera performance and production.

This project is working on the next generation of virtual presence and live performance technologies in a manner that supports the task-specific demands of communication, interaction, and production. The goals are to: improve the functionality, usability, and richness of the experience; support use by multiple people, possibly at multiple locations, engaged in work, artistic performance, or social activities; and avoid inducing greater fatigue than the alternative (non-mediated) experience.

Realizing these objectives entails further development and integration of several enabling technologies, including video acquisition and display architectures, spatially reactive yet controllable lights and cameras, tetherless tracking, video segmentation, multimodal synthesis, latency-reduction techniques, and novel GIS-like production interfaces. Numerous challenges must be overcome, including the seamless integration of video display and presentation of 3D content, visible from multiple angles. Spatially appropriate audio is equally important, especially in the context of human-human communication and the haptic modality (sense of touch) should also be supported. Participants must be allowed to move freely, while continuing to experience the relevant sights, sounds, and ground texture, as they would in a \"real\", non-mediated environment. Finally, recognizing the importance of delay minimization in distributed complex group activity, the project will investigate several mechanisms to reduce the impact of network latency.

Our methodology consists of an iterative design-implementation-evaluate cycle, in which prototype tools will be integrated in applications and deployed in real-world performance and collaborative design activities at an early stage, allowing for meaningful evaluation outside of an artificial laboratory environment. From this ongoing test feedback, the tools will be refined, with subsequent validation conducted on each iteration. This implies that efforts toward the project milestones, listed below, will proceed largely in parallel, with evaluation conducted initially on existing tools developed outside of our project, as a basis for comparison with our ensuing developments.

In the first year of this project, we have met several times as a project group using the Mingleverse group conferencing tool developed by our project champion. Several of our network investigators have also met in person, both at the GRAND AGM and in separate visits in Vancouver. Our efforts have concentrated largely on developing the technology components at the heart of the envisioned hardware and software platform needed to support the objectives above. Two primary applications serve as the use cases that motivate the particular tools, the World Opera and digital media production.

To date, we have developed a processing pipeline that extracts the video of performers from their background, transports this data to its intended destination(s), scales the image as appropriate to the performer\'s position in the scene, composites the video against a synthetic background, common to all sites. In addition, the performer\'s position, as determined from the video acquisition, is used to render spatialized audio output so that the corresponding voice e.g., of an opera performer, appears to originate from the correct location.

A prototype script-centered, zoomable interface is also under development that allows semantic organization and access to pre- and post-production media, for coordinating the digital media production pipeline. The overall goal here is eventually to form an integrated interface that extends from storyboarding, to pre-production setup, to the real-time production of content, and into post-production editing and filtering. To date we have implemented several key features of this script-centered, zoomable interface including tying supplemental materials to points in the script. These supplemental materials are “hot” in the sense that one can slide directly into a 3D view of a pre-visualized scene from the script-based interface and walk through the scenes, or zoom in and flip through a sequence of storyboard sketches. We also constrain the interaction, so that it is not a fully open-ended zoomable interface, but has vertical channels, with discrete snapping levels (though continuously animated) and with ties across channels for selection and annotation. Further features that are currently being added relate to the visual organization of the script and embedded materials. For instance, because only part of the script and the associated materials are visible at any time (depending on the zoom level) we are implementing a collapsing feature that is script aware. This is necessary since simple linear collapsing would not be sufficient, given that the filming of scripts is often non-linear. For example, scenes are often shot in groups based not on story timelines, but based on location and casting.

To coordinate the video acquisition with high fidelity, we are presently working on steerable camera array control, which integrates on OpenCV-based blob tracker with one or more pan-tilt-zoom cameras to follow performers, and then, shares the position information with other cameras to enable acquisition from multiple view angles, under both automated and manual (directorial) control. Our plan is then to incorporate a basic constraint satisfaction framework that allows for intelligent camera control based on a variety of factors, including the number of performers to track, manually specified preferences for view angle of each performer, and continuity. Additional work by Fels on a multi-camera system may satisfy some of the requirements for this objective.

Similarly, for high fidelity video rendering, that is, \"painting the pixels where they\'ll be most valuable\", three of the project investigators have partnered on an NSERC RTI application to acquire steerable video projectors for research in this area.

Looking ahead to the next year, a fundamental challenge in distributed performance, such as World Opera, is to present a display of performers across multiple sites that supports closely coupled interaction. This includes eye contact, synchronized audio, and performer mobility across the stage. At the same time, the media presentation must be compelling for the audience, which means that local performers should not have to turn their backs to the audience in order to engage in a face-to-face duet with other (remote) singers. For this reason, conventional display techniques, such as a projection screen at the rear of the stage, are inadequate. Instead, we must begin experimentation with new staging architectures and mobile display surfaces that provide differentiated content at different viewing angles, much like autostereoscopic and hologram techniques. In this manner, two singers, one real (local) and a second virtual (remote) could face each other while the audience sees their profiles. Investigation and experimentation with a range of technologies capable of achieving these effects will constitute an important component of the project, and hopefully entail expanded cooperation with the SHRDSP project.

In terms of latency-reduction evaluation, the tightly coupled nature of musical interaction offers a challenging and meaningful test domain. Here, we propose two techniques for investigation. The first achieves latency normalization between the conductor and all end-points by adding delay as necessary to the \"shorter\" network paths. This ensures that all performers receive their visual cue simultaneously, thereby reducing the worst-case latency between any two sites. The second technique builds on anticipatory video encoding of the conductor\'s gestures, based on machine learning techniques and motion capture, to deliver the visual cues in advance of the actual completion of the gesture. While this does not address the auditory delay between performers due to the network, it can help reduce or eliminate the conductor-to-performer delay (as well as that for local-to-conductor performers), which has significant impact for local audience as well.

Regarding digital media production in the coming year, we aim to integrate multi-touch capabilities into the zoomable interface thereby allowing directors and creative talent access to assets onsite. We will also explore the automated organization of digital footage based on the matching of the audio found in film takes to the script. This would limit the amount of raw footage the user must sift through to select clips during editing. The user would be able to select text from the script and the system would find matches based on the audio analysis of speech. In addition, because there would typically be many shots of the same dialogue under different conditions, we would further visually structure the returned set of matches according to dimensions of similarity. These dimensions of similarity may include the duration of the spoken text, close-up to wide-angle shots, the contrast ratio, script precision and the amount of motion in the shot. During editing the user could then move through the bulk of footage in a structured and meaningful fashion. Finally, assuming that our NSERC Equipment RTI is successful, we also aim to begin integrating our script centered interface with the real-time tracking of scene elements, camera focus control and projection based directorial control.