Project Name: Enhanced Communication in Simulation and Training

Project Leader

Gerald Penn

Project Co-Leader

Carl Gutwin

Researchers

• Nick Graham (Queens)
• Carl Gutwin (Saskatoon)
• Gerald Penn (Toronto)
• Cristina Conati (UBC)

Theme Distribution

• Theme 1 : 20%
• Theme 2 : 30%
• Theme 3 : 10%
• Theme 4 : 10%
• Theme 5 : 30%

Project Description

Immersive simulation and training environments increasingly involve groups of people acting in dynamic domains - for example, a military simulation might require members of a platoon to coordinate their actions during changing battlefield conditions. Such environments, particularly when carried out by distributed participants, often fail to capture the range of communication available to people carrying out a real task in a physical setting. Furthermore, synthetic actors (characters controlled by artificial intelligence) are now often part of these simulation environments, and here the communication problems are even more evident - these actors often communicate in jarring or cumbersome ways. Overall, problems of poor communication reduce the realism of a simulation, at times critically degrading the quality of the training.

To address this problem, this project will explore means of enhancing the richness of communication in immersive simulation and training environments, both verbal and non-verbal, and involving both humans and synthetic actors. Improved realism and richness in communication will in turn enable more natural behaviour on the part of the participants, which is then better predictable by user models developed through observation of human-human interactions (Nass, 2005; Pearson et al, 2006; Stytz & Banks, 2003). . In the first five years of the proposal, we will approach this topic through research into richness of communication in gaming, intelligent dialogue modelling in 'serious-game' battlefield simulations, and the use of ambient audio for communicating state information in virtual environments.

In all three areas, there is ample evidence that the lack of rich communication is responsible for poor collaborative interaction and a poor sense of immersion. One example is in the 'body language' given off by avatars - characters who are carrying heavy objects or are wounded, for example, should appear different from avatars who do not have these impediments. In addition, avatars must have realistic verbal and non-verbal social behaviour, not just in an operational context, but even in casual behaviours such as eating. In automated spoken dialogue systems, natural intonation, pauses and word choice elicit natural, and therefore more predictable responses, which reduces the interaction's reliance on elaborate error correction dialogues to train respondents. We will study the most effective ways to convey richness and a sense of immersion through the use of richer dialogue grammars, through richer graphical modes of interaction, and through selectively and automatically transitioning between different levels of richness in interface, each tailored to a participant's available amount of attention.

Excellence of the Research

Development of Highly Qualified Personnel (HQP)

This project provides excellent opportunities for students in computational linguistics, speech processing, HCI, and CSCW. These students will be gain experience and carry out important research in acoustic modelling, dialogue modelling, interaction design, distributed systems, visualization, and auralization. There are two main benefits to this project's training possibilities in particular. First, the project has a strong focus on an application environment that brings all of these research issues to bear on a working system. The students will also have opportunities for internships at VSIM, and similar facilities in other NATO countries with which CAE has partnerships. Second, the project brings together researchers on both sides of the communication equation - and the combination of training in both computational linguistics and HCI communication issues will result in with high levels of knowledge and skills.

Networking and Partnerships

CAE is involved in a NATO-centred network of military training and simulation developers which the project staff and students will have access through this collaboration. CAE has also recently adopted Bohemia Interactive's Virtual BattleSpace 2 as its serious gaming platform, which Bohemia Interactive has committed to supporting.

Knowledge and Technology Exchange and Exploitation

The project provides a direct benefit to Canada's defense and aerospace technology sectors. This technology also has civilian applications to video gaming and voice-controlled automation.

References to the Literature

Nass, C. (2005). Wired for Speech: How Voice Activates and Advances the Human-Computer Relationship. MIT Press.

Pearson, J., J. Hu, H. P. Branigan, M. J. Pickering and C. Nass (2006). Adaptive language behavior in HCI: How expectations and beliefs about a system affect users' word choice. Proceedings of the Conference on Human-computer Interaction (CHI 06), pp. 1177-1180.

Stytz, M. R., and S. B. Banks (2003). Progress and prospects for the development of computer-generated actors for military simulation: part I Introduction and background. Presence: Teleoperators and Virtual Environments, 12(3): 311-325.

Publications

Presentations

• 2 Minute Madness 2010