Presentation of Information - Multimodally with Multimedia

Advantages of Media Objects over Video

• Can customize content
  • Different audio for different languages
  • Replace video with image on mobile
• Can generate content dynamically
  • Ad insertion
  • Assemble from database based on user query

SMIL - a language to synchronise multimedia

• Layout: Positioning media objects
• Synchronisation: Timing of media objects
• Interactivity: User interaction with media objects
• Content control: Selection of media objects depending on client capabilities and user preferences

SMIL Timing and Synchronisation

• "seq": Objects in sequence
• "par": Objects in parallel
• Timing attributes

SMIL Sequential Composition

<seq>
  <audio src="rtsp://www.example.com/joe-audio"/>
  <audio src="tim-audio"/>
</seq>

SMIL Parallel Composition

<par>
  <video src="joe-video"/>
  <text src="http://www.example.com/joe-text"/>
</par>

Hierarchical Timing

Interactivity: Hyperlinks

• Support for HTML-like anchors

  <a href="bar">
    <img src="foo" />
  </a>

• Support for HTML-like image maps
• "Video maps"

  <video href="foo" />
    <area begin="5s" dur="5s" href="bar" />
  </video>

Content Control

• Goal: Adapt presentation to client capabilities and user preferences
  • Available bandwidth
  • Supported media types
  • Natural language
  • Captions for deaf
  • Screen size
  • ...

<switch>
  <audio src="foo" systemLanguage="fr" />
  <audio src="bar" />
</switch>

• SMIL V2.0 (HTML+TIME) example running
• SMIL V2.0 (HTML+TIME) example source

Multimodality

A current activity is considering devices that take gesture and voice input in addition to other interaction from users, and integrate them into a single dialogue. So far only position papers have been produced.