Talk given at the "Séminaire Muséologie: Le Web devient audivisuel" at IRI on Tuesday 10/04/2012, Paris, France

1. Addressing and annotating
multimedia fragments
Raphaël Troncy <raphael.troncy@eurecom.fr>

2. Use Case
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -2

3. Use Case
 Aidem received on her Facebook
wall a status message containing
a Media Fragment URI
 Use a ‘#’ !
 Highlight a
video
sequence
 Highlight a
region
to pay
attention to
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -3

4. What are Media Fragments?
0 20 temporal media fragment 35 t
spatial media fragment
track media fragment
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -4

5. Temporal Media Fragments
Original resource
length
Fragment beginning Playback progress Fragment end
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -5

6. Spatial Media Fragments
highlighted
fragment
semi-opaque
overlay
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -6

7. Media Fragments Use Cases
 Bookmark / Share parts (fragments) of
audio/video content
 Annotate media fragments
 Mash-ups
 Conserve bandwidth
http://www.w3.org/TR/media-frags-reqs/
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -7

8. Requirements
 r01: Temporal fragments:
 a clipping along the time dimension from a start to an end time that
are within the duration of the media resource
 r02: Spatial fragments:
 a clipping of an image region, only consider rectangular regions
 r03: Track fragments:
 a track as exposed by a container format of the media resource
 r04: Named fragments:
 A temporal media fragment that has been given a name through
some sort of annotation mechanism
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -8

9. Media URIs
 Using URI query part:
http://www.example.org/video.ogv?t=60,100
 Using URI fragment part:
http://www.example.org/video.ogv#t=60,100
 Mixing both:
http://www.example.org/video.ogv?t=60,100
#t=10,15
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -9

10. URI Fragments vs. URI Queries
#t=20,30 ?t=20,30
secondary resource, primary resource,
notion of context no notion of context
extraction needs to be no adaptation restrictions
expressible in byte ranges
no provisions for key-value pairs are sent to
communi-cating the server
fragments to the server
potentially cacheable not cacheable
 The media fragment URI syntax can be used
for URI queries
 We will focus on URI fragments
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 10

11. Media Fragments Resolution
 For the URI query part:
 The media file is only processed on server side
 The UA receives a new video file
 For the URI fragment part:
 Smart UA may strip out the fragment definition and
encode it into custom http headers (Range header)
 (Media) Servers will handle the request, slice the media
content and serve just the fragment (corresponding byte
ranges)
… while old ones will serve the whole resource
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 11

12. Influence of Media Formats
 Fragment extraction needs to be expressible in
terms of byte ranges
 Requirements for the different axes
 temporal: presence of intra-coded frames
(i.e., random access points)
 spatial: presence of independently coded spatial regions
 track: need to be identifiable by a name
 Conclusion: temporal and track axes are
realistic, spatial fragments can hardly be
expressed in terms of byte ranges
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 12

13. Can I use Media Fragments now?
<video id='v'
src='AudioAPI.webm#t=50,100'
onloadedmetadata='update()'
onpause='update()'
onplay='update()'
onseeked='update()'
controls></video>
 Firefox 9+ (or WebKit)
 No bandwidth saving
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 13

14. Can I use Media Fragments now?
 Sysnote over YouTube
 No bandwidth saving
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 14

15. Can I use Media Fragments now?
 Ninsuna
 Bandwidth
saving
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 15

16. Examining HTTP Traffic
 HTTP request:
 retrieving URI
 parsing key=values pairs from fragment part
 setting Range header
 HTTP response:
 checking Content-Type
and Content-Range-Mapping headers values
 attaching custom playback controls to page
 creating spatial dimension overlay (if specified)
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 16

17. Example: Requesting a Time Fragment
 A web developer specifies a video source with
a temporal fragment URI:
http://ninsuna.elis.ugent.be/DownloadServlet/mfwg/fragf2f.ogv#t=5,15
 key=value pair is analyzed,
fragment begin and end time are matched
t=5,15
 Media Fragments Extension analyses the
fragment part, retrieves beginning and end time
and sets proper Range header value:
Range: t:npt=5-15
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 17

18. Example: Requesting a Time Fragment
 The NinSuna server responds with the 206
Partial Content response and Content-Range-
Mapping header showing the mapped time
ranges and media fragment in the message
payload:
HTTP/1.1 206 Partial Content
Content-Type: video/ogg
Accept-Ranges: bytes, t, track, id
Content-Range: bytes 629578-1690588/4055466
Content-Range-Mapping:
{t:npt 4.8-14.8/0-38.33}={bytes 629578-690588/4055466}
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 18

19. Example: Requesting a Time Fragment
 ... and it won’t work!
 Because the
player does not
have the media
file header
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 19

20. Example: Requesting a Time Fragment
 Player needs to be initialized, thus bare
fragment is not playable
 To fix this, we add ;include-setup to the Range
header value:
Range: t:npt=5-15;include-setup
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 20

21. Example: Requesting a Time Fragment
 The response from the server is slightly different:
HTTP/1.1 206 Partial Content
Content-Type: multipart/byteranges;boundary=End
Content-Range-Mapping:
{t:npt 4.8-14.8/0-38.33;include-setup}
={bytes 0-5998,629578-1690588/4055466}
...
--End
Content-Type: video/ogg
Content-Range: bytes 0-5997/4055466
{binary data}
--End
Content-Type: video/ogg
Content-Range: bytes 629578-1690588/4055466
{binary data}
--End
--End--
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 21

22. Example: Requesting a Time Fragment
 This is why we need to attach a Stream Listener
to the HTTP channel:
HTTP/1.1 206 Partial Content
Content-Type:
multipart/byteranges;boundary=End
...
HTTP/1.1 206 Partial Content
--End
Content-Type: video/ogg Content-Type: video/ogg
Content-Range: bytes 0-5997/4055466 ...
{binary data} {binary data}
--End {binary data}
Content-Type: video/ogg
Content-Range: bytes 629578-
1690588/4055466
{binary data}
--End
--End--
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 22

23. Spatial Fragments
 Without additional information send to server
 Overlay is created by appending four additional
DIVs to the webpage
 They are styled to create the impression of
semi-opaque layer over the video element
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 23

24. NinSuna: Overview
 Fully integrated media adaptation and delivery
platform
 media adaptation and media packaging core is
independent of media formats
based on model-driven content adaptation & delivery technique
 only high-level adaptation operations such as scene
selection and frame rate scaling
no transcoding is applied
ideal candidate for MF implementation
 more information: http://ninsuna.elis.ugent.be
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 24

25. MF-specific Features of NinSuna
 Support for Media Fragment URI queries
 both HTTP and RTSP implementation
 try some URIs at
http://ninsuna.elis.ugent.be/MediaFragmentsServer#Test
http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_t
railer_2.mp4?track=5;6
 Support for Media Fragment URI fragments
 i.e., support for the MF-specific HTTP headers
 Combining Media Fragment URI queries and
fragments
 e.g., http://foo.com/media.mp4?t=10,40#t=5,10
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 25

26. Impact on the Existing Web Infrastructure
 Updating existing infrastructure is not trivial
 web servers need to be extended with
media extractors
MF-specific HTTP communication
 existing web caches are only effective with HTTP byte
ranges
specialized media caches need to be developed in the future to
cache the other cases
 user agents need to be extended with
a Media Fragment URI parser and interpreter
MF-specific HTTP communication
MF-specific visualization
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 26

27. Media Fragments Proxy
 Goal: make existing media resources, served
by generic HTTP Web servers, available as
Media Fragments
 Biggest problem for content providers
willing to support Media Fragments
 media extractor (dependent on underlying media
formats)
 Solution: an approach that
 works with existing HTTP Web servers (apache, IIS)
 works with existing Web caches
 works with not so smart user agents
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 27

28. Media Fragments Proxy http://foo.com/video.ogv#t=11,19
GET /?url=http://foo.com/video.ogv HTTP/1.1
Host: MFProxy.com
HTTP Accept: video/*
server Range: t:npt=11-19
Accept-Range-Redirect: bytes
HTTP/1.1 307 Temporary Redirect
Location: http://foo.com/video.ogv
-get (only) the header info
Accept-Ranges: bytes, t, track
Content-Length:a fragment-to-byte mapping
-try to find 0
Content-Type: video/ogg
-construct the redirect response
Content-Range-Mapping: t:npt 10-20/0-50
Range-Redirect: 24000-32000
Vary: Accept-Range-Redirect
GET /video.ogv HTTP/1.1
MF Proxy Host: foo.com
MF User Accept: video/*
Agent Range: bytes=24000-32000
http://ninsuna.elis.ugent.be/MFProxy?url=<mediaURI>
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 28

29. Annotating Media Fragments yuma.min.js
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 29

30. Annotating Media Fragments
Benchmarking: Sphinx, HTK,
Julius
 NER + full text index with the
transcription
 Interlinking with the Linked Data
Cloud to enable semantic search
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 30

31. 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 31

32. http://www.slideshare.net/troncy
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 32