Multimedia document processing using the Weblab platform: AXES project use case. OW2con'15, November 17, Paris.
•
1 like•476 views
AXES (http://axes-project.eu) was a project co-funded by the European Union’s Seventh Programme for Research, Technological Development and Demonstration. The goal of AXES was to develop tools that provide various types of users with new engaging ways to interact with audiovisual libraries, helping them to discover, browse, search and enrich video archives. The OW2 WebLab (http://weblab-project.org) integration platform for multimedia processing has been used to aggregate the services developed by 9 partners in order to build the three demonstrators targeting various user groups: the media professionals, the journalists and researchers as well as the home users. As a result of the AXES project, OW2 "Open AXES", is born. It provides a complete solution that gathers innovative audiovisual content analysis technologies (shot and keyframes detection, image classification, speech transcription, large scale indexing, similarity search, etc.) as well as an ergonomic interface to retrieve and navigate in video archives.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Multimedia document processing using the Weblab platform: AXES project use case. OW2con'15, November 17, Paris.
Similar to Multimedia document processing using the Weblab platform: AXES project use case. OW2con'15, November 17, Paris. (20)
More from OW2
More from OW2 (20)
Recently uploaded
Recently uploaded (20)
Multimedia document processing using the Weblab platform: AXES project use case. OW2con'15, November 17, Paris.
- 1. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. AXES project use case 17th November 2015 Multimedia document processing using the WebLab platform Yann Mombrun – Bruno Grilhères Advanced Information Processing
- 2. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. Advanced Information Processing • Airbus Defence and Space • Airbus Group subsidiary • With Airbus and Airbus Helicopters • Advanced Information Processing team (TCOIC4) •10 engineers and Ph.D students •R&T/R&D for unstructured document analysis (and data fusion) Multimedia document processing using the WebLab platform 17th November 2015 2
- 3. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. Unstructured document analysis From unstructured data... … to structured and actionable knowledge • Complex processing chain: data gathering, data normalisation, translation, information extraction, information fusion, indexing and search… • Data variability: media, format, language… è Need an integration platform Multimedia document processing using the WebLab platform 17th November 2015 3
- 4. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. What is WebLab? An integration platform • allowing the integration of a selection of software components (search engine, information extraction, translation, knowledge management, graphical representation using maps/networks, etc.) • allowing the interoperation of the selected components. • based on recognised standards (SOA, Web Services, Semantic Web) • enabling the composition of services inside complex processing chains A set of media mining services for documentprocessing to be reused in all the WebLab projects. Multimedia document processing using the WebLab platform 17th November 2015 4
- 5. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. What is WebLab? Internal studies Multimedia document processing using the WebLab platform 17th November 2015 5 Customer oriented projects A range of projects in the fields of media-mining, NLP & KM
- 6. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. Content providers and end-users Technical partners Administrative coordination Focus on the AXES Project 17th November 2015 Multimedia document processing using the WebLab platform 6 FP7 ICT 2011-2014 http://www.axes-project.eu
- 7. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. Not just search: ExploreBrowse Experience A ( u s e r / c e n t e r e d ( a p p r o a c h Our$aim$is$to$open$up$audiovisual$digital$libraries,$increasing$their$cul= an$early$stage.$Targeted$users$are$media$professionals,$educators,$stu= dents,$amateur$researchers$and$home$users. Developing tools providing new engaging ways to interact with audiovisual libraries… 17th November 2015 Multimedia document processing using the WebLab platform 7
- 8. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. ... for a multitude of end users... Media professionals Researchers Home users ! ! ! Intelligence Officers 17th November 2015 8 Multimedia document processing using the WebLab platform
- 9. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. … building on state-of-the-art content analysis techniques… Computer vision Speech recognition / audio analysis Search & navigation Weakly-supervised methods d to define on. Simi- constraints fier. These diction and approaches n addition, d to define eline to ex- 1, see Fig- to present om the ex- ack identi- first use a en link the proach for ncontrolled Figure 1. An overview of our processing pipeline. (a) A face de- tector is applied to each video frame. (b) Face tracks are created by associating face detections. (c) Facial points are localized. (d) Locally SIFT appearance descriptors are extracted on the facial features, and concatenated to form the final face descriptor. nose, see Figure 1. We then extract SIFT descriptors at these nine locations at three different scales, which we con- catenate to form a feature vector f ∈ IRD of dimension D = 3 × 9 × 128 = 3456. As the descriptors are computed at facial feature points, it is robust to pose and expression changes. Using the SIFT descriptor makes it also robust to small errors in localization. 3.2. Metric learning from face tracks Given a set of face tracks we can extract face pairs from them to learn a metric over the face descriptors in an unsu- verview of our processing pipeline. (a) A face de- d to each video frame. (b) Face tracks are created face detections. (c) Facial points are localized. (d) appearance descriptors are extracted on the facial ncatenated to form the final face descriptor. ure 1. We then extract SIFT descriptors at ations at three different scales, which we con- rm a feature vector f ∈ IRD of dimension 128 = 3456. As the descriptors are computed re points, it is robust to pose and expression ng the SIFT descriptor makes it also robust to 17th November 2015 9 Multimedia document processing using the WebLab platform
- 10. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. … building on state-of-the-art content analysis techniques… People Places Object categories Events 17th November 2015 10 Multimedia document processing using the WebLab platform
- 11. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. AXES final architecture Multimedia document processing using the WebLab platform Normalisation Shot Detection Text Processing Speech Processing Video Content Linking Messaging and Distribution Orchestration Image Indexing Text Indexing Text, Links and Image Indices AXES Pro UI AXES Home UI AXES Research UI 17th November 2015 11
- 12. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. OpenAXES OW2 project • Free (or) Open source version of AXES è OW2 WebLab subproject • Proposed to EC and asked by end users • Demonstration purposes (by Airbus and AXES partners) • Baseline for a following project • Using partners code or replace by similar components • Not fully Open source yet (TEC and KUL) • Complementary with WebLab demonstration è Focus on multimedia processing Multimedia document processing using the WebLab platform 17th November 2015 12
- 13. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. OpenAXES Free (or) Open Source Version of AXES Based on OW2 WebLab Already available in the current version : • Video Normalisation (Airbus DS / FFMpeg) • Shot/Scene detection (TEC) • Image concept extraction (KUL) • Spoken word & Metadata search (UT) • Speech to Text (En&Fr) (Airbus DS / Sphinx / LIUM) • Similar search (Airbus DS / Pastec) • On-the-fly search (UO) • Favorites / Like/ Most viewed / Video cutter (DCU) • Recommendations (UT) • Easy installation on Ubuntu 14.04, 15.10 and Mint 17 17th November 2015 Multimedia document processing using the WebLab platform 13
- 14. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. Demonstration Multimedia document processing using the WebLab platform 17th November 2015 14
- 15. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. Questions? Multimedia document processing using the WebLab platform 17th November 2015 15
- 17. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. OpenAXES Links • Hosted on OW2 Forge http://forge.ow2.org/projects/openaxes/ • Deliverables Versions http://forge.ow2.org/project/showfiles.php?group_id=436 – OpenAXES-1.1.0 2015-07-07 – OpenAXES-1.0.1 2015-05-19 – OpenAXES-1.0.0 2015-04-01 – OpenAXES-0.6.0 2015-02-16 – OpenAXES-0.5.0 2015-01-30 – OpenAXES-0.2.0 2014-12-16 • SVN svn://svn.forge.objectweb.org/svnroot/openaxes • Continuous Build http://bamboo.ow2.org/browse/OPENAXES • Bug Tracking http://jira.ow2.org/browse/OPENAXES
- 19. ©2014AirbusDefenceandSpace–Allrightsreserved.Thereproduction,distributionandutilizationofthisdocumentaswellasthecommunicationofitscontentstootherswithoutexpressauthorizationisprohibited.Offenders willbeheldliableforthepaymentofdamages.Allrightsreservedintheeventofthegrantofapatent,utilitymodelordesign. AXES Datasets • Year 1 • Mix of NISV, DW and BBC content: 40h • Year 2 • NISV PRO: 400h • BBC Eastenders:500h • BBC PRO: 480h • Year 3 • BBC Home: 1000h • DW RES: 100h • NISV RES: 1000h • Year 4 • NISV Home: 1000h • DW Home: 200h • BBC Benchmarks:400h • CNN Internet Archive: 100h