Enabling access to Linked Media with SPARQL-MM

Enabling access to Linked Media with
Thomas Kurz, Kai Schlegel, Prof. Dr. Harald Kosch
3rd International Workshop on Linked Media,WWW 2015, Florence

© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
The MICO Project and me
• www.mico-project.eu
• 36 Months Project
• started 10/13
• 3.4 Million Euro Project
• 7 Partners

The MICO project idea

Semantic Web Multimedia
Linked Media
Media
Fragment
Identiﬁers
Media
Annotation
Ontologies
5

Multimedia Retrieval
on theWeb of Data is ﬁnding
(fragments of) resources of an
structured and unstructured
nature (text, image, video,
concepts etc.) that satisfy an
information need.
Extension of the standard deﬁnition of information retrieval in:
Christopher D. Manning and Prabhakar Raghavan.An Introduction to Information Retrieval. Cambridge University Press, 2009.

Multimedia Query Languages
1. SQL-like approaches (WebSSQL or SQL/MM)
2. Object oriented approaches (MOQL or POQLMM)
3. Focusing XML metadata (MMDOC-QL or XQuery)
4. Visual Query Languages, like (MQueryVisual-MOQL)
5. Query-By-Example approaches, like WS-QBE
6. Metadata agnostic like MPQF
A Survey with ~40 MM Query Languages is coming soon …

Multimedia Query Languages:
Requirements and Features
• Spatial Operation
• Temporal Operation
• Freetext Search
• Functions: Aggreg., Sort., Group., Set-Fnct.
• Similarity Search (Top-k)
• Event / Object Concept
• Presentation Part
General Requirements:
Application Independency, Formal Semantics, Extendability, Closed, etc.
Speciﬁc Features:
• Fuzzy terms
• Relevance Feedback
• Camera Motion
• Object Motion (Trajectory)
• Query by Example (Image, Video,
Sketch, Humming)
• Condition Weighting
• Low-Level-Features (Audio, Video)

Semantic Web Query Languages
• There has been many in the past, but SPARQL
(SPARQL Query Language for RDF) is the winner
of the "natural selection".
• With SPARQL 1.1 many missing features have
become part of the standard.
• SPARQL is pretty good to extend (even without
changing the basic syntax)
BUT: It's missing some important features

What we want to reach
"Find scenes where Barack Obama is left beside the
Greenpeace Managing Director during an UN
climate conference ordered by length."
Features needed:
• Object Identiﬁcation
• Temporal Operation
• Spatial Operation
• Freetext Search
• Sorting by Temporal Selector

How could it look like?
"Find scenes where Barack Obama is left beside the Greenpeace MD during the
UN climate change summit ordered by length."
SELECT ?scene WHERE {
?a3 oa:hasBody ?event; oa:hasTarget ?s3. # there are resources about an event
?event schema:Event; schema:summary ?description.# which has a description
FILTER mm:fulltext-search(str (?description), "UN climate conference","en")
# about 'UN climate conference'
?a2 oa:hasBody :Barack_Obama; oa:hasTarget ?s2. # and there are resources about Obama
?a1 oa:hasBody ?p1; oa:hasTarget ?s1. # and there are resources
?p1 :ceo_of dbpedia:Greenpeace. # about the MD of Greenpeace.
FILTER mm:leftBeside(?s2,?s1) # Obama has to be at the left of the MD and
FILTER mm:intersects(?s3,?s2) # has to be appear at the same time like the event.
BIND (mm:boundingBox(?s1,?s2) AS ?scene) # Wrap the results to scenes
ORDER BY DESC(mm:duration(?scene)) # and show the longest first.

Goals
• Bring Multimedia Functionalities to SPARQL
(especially spatio-temporal support)
• Give transparent access to common
Metadata Models (Web Annotation
Selectors, Media Fragment URIs)
• Provide efﬁcient evaluation
• Make it more easy for users

Spatio-Temporal Object Model

Spatio-Temporal Object
Model: Example: Rectangle

Supported Standards
• Media Fragment URIs
• Spatial Fragment (currently Rectangle)
• Temporal Fragment (currently Interval)
• Web Annotation Model
• Media Fragment Selectors (http://www.w3.org/TR/media-frags/)
• SVG Basic Shapes (http://www.w3.org/TR/SVG/shapes.html)
http://example.org/video.mp4#t=1,5&xywh=10,20,30,30

SPARQL-MM Functions
Spatial Topological Relations
Spatial Predicates
based in the
Dimensionally Extended
nine-Intersection Model
+(DE-9IM)

SPARQL-MM Functions
Spatial Directional Relations

SPARQL-MM Functions:Temporal Relations

Multimedia Tests and
Accessors (not complete yet)
xsd:boolean mf:isMediaFragmentURI(Resource r)
xsd:boolean mf:isMediaFragmentSelector(Resource r)
xsd:boolean mf:isSVGSelector(Resource r)
mo:Rectangle mf:getBoundingBox(mo:SpatialEntity e)
xsd:double mf:getArea(mo:SpatialEntity e)
mo:Point mf:getXY(mo:Rectangle r)
xsd:double mf:getWidth(mo:Rectangle r)
xsd:double mf:getWidth(mo:Rectangle r)
xsd:double mf:getDuration(mo:TemporalEntity e)
xsd:double mf:getStart(mo:TemporalEntity e)
xsd:double mf:getEnd(mo:TemporalEntity e)

Fulltext Search
Based in Apache Marmotta Fulltext Search*
• mf:fulltext-search(text, query, [language]):
searches “text” for the words occurring in “query”, optionally
applying the language-speciﬁc processing for the given language;
query is a simple text literal (list of words) without any boolean
connectors; words are AND connected, i.e. all words have to be
found in the text for a successful match.
• mf:fulltext-query(text, query, [language]):
searches “text” using the boolean query string passed in “query”,
optionally applying language-speciﬁc processing for the given
language; query is a boolean query string following the syntax
used by PostgreSQL.
* http://marmotta.apache.org/kiwi/sparql.html

Status of Implementation
• SPARQL-MM 1.0 is out for a while, but:
• it is slow (based on Sesame Function Extensions)
• it is reduced
• it is very prototype (therefore still Snapshot)
https://oss.sonatype.org/content/repositories/snapshots/com/github/tkurz/sparql-mm/
• SPARQL-MM 2.0 will be available soon (planned for 8/2015)
• it is much faster (based on SQL-MM using PostGIS)
• it will be available as Open Source Marmotta Module*
• is has a big feature set
* maybe we should create a LIME Module that include implementations for various Linked Media
related things like Web Annotation endpoints etc. ?

Short Demo
<repositories>
<repository>
<id>oss-sonatype</id>
<name>oss-sonatype</name>
<url>
https://oss.sonatype.org/content/repositories/snapshots/
</url>
</repository>
</repositories>
<dependency>
<groupId>com.github.tkurz</groupId>
<artifactId>sparql-mm</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>

Future Plans
• SPARQL like for shapes
mm:like(mo:Circle(20,20,100), ?f)
• Extend Model to Layers
mm:behind(?f1, ?f2)
• SPARQL like for Resources
mm:like(?u1, ?u2, 0.4)

Thanks for your Attention!
Thomas Kurz
Knowledge and Media Technologies
Salzburg Research Forschungsgesellschaft m.b.H.
Jakob-Haringer-Straße 5/III | Salzburg,Austria
Tel. +43 662 2288-253| Fax +43 662 2288-222
thomas.kurz@salzburgresearch.at

Enabling access to Linked Media with SPARQL-MM

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Enabling access to Linked Media with SPARQL-MM

Similar to Enabling access to Linked Media with SPARQL-MM (20)

Recently uploaded

Recently uploaded (20)

Enabling access to Linked Media with SPARQL-MM