The Presentation about the benchmarking framework of Iguana @ ISWC2017

1. Iguana
A Generic Framework for Benchmarking the Read-Write Performance
of Triple Stores
Felix Conrads1
Jens Lehmann2
Muhammad Saleem1
Mohamed Morsey3
Axel-Cyrille Ngonga Ngomo1,4
1
AKSW, Germany
University of Leipzig
2
University of Bonn and
Fraunhofer IAIS
3
System and Network Engineering Group,
University of Amsterdam
4
Department of Computer Science,
University of Paderborn
ISWC 2017
Conrads et al. Iguana ISWC 2017 1 / 25

2. Motivation
Semantic Web Stack
Conrads et al. Iguana ISWC 2017 2 / 25

3. Motivation
- RDF stores are backbone of Linked Data
- No realistic scenario (e.g. no stresstest)
- No unified benchmark execution
Conrads et al. Iguana ISWC 2017 3 / 25

4. Characteristics
- Relevant, testing typical operations in the specific domain
- Portable, executable on different platforms, benchmarks and
datasets
- Scalable, possibility of small and large data sets, with variable rates
of updates and concurrent users
- Understandable, returns results using standard measures
Conrads et al. Iguana ISWC 2017 4 / 25

5. Architecture
Overview
Metrics
Experiments
Result ProcessorCore
Triple Stores Result Storages
Query results
Conrads et al. Iguana ISWC 2017 5 / 25

6. Architecture
Stresstest
Conrads et al. Iguana ISWC 2017 6 / 25

7. Architecture
Stresstest Overview
Stresstest System
Q
U
SPARQL
Worker
SPARQL
Worker
UPDATE
Worker
UPDATE
Worker
Query
Manager
Update
Manager
Conrads et al. Iguana ISWC 2017 7 / 25

8. Architecture
Stresstest
- Provides a realistic scenario
- Simulates several Users, querying and updating
- Network latency can be simulated
- Warmup can be configured
- Time limit or Number of query mixes
Conrads et al. Iguana ISWC 2017 8 / 25

9. Architecture
SPARQL Worker
Conrads et al. Iguana ISWC 2017 9 / 25

10. Architecture
SPARQL Worker
Workflow:
1. Will start at a fair random query(pattern)
2. Executes each query(pattern) and starts at beginning
3. Does this until end of Task
supports SPARQL 1.1 (and Patterns)
Conrads et al. Iguana ISWC 2017 10 / 25

11. Architecture
Query Pattern Instantiating
1 PREFIX dct:<http://purl.org/dc/terms/>
2 SELECT ?s ?p FROM <http://dbpedia.org>
3 WHERE
4 { ?s ?p %%v1%% .
5 %%v1%% dct:creator %%v2%% }
6 LIMIT 10
will be converted to:
1 PREFIX dct:<http://purl.org/dc/terms/>
2 SELECT ?v1 ?v2 FROM <http://dbpedia.org>
3 WHERE
4 {?s ?p ?v1 .
5 ?v1 dct:creator ?v2}
6 LIMIT K
Conrads et al. Iguana ISWC 2017 11 / 25

12. Architecture
UPDATE Worker
Conrads et al. Iguana ISWC 2017 12 / 25

13. Architecture
UPDATE Worker
- Uses either a file with UPDATE queries
- Or a path with files containing SPARQL updates
- Files have to be in format:
1 [0-9]+.(added|remove).sparql
Conrads et al. Iguana ISWC 2017 13 / 25

14. Architecture
UPDATE Worker
Workflow:
1. Will start according to UPDATE strategy
2. will execute next update
3. Waits an amount of time
4. Does this until end of task
or no more updates
Conrads et al. Iguana ISWC 2017 14 / 25

15. Architecture
Metrics & Results
Conrads et al. Iguana ISWC 2017 15 / 25

16. Architecture
Metrics
- Queries per Second (QPS)
- Query Mixes per Hour (QMPH)
- No. Of Queries per Hour (NoQPH)
- Each Query Execution (EQE)
Conrads et al. Iguana ISWC 2017 16 / 25

17. Architecture
Results
- Results will be stored in files as CSV
- ... as NTriple file
- ... in a Triple Store
Conrads et al. Iguana ISWC 2017 17 / 25

18. Architecture
Extensible
It can be extended through
- Tasks
- Data Generators
- Query Handlers
- Metrics (KPI)
- Result Storages
Conrads et al. Iguana ISWC 2017 18 / 25

19. Configuration
- All test were executed on the same machine
- Datasets: DBpedia (100%,50%,10%) & SWDF
- 1 hour execution
- 20 min warmup
- 250 complex queries
- DBpedia Live changesets
- 1,4,16 SPARQL Users
- 0, 1 Update Users
Conrads et al. Iguana ISWC 2017 19 / 25

20. Baseline
10% 50% 100%
10
0
10
1
10
2
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph
Fuseki
Virtuoso
(a) DBpedia
10
2
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph
Fuseki
Virtuoso
(b) SWDF
Conrads et al. Iguana ISWC 2017 20 / 25

21. Parallel Request
1 4 16
10
1
10
2
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph
Fuseki
Virtuoso
(a) DBpedia 10%
1 4 16
10
1
10
2
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph
Fuseki
Virtuoso
(b) DBpedia 50%
1 4 16
10
1
10
2
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph
Fuseki
Virtuoso
(c) DBpedia 100%
1 4 16
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph
Fuseki
Virtuoso
(d) SWDF 100%
Conrads et al. Iguana ISWC 2017 21 / 25

22. Effect of Updates
10% 50% 100%
10
1
10
2
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph 0 update
Blazegraph 1 update
Fuseki 0 update
Fuseki 1 update
Virtuoso 0 update
Virtuoso 1 update
(a) DBpedia
10
2
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph 0 update
Blazegraph 1 update
Fuseki 0 update
Fuseki 1 update
Virtuoso 0 update
Virtuoso 1 update
(b) SWDF
Conrads et al. Iguana ISWC 2017 22 / 25

23. Update & Parallel Request
1 4 16
10
1
10
2
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph 0 update
Blazegraph 1 update
Fuseki 0 update
Fuseki 1 update
Virtuoso 0 update
Virtuoso 1 update
(a) DBpedia 10%
1 4 16
10
1
10
2
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph 0 update
Blazegraph 1 update
Fuseki 0 update
Fuseki 1 update
Virtuoso 0 update
Virtuoso 1 update
(b) DBpedia 50%
1 4 16
10
1
10
2
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph 0 update
Blazegraph 1 update
Fuseki 0 update
Fuseki 1 update
Virtuoso 0 update
Virtuoso 1 update
(c) DBpedia 100%
1 4 16
10
3
10
4
10
5
10
6
10
7
No.ofqueriesperhour
Blazegraph 0 update
Blazegraph 1 update
Fuseki 0 update
Fuseki 1 update
Virtuoso 0 update
Virtuoso 1 update
(d) SWDF 100%
Conrads et al. Iguana ISWC 2017 23 / 25

24. Summary
Iguana provides a unified SPARQL Benchmark Execution Framework
... can provide new insights for triple stores
... can be executed on every Dataset, Endpoint, and
SPARQL/UPDATE queries
Outlook
RDF Streaming
RDF Configuration
Any suggestions?
Conrads et al. Iguana ISWC 2017 24 / 25

25. Thats all folks!
Any Questions?
Iguana is Open Source
Github: http://github.com/dice-group/IGUANA
Project Site: http://iguana-benchmark.eu
Contact: conrads@informatik.uni-leipzig.de
Feel free to code with us!
Conrads et al. Iguana ISWC 2017 25 / 25