Query Soundness Experiment

This page describes the program and the files for the experimental evaluation of checking query soundness using completeness statements. The main reference of the experiment which contains the formalization, techniques, and discussion of the experimental results is under review for a journal.

Graph

We use the direct-statement fragment (i.e., with no qualifiers nor references) of Wikidata, consisting of around 110 mio triples: Wikidata dump in Feb 2016. In the experiment, we use Jena-TDB as the triple store. For convenience, the zipped TDB files that store the data graph for the experiment is available here.

Queries

The queries in the experiments are generated from the human-made queries openly available on Wikidata: Wikidata queries in April 2016 We extract the BGPs of those queries and transform the vocabulary of the queries to the direct-statement vocabulary. Depending on the cases, there are three sets of queries which differ only in queries that we remove due to cross-product joins wrt. the respective cases:

Query set for oneTP and oneTPoneTP cases: queries-one.txt
Query set for TwoTPs case: queries-two.txt
Query set for ThreeTPs case: queries-three.txt

From these queries, the experiment program will generate on the fly the corresponding queries with negation, depending on the cases, which we will later check for their soundness.

Completeness Statements

The completeness statements (and completeness templates) are generated on the fly by the experiment program based on the queries.

Experiment Program

The reasoning program and experiment framework are implemented in Java using the Apache Jena library. The source code is available as an Eclipse workspace.
The ready-to-use JAR file of the program is available here.

To run the program, the folder of the TDB files (‘tdb-PREPROCESSED-wikidata-simple-statements-20160201’) must be put together in the folder of the JAR file. The command scheme is as follows:

java -jar soundness-experiment.jar [observation repeats] [warming up] [case] [query filename]

The parameters used are:

observation repeats: how many repeats have to be performed to take measurements (for reliability)
warming up: is warming up needed (1 for yes, 0 for no)
case: the case of the experiment (1 for oneTP, 2 for oneTPoneTP, 3 for twoTPsTO, 4 for twoTPsSE, 5 for threeTPsTO)
query filename: the query file of the experiments

In the experiments, we run the following commands for the cases oneTP, oneTPoneTP, twoTPsTO, twoTPsSE, and threeTPsTO, respectively:

java -jar soundness-experiment.jar 10 1 1 queries-one.txt > case-oneTP.txt

java -jar soundness-experiment.jar 10 1 2 queries-one.txt > case-oneTPoneTP.txt

java -jar soundness-experiment.jar 10 1 3 queries-two.txt > case-twoTPsTO.txt

java -jar soundness-experiment.jar 10 1 4 queries-two.txt > case-twoTPsSE.txt

java -jar soundness-experiment.jar 10 1 5 queries-three.txt > case-threeTPsTO.txt

Note the experiment output is buffered via ‘>’. The summary of the experiment results is of the CSV-style, where the rows represent queries and the column names are: query length, number of query answers, query evaluation time, number of sound answers, query evaluation time performed when checking answer soundness, answer soundness checking time, total answer soundness checking time (incl. query evaluation time), is pattern sound, and pattern soundness checking time. This summary is provided at the bottom of the output of the experiment program under the section “STATISTICS”.

~this manual file is created by Fariz Darari, email: fariz.darari@stud-inf.unibz.it