Evaluation of pattern-based ontology engineering approaches

Using ontology design patterns and pattern based approaches for engineering ontologies is believed to increase the efficiency and quality of constructing and maintaining ontologies. The pattern approach allows for interacting with the ontology at a higher level of abstraction than the low-level formats provided by RDF and OWL, with the expected results that recognised best-practice modelling principles are more easily followed. Also, the increased regularity the ontologies gain from a pattern-based approach is believed to lower the cost of using, maintaining and updating the ontology.

Yet, there are no established metrics for evaluating the efficiency and quality gained for ontology engineering tasks from pattern based approaches. Also, no benchmarks for comparing tools and methods that provide support for pattern based approaches to ontology engineering are available.

Tasks

develop formal metrics and methods for evaluating and comparing different pattern based ontology engineering approaches
develop a (small) benchmark for evaluating pattern based approaches using well known ontologies and/or knowledge bases that preferrably represent different ontology modelling approaches and domains (e.g., from generated and highly regular, to manually built with few repeated patterns)
evaluate a selection of different tools for pattern based ontology engineering, including OTTR, using the developed metrics and benchmark
publish the benchmark and evaluation as an open resource; see “Review Criteria“ in https://iswc2020.semanticweb.org/calls/call-for-resources-track-papers/ for useful tips on how this should be done

Possible extensions

Implement support for using the developed metrics in open-source OTTR software Lutra
Description and discussion of the type of updates are performed on the ontologies, and if any advances to tools and methods could cater for these.
In light of the developed metrics, discuss what an “optimal” design of an OTTR template library could look like for different scenarios:
- easy to update
- easy to use
- easy to manage
- other?

Suggested plan

Benchmark

Define scope for “pattern-based ontology engineering approaches” and identify all approaches and tools that fall into the scope. Decide on additional selection criteria (e.g., the tool must be available).
Identify 2-4 different ontologies that will make out the benchmark. Suggested requirements for selecting the ontologies:
- openly available with a suitable licence
- preferably in different versions
- some developed using some pattern-based technique or tool
Suggestions: the pizza ontology, the plant ontology, [some EU IOT ontology], schema.org? Need to check if the above conditions hold for these.
Optional(*): Identify different versions of each of the ontologies, together with important updates (later (or earlier) versions) of the ontologies. Find and describe/represent the diff of each update.

Metrics

Possible metrics:

Size of the pattern representation (e.g., how many templates and instances does it take to represent the ontology; counted and measured in characters)
Run-time to produce the ontologies
Optional(*): Number of changes, and what types of changes (this must be defined), required to implement the updates between versions of the ontologies.

Evaluation

Represent the ontologies (and their versions) in the benchmark using the different approaches, e.g., in the case of OTTR: develop a template library that represents the patterns used by each of the ontologies, and make instances that expand to exactly the ontology.
Test metrics.

Veileder(e)

Martin Georg Skjæveland Universitetet i Oslo