Is there something better than Blazegraph for Wikidata?

The main contender: Virtuoso

Virtuoso is a SPARQL implementation developed for more than a decade by a small company, OpenLink Software. It is at its core a SQL database targeting OLAP workloads with a layer on top converting SPARQL to SQL. It seems to provide great performances, powering very large endpoints like Uniprot. However, according to WDQS Backend Alternative work, Virtuoso is also tuned for bulk-load with high-frequency read, and not for read/write. But, this is also the case with Blazegraph. So, it might be interesting to do a good benchmark to see if it actually outperforms Blazegraph or not.

The read-only systems: QLever, HDT, QEndpoint...

This section has been rewritten after a great feedback from Pavel Klinov. Thank you so much!

To my knowledge, all of these systems rely on derived techniques introduced by RDF-3X. They encode the RDF terms using consecutive integers and build fast and compact indexes using this property. It seems possible to make these systems read-write by implementing techniques inspired by LSM trees. The basic idea of LSM trees is to build immutable partial indexes. When updates are done, the systems batch changes in memory, and, after a given threshold, create a new partial index. To speed up reads, the implementations often run background tasks to create bigger indexes by merging the smaller ones together. But it seems to be a significant implementation work. So, it is likely that these systems will not be suitable for Wikidata in the short term.

The toolkits: Apache Jena and Eclipse RDF4J

Jena and RDF4J are two well-maintained, mature, and full of features RDF toolkits written in Java. They both provide native storage systems with SPARQL. However, they seem to be mostly focused on light or medium workloads. Their latest storage systems, TDB 2 for Jena and LMDB for RDF4J both allow only a single writer thread and use a copy-on-write mechanism. So, we end up with the same problem as Virtuoso. To my knowledge, they seem to not be widely used for large datasets, opposite to Virtuoso. I would be very curious to see good benchmarks on how well they behave, especially now that RDF4J seems to have made some performance efforts in the past year.

The work-in-progress system: Oxigraph

Oxigraph is a work-in-progress SPARQL system aiming at write-heavy workloads. It is based on the RocksDB key-value store that targets such workloads. It is developed by myself as a hobby project, at a very slow speed. The SPARQL implementation is complete and basic OLTP operations seem fairly competitive. However, before being able to maybe work at Wikidata-scale it still requires at least two significant improvements. The initial loader is quite slow. Loading Wikidata takes more than a week currently. Also, Oxigraph does not contain yet a proper query planner and optimizer, leading to abysmal performances for complex analytical queries. So, Oxigraph is unlikely to be able to solve Wikidata problems very soon, even when omitting the "bus factor 1" problem.

The dead experiment: History Query Service

The History Query Service is a tool I developed to run big analytical queries on Wikidata history. To this aim, I built a custom storage tailored for its need using RocksDB and used RDF4J on top of it for the SPARQL evaluation. Sadly it suffered from strong shortcomings: The process to build the index took a week on 2018 full Wikidata history dumps and I never managed to make it work again on more recent dumps. Also, it never supported updates. The query evaluation itself was based on the current state of RDF4J at this time, leading to fair but not great query plans and quite slow execution time.

However, I believe it has shown some interesting ideas. Building custom storage for Wikidata history is possible and does not lead to huge space expansion compared to the indexing of only the latest version while allowing fairly efficient querying. I hope at one point to be able to build an optimized version 2 using Oxigraph. But, to be worth it, it requires Oxigraph to have a good and fast SPARQL query planner and evaluator. So, it is not for tomorrow.