Building Block View

Overall System

The Query Service is made up of a few fairly distinct components.

Building Block	Responsibility
User Interfaces	Allow user interaction with the Query Service
Backend	Index data and serve SPARQL queries
Updating	Keep the Backend up to date with changes from a Wikibase Repository

User Interfaces

Building Block	Responsibility
Query Frontend	Provide users with an interface to create entirly custom SPARQL queries and visualize results
Query Builder	Provider users with a simple visual interface for generating lists using the query service

Backend

Building Block	Responsibility
Query Proxy	Force setting of timeouts, access-control and basic caching
Blazegraph	Graph database with a SPARQL API
Data Loader	Loads data into Blazegraph from an RDF rump

Data Loader

The Data Loader currently currently exists in 2 different flavours, which are used in difference deployments.
Only 1 of the choices for munging / RDF conversation is needed per deployment.

Building Block	Responsibility
loadData	Load RDF into Blazegraph
munger	Convert Wikibase Repository RDF triples into different triples for Blazegraph
TurtleDumpConverter	Convert Wikibase Repository RDF triples into different triples for Blazegraph in a Hadoop cluster

Updating

The Updating system curently exists in 3 different flavours, which are used in different deployments.
Only 1 of these update paths is needed in each deployment.

Building Block	Responsibility
Streaming Updater	Updating Blazegraph using a stream of RDF change events
KafkaPoller	Updating Blazegraph using a stream of events created by EventBus from Kafka
RCPoller	Updating Blazegraph by polling Recent Changes for new changes

Streaming Updater

Building Block	Responsibility
Consumer	Consumes a stream of RDF changes and writes them to Blazegraph
Producer	Consumes a stream of events from MediaWiki (Revision creations, Page deletions / undeletions), and emits a stream of RDF changes