Support LDES distributions for dataset analysis

[ddeboer]

Problem

The NDE Dataset Register currently holds 16 LDES distributions (dct:conformsTo <https://w3id.org/ldes/specification>), all published by the Rijksmuseum. When the Dataset Knowledge Graph (DKG) analyses these, it fetches the LDES collection head (.../collection.json) as a single document and summarizes only the handful of triples it contains – the ldes:EventStream / tree view descriptor – instead of the actual members.

Concrete example – dataset https://id.rijksmuseum.nl/260210 (“Against Opacity”):

• DKG’s VoID summary reports 2 distinct subjects, 10 properties.
• The dataset actually holds 21,189 objects.

DKG sees ~0.01% of the data because it does not follow the LDES/TREE pagination.

Why LDES is the right harvest target

Each registered dataset typically offers three distribution flavours; only LDES is suitable for reconstructing the full RDF content:

API (dct:conformsTo)	Members inline?	Cost to get full content
LDES w3id.org/ldes/specification	Yes – leaf pages embed each member’s full RDF in @graph	One traversal of the view
linked.art search linked.art/api/1.0/search/	No – pages list {id, type} stubs only	Page all stubs, then dereference each object (N extra requests)
IIIF Change Discovery iiif.io/api/discovery/1.0/	No – it is an ActivityStreams change feed pointing at IIIF manifests	Page the feed, then fetch each manifest

The LDES leaf pages carry members like:

{ "@type": "Create",                         // ActivityStreams version wrapper
  "object": { "canonical": ".../200110641" },
  "published": "2026-01-26T23:37:54Z",       // timestampPath: as:published
  "@graph": { /* full linked.art description of the object */ } }

The view declares ldes:LatestVersionSubset (amount 1), so a traversal of this view yields the current state per entity (versionOfPath: as:object) – exactly the snapshot DKG needs.

Scale (across all 16 LDES datasets)

Set	Objects
260251 “Alle gepubliceerde objecten” (superset = whole museum)	837,962
260239 “Entire Public Domain Set”	735,593
260241 “biografisch”	82,664
260210 “Against Opacity”	21,189
260250 “prenten”	12,756
260211 “Indisch erfgoed”	11,827
10 further thematic sets	50 – 3,310 each

The naive sum is 1,716,901, but the sets overlap heavily – 260251 is the whole museum and 260239 is its public-domain subset, so the distinct universe is ≈ 838k objects. Harvesting all 16 independently would re-fetch shared objects many times over.

Proposed approach (for discussion)

Add an LDES/TREE reader in LDE – most naturally a new package (or an extension of distribution-downloader) that:

1. Walks the TREE view (year → month → day partitions here), following tree:relation nodes.
2. Extracts each member’s @graph, folding versions to current state via LatestVersionSubset (honouring Delete).
3. Materializes a current-state RDF snapshot, which the existing sparql-importer / sparql-qlever chain indexes and DKG analyses like any other dataset.

This reuses the entire existing analysis pipeline (VoID, SHACL, media, validity) unchanged – an LDES simply becomes another way to obtain the snapshot RDF.

Design question: full reconstitution vs. incremental stream analysis

Recommendation: full reconstitution → QLever → analyse, with incrementality at the replication layer, not the analysis layer.

• DKG’s core outputs are VoID aggregates (distinct subjects/objects/IRIs, class & property partitions, datatype partitions). These are global set-cardinality metrics that require the full current-state graph in a queryable store – which QLever already is. Reusing the standard pipeline is simple and correct.
• Truly incremental analysis (updating VoID counts per Create/Update/Delete event) means maintaining stateful materialized aggregates with retraction – an Update/Delete must retract the superseded version’s contributions to every partition. That is complex and correctness-fragile, and it does not remove the need to hold current state (i.e. you still need something QLever-shaped).
• The real cost is the network harvest of the two ~800k-object streams, not the QLever index build. LDES is designed for cheap incremental sync: replicate once, then fetch only pages newer than the last run (via the as:published timestamp relations). So capture the win where it exists – persist the local replica and sync deltas into it, then re-index the snapshot. This pairs with DKG’s existing “skip unchanged” behaviour.

So: build the reader as a snapshot source first; add incremental replication as a second phase; only revisit incremental analysis if snapshot re-indexing is ever proven to be the bottleneck.

Caveats / things to decide

• Cross-set overlap / dedup. All 16 sets are Rijksmuseum and heavily overlapping (838k superset + subsets). The reader should ideally share one replica / dedupe across sets rather than harvest each independently (~1.7M fetches for ~838k distinct objects).
• Content is linked.art (CIDOC-CRM), not schema.org. Members are HumanMadeObject with produced_by / classified_as / identified_by etc. – zero schema.org. VoID summaries will therefore report CIDOC-CRM classes/properties, and the SCHEMA-AP-NDE conformance metric will read non-conformant. That is a modelling choice by the publisher, orthogonal to harvesting – but we should decide whether SCHEMA-AP conformance should even apply to object-level records published in a non-schema.org profile.
• Pagination granularity. This LDES partitions down to day-level leaf pages, so a full harvest is many small requests – bounded and cacheable, but worth respecting rate limits.