LatestVersionSubset retention does not work as expected

[rorlic]

When configuring a view with retention policy LatestVersionSubset the LDES Server should ensure that only the N last versions of a state objects are available in the view. The same is true for the event source retention: all members except for the M last ones should be deleted from the database.

I use the occupancy of the parking lots of the city of Ghent as a data sources (see broker tutorial) which produces 5 new members at regular intervals (every 5 minutes).

I have configured a LDES with the event source keeping the last 3 versions:

@prefix dcterms:     <http://purl.org/dc/terms/> .
@prefix xsd:         <http://www.w3.org/2001/XMLSchema#> .
@prefix sh:          <http://www.w3.org/ns/shacl#> .
@prefix ldes:        <https://w3id.org/ldes#> .
@prefix tree:        <https://w3id.org/tree#> .
@prefix mv:          <http://schema.mobivoc.org/#> .

</occupancy> a ldes:EventStream ;
	tree:shape [ a sh:NodeShape ] ;
	ldes:timestampPath dcterms:modified ;
	ldes:versionOfPath dcterms:isVersionOf ;
	ldes:eventSource [
		a ldes:EventSource ;
		ldes:retentionPolicy [
			a ldes:LatestVersionSubset ;
			ldes:amount 3 ;
		]
	] .

and I define the view to contain the last 2 versions:

@prefix tree:         <https://w3id.org/tree#>.
@prefix xsd:          <http://www.w3.org/2001/XMLSchema#> .
@prefix ldes:         <https://w3id.org/ldes#> .
@prefix geosparql:	  <http://www.opengis.net/ont/geosparql#> .
@prefix terms:        <http://purl.org/dc/terms/> .

</occupancy/by-parking> a tree:Node ; 
  tree:viewDescription [ 
		a tree:ViewDescription ;
    tree:fragmentationStrategy ([
			a tree:ReferenceFragmentation ;
			tree:fragmentationPath terms:isVersionOf ; 
			tree:fragmentationKey "parking-lot"
		]) ;
		tree:pageSize "15"^^xsd:integer ;
		ldes:retentionPolicy [
			a ldes:LatestVersionSubset ;
			ldes:amount 2 ;
		]
	] .

After a while I see that

• the view correctly contains the last 2 versions:

SELECT p.page_id, p.partial_url, count(*)
FROM pages p
join page_members pm on p.page_id = pm.page_id
WHERE immutable = 'f'
group by p.page_id
ORDER BY page_id

results in:

page_id,partial_url,count
17,/occupancy/by-parking?parking-lot=https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-wondelgem-industrieweg&pageNumber=1,2
18,/occupancy/by-parking?parking-lot=https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-loopexpo&pageNumber=1,2
19,/occupancy/by-parking?parking-lot=https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-oostakker&pageNumber=1,2
20,/occupancy/by-parking?parking-lot=https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-bourgoyen&pageNumber=1,2
21,/occupancy/by-parking?parking-lot=https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-gentbrugge-arsenaal&pageNumber=1,2

• the view contains the correct versions:

SELECT member_id, page_id, bucket_id, view_id FROM page_members ORDER BY member_id

results:

member_id,page_id,bucket_id,view_id
41,17,4,1
42,21,16,1
43,20,10,1
44,19,7,1
45,18,13,1
46,17,4,1
47,21,16,1
48,19,7,1
49,20,10,1
50,18,13,1

• the members are not correctly deleted:

SELECT member_id, timestamp, subject FROM members ORDER BY member_id

results:

member_id,timestamp,subject
1,2024-12-23 14:22:51,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-wondelgem-industrieweg#2024-12-23T15:22:51+01:00
2,2024-12-23 14:24:42,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-oostakker#2024-12-23T15:24:42+01:00
3,2024-12-23 14:24:42,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-bourgoyen#2024-12-23T15:24:42+01:00
4,2024-12-23 14:24:42,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-loopexpo#2024-12-23T15:24:42+01:00
5,2024-12-23 14:24:42,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-gentbrugge-arsenaal#2024-12-23T15:24:42+01:00
6,2024-12-23 14:28:23,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-wondelgem-industrieweg#2024-12-23T15:28:23+01:00
7,2024-12-23 14:29:45,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-oostakker#2024-12-23T15:29:45+01:00
8,2024-12-23 14:29:45,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-gentbrugge-arsenaal#2024-12-23T15:29:45+01:00
9,2024-12-23 14:29:45,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-loopexpo#2024-12-23T15:29:45+01:00
10,2024-12-23 14:29:45,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-bourgoyen#2024-12-23T15:29:45+01:00
11,2024-12-23 14:34:09,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-wondelgem-industrieweg#2024-12-23T15:34:09+01:00
12,2024-12-23 14:34:51,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-bourgoyen#2024-12-23T15:34:51+01:00
13,2024-12-23 14:34:51,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-loopexpo#2024-12-23T15:34:51+01:00
14,2024-12-23 14:34:51,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-oostakker#2024-12-23T15:34:51+01:00
15,2024-12-23 14:34:51,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-gentbrugge-arsenaal#2024-12-23T15:34:51+01:00
41,2024-12-23 15:03:57,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-wondelgem-industrieweg#2024-12-23T16:03:57+01:00
42,2024-12-23 15:04:08,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-gentbrugge-arsenaal#2024-12-23T16:04:08+01:00
43,2024-12-23 15:04:08,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-bourgoyen#2024-12-23T16:04:08+01:00
44,2024-12-23 15:04:08,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-oostakker#2024-12-23T16:04:08+01:00
45,2024-12-23 15:04:08,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-loopexpo#2024-12-23T16:04:08+01:00
46,2024-12-23 15:09:54,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-wondelgem-industrieweg#2024-12-23T16:09:54+01:00
47,2024-12-23 15:10:12,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-gentbrugge-arsenaal#2024-12-23T16:10:12+01:00
48,2024-12-23 15:10:12,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-oostakker#2024-12-23T16:10:12+01:00
49,2024-12-23 15:10:12,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-bourgoyen#2024-12-23T16:10:12+01:00
50,2024-12-23 15:10:12,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-loopexpo#2024-12-23T16:10:12+01:00
51,2024-12-23 15:13:39,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-wondelgem-industrieweg#2024-12-23T16:13:39+01:00
52,2024-12-23 15:14:14,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-oostakker#2024-12-23T16:14:14+01:00
53,2024-12-23 15:14:14,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-loopexpo#2024-12-23T16:14:14+01:00
54,2024-12-23 15:14:14,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-bourgoyen#2024-12-23T16:14:14+01:00
55,2024-12-23 15:14:14,https://stad.gent/nl/mobiliteit-openbare-werken/parkeren/park-and-ride-pr/pr-gentbrugge-arsenaal#2024-12-23T16:14:14+01:00

The expected result is that the members table (event source) contains the latest 15 members (5 parking lots x 3 versions) and that the view contains the latest 10 members (5 parking lots x 2 versions).

It appears that the LDES Server incorrectly keeps the first 3 versions of each parking lot and in addition also the 2 last versions because the view still relies on them?

Anyway, I would expect only the last 15 members to physically be in the members table and the 10 last members to be associated in the page_members table.