OutOfMemoryError while executing SPARQL Query

[naturzukunft]

Hi there, i just playing a bit with sparql on my server (rdf4j embedded in a spring-boot app). If i start using subqueries, i get a

Java heap space. Original exception type: java.lang.OutOfMemoryError. 
	at org.eclipse.rdf4j.repository.sail.SailTupleQuery.evaluate(SailTupleQuery.java:58)
	at org.eclipse.rdf4j.federated.structures.FedXTupleQuery.evaluate(FedXTupleQuery.java:63)

The sample query.

Maybe the query is bad! and maybe the server is configured with less memory. What would be the recommended memory to give to the app ?

Thanks Fredy

[hmottestad]

Can you tell us about your data and your server?

[hmottestad]

How big is your dataset?

[naturzukunft]

currently i didn't have a possibility to check that without SPARQL.
If i send this query it's working with a limit ~ 15.500 and breaks between 15.500 and 16.000.

[naturzukunft]

ls -lsa rdf4jRepos/repositories/rdfPubRepository/
total 3213
   9 drwxr-xr-x 3 _apt systemd-journal      18 Apr 30 11:25 .
   9 drwxr-xr-x 6 _apt systemd-journal       6 Apr 20 17:49 ..
   5 -rw-r--r-- 1 _apt systemd-journal     578 Apr 20 17:49 config.ttl
   5 -rw-r--r-- 1 _apt systemd-journal     319 May  1 17:13 contexts.dat
   1 drwxr-xr-x 2 _apt systemd-journal       4 Apr 21 23:29 lock
   1 -rw-r--r-- 1 _apt systemd-journal       4 Apr 20 17:49 namespaces.dat
   1 -rw-r--r-- 1 _apt systemd-journal       5 Apr 20 17:49 nativerdf.ver
   1 -rw-r--r-- 1 _apt systemd-journal     453 May  1 17:13 triples-posc.alloc
1353 -rw-r--r-- 1 _apt systemd-journal 7317501 May  1 17:13 triples-posc.dat
   5 -rw-r--r-- 1 _apt systemd-journal     106 Apr 20 17:49 triples.prop
   1 -rw-r--r-- 1 _apt systemd-journal     429 May  1 17:13 triples-spoc.alloc
1269 -rw-r--r-- 1 _apt systemd-journal 6897661 May  1 17:13 triples-spoc.dat
   1 -rw-r--r-- 1 _apt systemd-journal       0 Apr 20 17:49 txncache.alloc
   1 -rw-r--r-- 1 _apt systemd-journal      16 May  1 17:13 txncache.dat
   1 -rw-r--r-- 1 _apt systemd-journal       0 May  1 17:13 txn-status
 349 -rw-r--r-- 1 _apt systemd-journal 1001068 May  1 17:13 values.dat
 149 -rw-r--r-- 1 _apt systemd-journal  280312 May  1 17:13 values.hash
  61 -rw-r--r-- 1 _apt systemd-journal  109752 May  1 17:13 values.id

[abrokenjester]

I'm noticing this uses a FedX repository - the problem might be specific to how it handle caching during evaluation of federated joins. @aschwarte10 you have any thoughts?

[naturzukunft]

Indeed i decided to separate the data and the type rdf-definitions see OneForAllApRepositoryManager
Maybe this was a bad idea ?!
The idea was that since it is a generic store, admin users would be able to import/manage rdf types. and that this management would happen independently of the application data.
The federation is created especially for the SPARQL endpoint, so that rdf-type queries can also be made via SPQRQL.

[naturzukunft]

Currently i ended up with this query it's maybe a bit crazy to get all entries arround the world. But that is currently the wish of the wordPress Plugin-Developer. It would also probably be possible to work with smaller map cut-outs. As soon i asked for lat & lon, i get a
QueryEvaluationException: @ rdfPubRepository - Query evaluation took too long.

unfortunately, i can see now that with my limited experience in triple stores and SPARQL, i'm quickly reaching the limits. and i don't know yet if it's the database design or the query.

[hmottestad]

It would be nice to rule out the NativeStore since our assumption is that it's related to the federated store. Could you replicate your setup with just a single NativeStore and check if your queries succeed?

[naturzukunft]

I'll see, if this is easily possible.
In the meanwhile we fixed (for now) the outOfMemory. (by increasing the memory of the vm ->768M)

Since a few minutes i am able to reproduce errors locally. now i get locally :

org.eclipse.rdf4j.query.QueryInterruptedException: Query evaluation took too long
	at org.eclipse.rdf4j.query.impl.AbstractParserQuery$QueryInterruptIteration.throwInterruptedException(AbstractParserQuery.java:75) ~[rdf4j-queryparser-api-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.common.iteration.TimeLimitIteration.checkInterrupted(TimeLimitIteration.java:102) ~[rdf4j-util-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.common.iteration.TimeLimitIteration.hasNext(TimeLimitIteration.java:49) ~[rdf4j-util-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.common.iteration.ConvertingIteration.hasNext(ConvertingIteration.java:62) ~[rdf4j-util-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.common.iteration.IterationWrapper.hasNext(IterationWrapper.java:63) ~[rdf4j-util-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.federated.evaluation.iterator.GraphToBindingSetConversionIteration.hasNext(GraphToBindingSetConversionIteration.java:36) ~[rdf4j-tools-federation-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.federated.evaluation.iterator.CloseDependentConnectionIteration.hasNext(CloseDependentConnectionIteration.java:42) ~[rdf4j-tools-federation-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.federated.evaluation.iterator.StopRemainingExecutionsOnCloseIteration.hasNext(StopRemainingExecutionsOnCloseIteration.java:45) ~[rdf4j-tools-federation-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.common.iteration.IterationWrapper.hasNext(IterationWrapper.java:63) ~[rdf4j-util-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.sail.helpers.SailBaseIteration.hasNext(SailBaseIteration.java:41) ~[rdf4j-sail-api-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.common.iteration.IterationWrapper.hasNext(IterationWrapper.java:63) ~[rdf4j-util-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.common.iteration.FilterIteration.findNextElement(FilterIteration.java:74) ~[rdf4j-util-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.common.iteration.FilterIteration.hasNext(FilterIteration.java:45) ~[rdf4j-util-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.common.iteration.ConvertingIteration.hasNext(ConvertingIteration.java:62) ~[rdf4j-util-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.common.iteration.IterationWrapper.hasNext(IterationWrapper.java:63) ~[rdf4j-util-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.query.QueryResults.report(QueryResults.java:296) ~[rdf4j-query-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.repository.sail.SailGraphQuery.evaluate(SailGraphQuery.java:136) ~[rdf4j-repository-sail-3.7.4.jar:3.7.4]
	at org.eclipse.rdf4j.federated.structures.FedXGraphQuery.evaluate(FedXGraphQuery.java:50) ~[rdf4j-tools-federation-3.7.4.jar:3.7.4]
	at de.naturzukunft.rdf4j.utils.sparql.QueryTypes$1.evaluate(QueryTypes.java:36) ~[classes/:na]
	at de.naturzukunft.rdf4j.utils.sparql.SparqlQueryEvaluatorRdf4j.evaluate(SparqlQueryEvaluatorRdf4j.java:33) ~[classes/:na]
	at org.linkedopenactors.rdfpub.repository.AbstractApRepository.executeSparql(AbstractApRepository.java:67) ~[classes/:na]
	at org.linkedopenactors.rdfpub.usecases.querycollection.ContentEnricherSparql.process(ContentEnricherSparql.java:63) ~[classes/:na]

after an import process has been running for a while.
I will continue debugging and report back.

[hmottestad]

That looks like you are passing in a timeout param on your query.

[naturzukunft]

I was searching, debugging, changing a lot in the last days. The problem was probably simply that the server had not been given enough memory. I could not find any memory problem. Thanks for the help

In the meanwhile we fixed (for now) the outOfMemory. (by increasing the memory of the vm ->768M)