Broken OAI records for some datasets with UTF-8 characters

[landreev]

(Edit: everything entered in this issue so far was a red herring; I'm deleting everything and rewriting the issue from scratch)

The issue was opened as a followup to an RT ticket from an outside institutions. They were unable to harvest the full OAI set from IQSS, because it was reliably bombing on the same invalid record.

The problem was traced to something inside the OAI library, it is being tracked in gdcc/xoai#188 in the xoai repo. I have a rough idea of how to produce a quick fix.

[cmbz]

2023/09/13: Added to the Global Backlog, with a very strong proposal for inclusion in release 6.1.

[cmbz]

2023/09/18

• Moving to sprint ready because work is already underway

[cmbz]

2023/09/25: Applied the NIH GREI 2.4.1B tag to reflect value to harvesting.

[landreev]

Much of the discussion related to this bug happened on slack between @poikilotherm, @jggautier, @scolapasta and I, so just to add a quick summary here:

• Julian identified the records that were broken, and I put together a quick script that went and padded the metadata exports with the right number of spaces; the total number of affected records was 50-something out of almost 80K published records. This is, obviously, not a fix, but a bandaid (somebody may have already published another dataset with another similarly broken record produced as a result... :( ).
• I confirmed that the quick xoai patch that I produced (commit in the linked issue) works. So I have a patched xoai jar that can be deployed with 6.0 (I'm planning to do that in the next few days).
• Oliver had a cleaner/proper solution in mind.
• Once a proper fix is made on the xoai side and the updated version is released, I'll make a PR here for a pom file update (trivial amount of work).
• Dataverse harvesters process one record at a time (via ListIdentifiers -> GetRecord), so it only fails for the specific records affected by the bug, when Dataverses harvest from each other. But it is a more serious problem for non-Dataverse harvesters that rely on ListRecords, because the entire harvest fails once it reaches the page that contains an affected record.

[poikilotherm]

Just to make sure this doesn't get lost: there is an upstream lib PR waiting for review gdcc/xoai#192

[landreev]

Thank you @poikilotherm, much appreciated! I was a little busy putting out some fires, but looking at the xoai pr now.

[poikilotherm]

XOAI 5.2.0 is on it's way to Central, might take a moment until it's retrievable.

[landreev]

OK, we'll move this along and update the Dataverse pom file. And then we'll just need to test/QA the result.

[landreev]

Quick question: I've been assuming that the xoai jars v. 5.1+ are no longer compatible with pre-6.0 versions of Dataverse, is that actually correct?
There is this sentence in the 5.1 README:

Switching to Java 17 for compilation and testing, but keeping compatibility with Java 11 for JARs

[poikilotherm]

Should be compatible.

[landreev]

I'm going to make a quick PR incorporating xoai-5.2 into develop.
And I will also try building a patched 5.14, since there appears to be interest in that in the community.

[landreev]

@DS-INRA (and everybody who may be interested) The linked PR #10012 has been merged, so the bug is now fixed for real in the develop branch.
I have also tested these newly released xoai-5.2 jars with Dataverse 5.14, and confirmed that they appear to be working, fixing this bug there as well.

To clarify, I have tested patching these jars in place, on an instance where the standard 5.14 release was deployed. I haven't tried building dataverse-5.14.war from sources with these libraries, but can't think of a reason why that wouldn't work either. If there is interest, I can produce a patched dataverse-5.14.war as well.

The library jars in question can be found on maven central - at
https://repo.maven.apache.org/maven2/io/gdcc/xoai-common/5.2.0/xoai-common-5.2.0.jar
https://repo.maven.apache.org/maven2/io/gdcc/xoai-data-provider/5.2.0/xoai-data-provider-5.2.0.jar
https://repo.maven.apache.org/maven2/io/gdcc/xoai-service-provider/5.2.0/xoai-service-provider-5.2.0.jar
https://repo.maven.apache.org/maven2/io/gdcc/xoai-xmlio/5.2.0/xoai-xmlio-5.2.0.jar