Upgrade Solr to 9.4.1

[landreev]

What this PR does / why we need it:

There's a known security issue in the previously recommended version 9.3.0.
While the risk of a successful exploit is not significant unless the Solr instance is accessible from the outside (which we have always recommended against; with 9.3.0 running on localhost only out of the box), we have decided to upgrade the recommended version to 9.4.1 just in case.

Which issue(s) this PR closes:

Closes #

(there's a security issue where this was discussed; there's no associated issue in the main project)

Special notes for your reviewer:

Note that in addition to the upgrade of the recommended version we are using the opportunity to get rid of the specific Solr version in our source tree in the directory used to distribute the configuration files. Meaning, the config files that used to be under conf/solr/9.3.0 are now directly under conf/solr. This will make it easier to trace the history of updates in these files.

Suggestions on how to test this:

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

[qqmyers]

Looks good to me. Thanks for moving the solr files out of a version specific subdir!

[qqmyers]

I do see the Jenkins build failing with TASK [dataverse : branch: place solrconfig] ************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Source /opt/dataverse/dataverse/conf/solr/9.3.0/solrconfig.xml not found"} - looks like we need an update there for the dir change?

The Shellspec tests are failing as well - not sure there since the file didn't change and I don't see any 9.3.0 references in it. Is this just because Shellspec doesn't usually run on that file and only trigger because it is now in a new place?

[landreev]

"Source /opt/dataverse/dataverse/conf/solr/9.3.0/solrconfig.xml not found"

OK, I was wrong, in my optimistic assumption that this wasn't going to affect Jenkins - from a very brief look at dataverse-ansible, I somehow got the idea that it would build the installer bundle and get the config files from it... but, obviously, that's not what happens.

[qqmyers]

FWIW - the Shellspec failures are due to 9.3.0 in the path in

dataverse/tests/shell/spec/update_fields_spec.sh

Lines 4 to 13 in 853965e

	../../conf/solr/9.3.0/update-fields.sh "$@"
	}
	Describe "Update fields command"
	Describe "can operate on upstream data"
	copyUpstreamSchema() { cp ../../conf/solr/9.3.0/schema.xml data/solr/upstream-schema.xml; }
	AfterAll 'copyUpstreamSchema'
	Path schema-xml="../../conf/solr/9.3.0/schema.xml"

- saw it while updating to 9.6.1 at QDR.

Speaking of which - any interest in just going to solr/solrj 9.6.1 here? I can make a separate PR that can be used later if not.

[qqmyers]

FWIW: "9.3.0" also appears in a few more places - parent pom and docker-related files - that should probably be updated.

[landreev]

Speaking of which - any interest in just going to solr/solrj 9.6.1 here? I can make a separate PR that can be used later if not.

I asked early on about going with 9.6.1 for the solr server upgrade as well, in the sec. issue:

The current latest stable version is 9.6.1. Are there any reasons NOT to go with that, instead of 9.4.1, as specified above?

And I just assumed it would be safest to keep solrj version matched to that. I eventually decided to

Unless somebody has good reasons to stop me, I'm planning to go with 9.4.1 for the Harvard prod. solr upgrade this Friday, and to use that version in the changes in the main source tree. Just to be conservative/cautious; and to avoid changing the solrj dependency.

If you think it's worth it, we can still bump it. But 9.4.1 by now has the advantage of having been tested quite thoroughly with the optimization PRs, both in solrj and solr itself. I didn't immediately see anything super important in 9.6.1 - is there?

[coveralls]

coverage: 20.664% (+0.005%) from 20.659%
when pulling 6496054 on 92sec-upgrade-solr-9.4.1
into d113d7c on develop.

[landreev]

FWIW - the Shellspec failures are due to 9.3.0 in the path in

Thanks. I can't immediately figure out why it's still failing after I fixed the path though. https://github.com/IQSS/dataverse/actions/runs/9598126298/job/26468684721?pr=10636

[qqmyers]

w.r.t. 9.6.1 - I don't think we need to update now. After this is merged, I might make a PR to capture the (minor) solrconfig.xml changes I made at QDR, but that could sit/be further updated before we decide to update again.

[qqmyers]

re: shellcheck - might be this issue: koalaman/shellcheck#2700 - not sure if that's related to the review dog warnings now showing in the code.

[coveralls]

coverage: 20.661% (+0.002%) from 20.659%
when pulling 241a91f on 92sec-upgrade-solr-9.4.1
into d113d7c on develop.

[coveralls]

coverage: 20.661% (+0.002%) from 20.659%
when pulling a37bddd on 92sec-upgrade-solr-9.4.1
into d113d7c on develop.

[coveralls]

coverage: 20.661% (+0.002%) from 20.659%
when pulling b7b750a on 92sec-upgrade-solr-9.4.1
into d113d7c on develop.

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:92sec-upgrade-solr-9.4.1

ghcr.io/gdcc/configbaker:92sec-upgrade-solr-9.4.1

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.