Merge branch 'develop' into 10531-contrib IQSS#10531

Conflicts:
doc/sphinx-guides/source/developers/documentation.rst

conf/solr/9.3.0/schema.xml → conf/solr/schema.xml

@@ -326,6 +326,7 @@
     <field name="journalVolumeIssue" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="keyword" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="keywordValue" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="keywordTermURI" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="keywordVocabulary" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="keywordVocabularyURI" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="kindOfData" type="text_en" multiValued="true" stored="true" indexed="true"/>
@@ -565,6 +566,7 @@
     <copyField source="journalVolumeIssue" dest="_text_" maxChars="3000"/>
     <copyField source="keyword" dest="_text_" maxChars="3000"/>
     <copyField source="keywordValue" dest="_text_" maxChars="3000"/>
+    <copyField source="keywordTermURI" dest="_text_" maxChars="3000"/>
     <copyField source="keywordVocabulary" dest="_text_" maxChars="3000"/>
     <copyField source="keywordVocabularyURI" dest="_text_" maxChars="3000"/>
     <copyField source="kindOfData" dest="_text_" maxChars="3000"/>

doc/release-notes/10137-2-add-disable-reason-flag.md

@@ -0,0 +1,6 @@
+## Release Highlights
+
+### Feature flag to remove the required "reason" field in the "Return To Author" dialog 
+
+A reason field, that is required to not be empty, was added in v6.2. Installations that handle author communications through email or another system may prefer to not be required to use this new field. v6.2 includes a new 
+disable-return-to-author-reason feature flag that can be enabled to drop the reason field from the dialog and make sending a reason optional in the api/datasets/{id}/returnToAuthor call.  

doc/release-notes/10288-add-term_uri-metadata-in-keyword-block.md

@@ -0,0 +1,53 @@
+### New keywordTermURI Metadata in keyword Metadata Block
+
+Adding a new metadata `keywordTermURI` to the `keyword` metadata block to facilitate the integration of controlled vocabulary services, in particular by adding the possibility of saving the "term" and its associated URI. For more information, see #10288 and PR #10371.
+
+## Upgrade Instructions
+
+1\. Update the Citation metadata block
+
+- `wget https://github.com/IQSS/dataverse/releases/download/v6.3/citation.tsv`
+- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"`
+
+2\. Update your Solr `schema.xml` to include the new field.
+
+   For details, please see https://guides.dataverse.org/en/latest/admin/metadatacustomization.html#updating-the-solr-schema
+
+
+3\. Reindex Solr.
+
+   Once the schema.xml is updated, Solr must be restarted and a reindex initiated.
+   For details, see https://guides.dataverse.org/en/latest/admin/solr-search-index.html but here is the reindex command:
+
+   `curl http://localhost:8080/api/admin/index`
+
+
+4\. Run ReExportAll to update dataset metadata exports. Follow the instructions in the [Metadata Export of Admin Guide](https://guides.dataverse.org/en/latest/admin/metadataexport.html#batch-exports-through-the-api).
+
+
+## Notes for Dataverse Installation Administrators
+
+### Data migration to the new `keywordTermURI` field
+
+You can migrate your `keywordValue` data containing URIs to the new `keywordTermURI` field.
+In case of data migration, view the affected data with the following database query:
+
+```
+SELECT value FROM datasetfieldvalue dfv
+INNER JOIN datasetfield df ON df.id = dfv.datasetfield_id 
+WHERE df.datasetfieldtype_id = (SELECT id FROM datasetfieldtype WHERE name = 'keywordValue')
+AND value ILIKE 'http%';
+```
+
+If you wish to migrate your data, a database update is then necessary:
+
+```
+UPDATE datasetfield df
+SET datasetfieldtype_id  = (SELECT id FROM datasetfieldtype WHERE name = 'keywordTermURI')
+FROM datasetfieldvalue dfv
+WHERE dfv.datasetfield_id  = df.id 
+AND df.datasetfieldtype_id = (SELECT id FROM datasetfieldtype WHERE name = 'keywordValue')
+AND dfv.value ILIKE 'http%';
+```
+
+A ['Reindex in Place'](https://guides.dataverse.org/en/latest/admin/solr-search-index.html#reindex-in-place) will be required and ReExportAll will need to be run to update the metadata exports of the dataset. Follow the directions in the [Admin Guide](http://guides.dataverse.org/en/latest/admin/metadataexport.html#batch-exports-through-the-api).

doc/release-notes/10494-payara-upgrade.md

@@ -0,0 +1,119 @@
+# Upgrade Payara to v6.2024.6
+
+With this version of Dataverse, we encourage you to upgrade to version 6.2024.6.
+This will address security issues accumulated since the release of 6.2023.8, which was required since Dataverse release 6.0.
+
+## Instructions for Upgrading
+
+If you are using GDCC containers, this upgrade is included when pulling new release images.
+No manual intervention is necessary.
+
+We recommend you ensure you followed all update instructions from the past releases regarding Payara.
+(Latest Payara update was for [v6.0](https://github.com/IQSS/dataverse/releases/tag/v6.0))
+
+Upgrading requires a maintenance window and downtime. Please plan ahead, create backups of your database, etc.
+
+The steps below are a simple matter of reusing your existing domain directory with the new distribution.
+But we also recommend that you review the Payara upgrade instructions as it could be helpful during any troubleshooting:
+[Payara Release Notes](https://docs.payara.fish/community/docs/Release%20Notes/Release%20Notes%206.2024.6.html)
+We assume you are already on a Dataverse 6.x installation, using a Payara 6.x release.
+
+```shell
+export PAYARA=/usr/local/payara6
+```
+
+(or `setenv PAYARA /usr/local/payara6` if you are using a `csh`-like shell)
+
+1\. Undeploy the previous version
+
+```shell
+    $PAYARA/bin/asadmin list-applications
+    $PAYARA/bin/asadmin undeploy dataverse<-version>
+```
+
+2\. Stop Payara
+
+```shell
+    service payara stop
+    rm -rf $PAYARA/glassfish/domains/domain1/generated
+    rm -rf $PAYARA/glassfish/domains/domain1/osgi-cache
+    rm -rf $PAYARA/glassfish/domains/domain1/lib/databases
+```
+
+3\. Move the current Payara directory out of the way
+
+```shell
+    mv $PAYARA $PAYARA.MOVED
+```
+
+4\. Download the new Payara version (6.2024.6), and unzip it in its place
+
+5\. Replace the brand new payara/glassfish/domains/domain1 with your old, preserved domain1
+
+6\. Make sure that you have the following `--add-opens` options in your domain.xml. If not present, add them:
+
+```diff
+--- payara-6.2023.8/glassfish/domains/domain1/config/domain.xml
++++ payara-6.2024.6/glassfish/domains/domain1/config/domain.xml
+@@ -212,12 +212,16 @@
+                 <jvm-options>--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED</jvm-options>
+                 <jvm-options>--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED</jvm-options>
+                 <jvm-options>--add-opens=java.logging/java.util.logging=ALL-UNNAMED</jvm-options>
++                <jvm-options>--add-opens=java.management/javax.management=ALL-UNNAMED</jvm-options>
++                <jvm-options>--add-opens=java.management/javax.management.openmbean=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-exports=java.base/sun.net.www=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-exports=java.base/sun.security.util=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-opens=java.base/java.lang.invoke=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-opens=java.desktop/java.beans=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-exports=jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED</jvm-options>
++                <jvm-options>[17|]--add-opens=java.base/java.io=ALL-UNNAMED</jvm-options>
++                <jvm-options>[21|]--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED</jvm-options>
+                 <jvm-options>-Xmx512m</jvm-options>
+                 <jvm-options>-XX:NewRatio=2</jvm-options>
+                 <jvm-options>-XX:+UnlockDiagnosticVMOptions</jvm-options>
+@@ -447,12 +451,16 @@
+                 <jvm-options>--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED</jvm-options>
+                 <jvm-options>--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED</jvm-options>
+                 <jvm-options>--add-opens=java.logging/java.util.logging=ALL-UNNAMED</jvm-options>
++                <jvm-options>--add-opens=java.management/javax.management=ALL-UNNAMED</jvm-options>
++                <jvm-options>--add-opens=java.management/javax.management.openmbean=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-exports=java.base/sun.net.www=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-exports=java.base/sun.security.util=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-opens=java.base/java.lang.invoke=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-opens=java.desktop/java.beans=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-exports=jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED</jvm-options>
+                 <jvm-options>[17|]--add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED</jvm-options>
++                <jvm-options>[17|]--add-opens=java.base/java.io=ALL-UNNAMED</jvm-options>
++                <jvm-options>[21|]--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED</jvm-options>
+                 <jvm-options>-Xmx512m</jvm-options>
+                 <jvm-options>-XX:NewRatio=2</jvm-options>
+                 <jvm-options>-XX:+UnlockDiagnosticVMOptions</jvm-options>
+```
+(You can also save this as a patch file and try to apply it.)
+
+TODO: For the combined 6.3 release note, I would consider replacing the patch format above with just the 4 specific options, for clarity etc. (L.A.) As in: 
+```
+         <jvm-options>--add-opens=java.management/javax.management=ALL-UNNAMED</jvm-options>
+         <jvm-options>--add-opens=java.management/javax.management.openmbean=ALL-UNNAMED</jvm-options>
+         <jvm-options>[17|]--add-opens=java.base/java.io=ALL-UNNAMED</jvm-options>
+         <jvm-options>[21|]--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED</jvm-options>
+```
+
+7\. Start Payara
+
+```shell
+    service payara start
+```
+
+8\. Deploy this version.
+
+```shell
+    $PAYARA/bin/asadmin deploy dataverse-6.3.war
+```
+
+9\. Restart payara
+
+```shell
+    service payara stop
+    service payara start

doc/release-notes/10503-cvoc-hidden-html-fields.md

@@ -0,0 +1,11 @@
+## Release Highlights
+
+### Updates on Support for External Vocabulary Services
+
+#### Hidden HTML Fields
+
+External Controlled Vocabulary scripts, configured via [:CVocConf](https://guides.dataverse.org/en/6.3/installation/config.html#cvocconf), can now access the values of managed fields as well as the term-uri-field for use in constructing the metadata view for a dataset.
+
+Those values are hidden and can be found with the html attribute `data-cvoc-metadata-name`.
+
+For more information, see [#10503](https://github.com/IQSS/dataverse/pull/10503).

doc/release-notes/10565-banner-test-improvements.md

@@ -0,0 +1 @@
+The endpoint `api/admin/bannerMessage` has been extended so the ID is returned when created

doc/release-notes/10570-extra-facet-settings.md

@@ -0,0 +1,4 @@
+Extra settings have been added giving an instance admin more choices in
+selectively limiting the availability of search facets on the Collection and Dataset pages.
+See the [Disable Solr Facets](https://guides.dataverse.org/en/6.3/installation/config.html#DisableSolrFacets) sections of the Config Guide for more info.
+

doc/release-notes/10579-avoid-solr-deletes.md

@@ -0,0 +1,9 @@
+A features flag called "reduce-solr-deletes" has been added to improve how datafiles are indexed. When the flag is enabled, 
+Dataverse wil avoid pre-emptively deleting existing solr documents for the files prior to sending updated information. This
+should improve performance and will allow additional optimizations going forward.
+
+The /api/admin/index/status and /api/admin/index/clear-orphans calls
+(see https://guides.dataverse.org/en/latest/admin/solr-search-index.html#index-and-database-consistency)
+will now find and remove (respectively) additional permissions related solr documents that were not being detected before.
+Reducing the overall number of documents will improve solr performance and large sites may wish to periodically call the 
+clear-orphans API.

doc/release-notes/10611-harvested-origin-facet.md

@@ -0,0 +1,10 @@
+NOTE that this release note supercedes the 10464-add-name-harvesting-client-facet.md note from the PR 10464.
+
+An option has been added to index the name of the Harvesting Client as the "Metadata Source" of harvested datasets and files; if enabled, the Metadata Source facet will be showing separate entries for the content harvested from different sources, instead of the current, default behavior where there is one "Harvested" facet for all such content.
+
+
+TODO: for the v6.3 release note:
+If you choose to enable the extended "Metadata Souce" facet for harvested content, set the optional feature flage (jvm option) `dataverse.feature.index-harvested-metadata-source=true` before reindexing.
+
+[Please note that the upgrade instruction in 6.3 will contain a suggestion to run full reindex, as part of the Solr upgrade, so the sentence above will need to be added to that section]
+

doc/release-notes/8985-deprecate-rsync.md

@@ -0,0 +1,8 @@
+Support for rsync has been deprecated. Information has been removed from the guides for rsync and related software such as Data Capture Module (DCM) and Repository Storage Abstraction Layer (RSAL). You can still find this information in [older versions](https://guides.dataverse.org/en/6.2/developers/big-data-support.html#data-capture-module-dcm) of the guides.
+
+The following related database settings have been deprecated as well:
+
+- :DataCaptureModuleUrl
+- :DownloadMethods
+- :LocalDataAccessPath
+- :RepositoryStorageAbstractionLayerUrl

doc/release-notes/9276-doc-cvoc-index-in.md

@@ -0,0 +1,18 @@
+## Release Highlights
+
+### Updates on Support for External Vocabulary Services
+
+Multiple extensions of the External Vocabulary mechanism have been added. These extensions allow interaction with services based on the Ontoportal software and are expected to be generally useful for other service types.
+
+These changes include:
+
+#### Improved Indexing with Compound Fields
+
+When using an external vocabulary service with compound fields, you can now specify which field(s) will include additional indexed information, such as translations of an entry into other languages. This is done by adding the `indexIn` in `retrieval-filtering`. (#10505)
+For more information, please check [GDCC/dataverse-external-vocab-support documentation](https://github.com/gdcc/dataverse-external-vocab-support/tree/main/docs).
+
+#### Broader Support for Indexing Service Responses
+
+Indexing of the results from `retrieval-filtering` responses can now handle additional formats including Json Arrays of Strings and values from arbitrary keys within a JSON Object. (#10505)
+
+**** This documentation must be merged with 9276-allow-flexible-params-in-retrievaluri-cvoc.md (#10404)

doc/release-notes/solr-9.4.1.md

@@ -0,0 +1,14 @@
+Solr 9.4.1 is now the version recommended in our installation guides and used with automated testing. There is a known security issue in the previously recommended version 9.3.0: https://nvd.nist.gov/vuln/detail/CVE-2023-36478. While the risk of an exploit should not be significant unless the Solr instance is accessible from the outside networks (which we have always recommended against), existing Dataverse installations should consider upgrading.
+
+For the upgrade instructions section:
+
+[note that 6.3 will contain other solr-related changes, so the instructions may need to contain information merged from multiple release notes!]
+
+If you are upgrading Solr:
+ - Install solr-9.4.1 following the instructions from the Installation guide.
+ - Run a full reindex to populate the search catalog.
+ - Note that it may be possible to skip the reindexing step by simply moving the existing `.../server/solr/collection1/` under the new `solr-9.4.1` installation directory. This however has not been thoroughly tested and is not officially supported.
+
+
+
+

doc/sphinx-guides/source/_static/installation/files/etc/init.d/solr

@@ -5,7 +5,7 @@
 # chkconfig: 35 92 08
 # description: Starts and stops Apache Solr
 
-SOLR_DIR="/usr/local/solr/solr-9.3.0"
+SOLR_DIR="/usr/local/solr/solr-9.4.1"
 SOLR_COMMAND="bin/solr"
 SOLR_ARGS="-m 1g"
 SOLR_USER=solr

doc/sphinx-guides/source/_static/installation/files/etc/systemd/solr.service

@@ -5,9 +5,9 @@ After = syslog.target network.target remote-fs.target nss-lookup.target
 [Service]
 User = solr
 Type = forking
-WorkingDirectory = /usr/local/solr/solr-9.3.0
-ExecStart = /usr/local/solr/solr-9.3.0/bin/solr start -m 1g
-ExecStop = /usr/local/solr/solr-9.3.0/bin/solr stop
+WorkingDirectory = /usr/local/solr/solr-9.4.1
+ExecStart = /usr/local/solr/solr-9.4.1/bin/solr start -m 1g
+ExecStop = /usr/local/solr/solr-9.4.1/bin/solr stop
 LimitNOFILE=65000
 LimitNPROC=65000
 Restart=on-failure