Merge branch 'develop' into 10517-dataset-types #10517

doc/release-notes/10733-add-publication-status-to-search-api-results.md

@@ -0,0 +1,14 @@
+Search API (/api/search) response will now include publicationStatuses in the Json response as long as the list is not empty
+
+Example:
+```javascript
+"items": [
+    {
+        "name": "Darwin's Finches",
+        ...
+        "publicationStatuses": [
+            "Unpublished",
+            "Draft"
+        ],
+(etc, etc)
+```

doc/release-notes/10749-dataverse-user-permissions-api-extension.md

@@ -0,0 +1 @@
+New API endpoint "dataverses/{identifier}/userPermissions" for obtaining the user permissions on a dataverse.

doc/release-notes/10758-rust-client.md

@@ -0,0 +1,3 @@
+### Rust API client library
+
+An API client library for the Rust programming language is now available at https://github.com/gdcc/rust-dataverse and has been added to the [list of client libraries](https://dataverse-guide--10758.org.readthedocs.build/en/10758/api/client-libraries.html) in the API Guide. See also #10758.

doc/sphinx-guides/source/api/client-libraries.rst

@@ -78,3 +78,10 @@ Ruby
 https://github.com/libis/dataverse_api is a Ruby gem for Dataverse APIs. It is registered as a library on Rubygems (https://rubygems.org/search?query=dataverse).
 
 The gem is created and maintained by the LIBIS team (https://www.libis.be) at the University of Leuven (https://www.kuleuven.be).
+
+Rust
+----
+
+https://github.com/gdcc/rust-dataverse
+
+The Rust Dataverse client is a comprehensive crate designed for seamless interaction with the Dataverse API. It facilitates essential operations such as collection, dataset, and file management. Additionally, the crate includes a user-friendly command-line interface (CLI) that brings the full functionality of the library to the command line. This project is actively maintained by `Jan Range <https://github.com/jr-1991>`_.

doc/sphinx-guides/source/api/native-api.rst

@@ -678,6 +678,29 @@ The fully expanded example above (without environment variables) looks like this
 
 Note: You must have "Edit Dataverse" permission in the given Dataverse to invoke this endpoint.
 
+Get User Permissions on a Dataverse
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This API call returns the permissions that the calling user has on a particular dataverse.
+
+In particular, the user permissions that this API call checks, returned as booleans, are the following:
+
+* Can add a dataverse
+* Can add a dataset
+* Can view the unpublished dataverse
+* Can edit the dataverse
+* Can manage the dataverse permissions
+* Can publish the dataverse
+* Can delete the dataverse
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export ID=root
+
+  curl -H "X-Dataverse-key: $API_TOKEN" -X GET "$SERVER_URL/api/dataverses/$ID/userPermissions"
+
 .. _create-dataset-command: 
 
 Create a Dataset in a Dataverse Collection

doc/sphinx-guides/source/api/search.rst

@@ -114,6 +114,9 @@ https://demo.dataverse.org/api/search?q=trees
                     "identifier_of_dataverse":"dvbe69f5e1",
                     "name_of_dataverse":"dvbe69f5e1",
                     "citation":"Finch, Fiona; Spruce, Sabrina; Poe, Edgar Allen; Mulligan, Hercules, 2019, \"Darwin's Finches\", https://doi.org/10.70122/FK2/MB5VGR, Root, V3",
+                    "publicationStatuses": [
+                        "Published"
+                    ],
                     "storageIdentifier":"file://10.70122/FK2/MB5VGR",
                     "subjects":[  
                        "Astronomy and Astrophysics",
@@ -207,6 +210,9 @@ In this example, ``show_relevance=true`` matches per field are shown. Available
                     "published_at":"2016-05-10T12:57:45Z",
                     "citationHtml":"Finch, Fiona, 2016, \"Darwin's Finches\", <a href=\"http://dx.doi.org/10.5072/FK2/G2VPE7\" target=\"_blank\">http://dx.doi.org/10.5072/FK2/G2VPE7</a>, Root Dataverse, V1",
                     "citation":"Finch, Fiona, 2016, \"Darwin's Finches\", http://dx.doi.org/10.5072/FK2/G2VPE7, Root Dataverse, V1",
+                    "publicationStatuses": [
+                        "Published"
+                    ],
                     "matches":[
                         {
                             "authorName":{
@@ -297,6 +303,9 @@ The above example ``fq=publicationStatus:Published`` retrieves only "RELEASED" v
                     "identifier_of_dataverse": "rahman",
                     "name_of_dataverse": "mdmizanur rahman Dataverse collection",
                     "citation": "Finch, Fiona, 2019, \"Darwin's Finches\", https://doi.org/10.70122/FK2/GUAS41, Demo Dataverse, V1",
+                    "publicationStatuses": [
+                        "Published"
+                    ],
                     "storageIdentifier": "file://10.70122/FK2/GUAS41",
                     "subjects": [
                         "Medicine, Health and Life Sciences"
@@ -330,6 +339,9 @@ The above example ``fq=publicationStatus:Published`` retrieves only "RELEASED" v
                     "identifier_of_dataverse": "demo",
                     "name_of_dataverse": "Demo Dataverse",
                     "citation": "Finch, Fiona, 2020, \"Darwin's Finches\", https://doi.org/10.70122/FK2/7ZXYRH, Demo Dataverse, V1",
+                    "publicationStatuses": [
+                        "Published"
+                    ],
                     "storageIdentifier": "file://10.70122/FK2/7ZXYRH",
                     "subjects": [
                         "Medicine, Health and Life Sciences"
@@ -386,6 +398,10 @@ The above example ``metadata_fields=citation:*`` returns under "metadataBlocks"
                     "identifier_of_dataverse": "Sample_data",
                     "name_of_dataverse": "Sample Data",
                     "citation": "Métropole, 2021, \"JDD avec GeoJson 2021-07-13T10:23:46.409Z\", https://doi.org/10.5072/FK2/GIWCKB, Root, DRAFT VERSION",
+                    "publicationStatuses": [
+                        "Unpublished",
+                        "Draft"
+                    ],
                     "storageIdentifier": "file://10.5072/FK2/GIWCKB",
                     "subjects": [
                         "Other"

doc/sphinx-guides/source/contributor/code.md

@@ -20,6 +20,7 @@ The primary codebase and issue tracker for Dataverse is <https://github.com/IQSS
 - <https://github.com/IQSS/dataverse-client-javascript> (TypeScript)
 - <https://github.com/gdcc/dataverse-previewers> (Javascript)
 - <https://github.com/gdcc/pyDataverse> (Python)
+- <https://github.com/gdcc/rust-dataverse> (Rust)
 - <https://github.com/gdcc/dataverse-ansible> (Ansible)
 - <https://github.com/gdcc/dv-metrics> (Javascript)
 

doc/sphinx-guides/source/contributor/index.md

@@ -43,7 +43,7 @@ If you speak multiple languages, you are very welcome to help us translate Datav
 
 ## Code
 
-Dataverse is open source and we love code contributions. Developers are not limited to the main Dataverse code in this git repo. We have projects in C, C++, Go, Java, Javascript, Julia, PHP, Python, R, Ruby, TypeScript and more. To get started, please see the following pages:
+Dataverse is open source and we love code contributions. Developers are not limited to the main Dataverse code in this git repo. We have projects in C, C++, Go, Java, Javascript, Julia, PHP, Python, R, Ruby, Rust, TypeScript and more. To get started, please see the following pages:
 
 ```{toctree}
 :maxdepth: 1

src/main/java/edu/harvard/iq/dataverse/DataCitation.java

@@ -15,6 +15,7 @@
 import java.io.OutputStream;
 import java.io.OutputStreamWriter;
 import java.io.Writer;
+import java.nio.charset.StandardCharsets;
 import java.text.SimpleDateFormat;
 import java.util.ArrayList;
 import java.util.Date;
@@ -253,7 +254,7 @@ public String toBibtexString() {
 
     public void writeAsBibtexCitation(OutputStream os) throws IOException {
         // Use UTF-8
-        Writer out = new BufferedWriter(new OutputStreamWriter(os, "utf-8"));
+        Writer out = new BufferedWriter(new OutputStreamWriter(os, StandardCharsets.UTF_8));
         if(getFileTitle() !=null && isDirect()) {
             out.write("@incollection{");
         } else {
@@ -317,7 +318,7 @@ public String toRISString() {
 
     public void writeAsRISCitation(OutputStream os) throws IOException {
         // Use UTF-8
-        Writer out = new BufferedWriter(new OutputStreamWriter(os, "utf-8"));
+        Writer out = new BufferedWriter(new OutputStreamWriter(os, StandardCharsets.UTF_8));
         out.write("Provider: " + publisher + "\r\n");
         out.write("Content: text/plain; charset=\"utf-8\"" + "\r\n");
         // Using type "DATA" - see https://github.com/IQSS/dataverse/issues/4816

src/main/java/edu/harvard/iq/dataverse/Shib.java

@@ -19,6 +19,7 @@
 import org.apache.commons.lang3.StringUtils;
 
 import java.io.IOException;
+import java.nio.charset.StandardCharsets;
 import java.sql.Timestamp;
 import java.util.ArrayList;
 import java.util.Arrays;
@@ -458,9 +459,9 @@ private String getRequiredValueFromAssertion(String key) throws Exception {
         if (attributeValue.isEmpty()) {
             throw new Exception(key + " was empty");
         }
-		if(systemConfig.isShibAttributeCharacterSetConversionEnabled()) {
-			attributeValue= new String( attributeValue.getBytes("ISO-8859-1"), "UTF-8");
-		}
+        if (systemConfig.isShibAttributeCharacterSetConversionEnabled()) {
+            attributeValue= new String( attributeValue.getBytes(StandardCharsets.ISO_8859_1), StandardCharsets.UTF_8);
+        }
         String trimmedValue = attributeValue.trim();
         logger.fine("The SAML assertion for \"" + key + "\" (required) was \"" + attributeValue + "\" and was trimmed to \"" + trimmedValue + "\".");
         return trimmedValue;

src/main/java/edu/harvard/iq/dataverse/api/Access.java

@@ -286,7 +286,7 @@ private DataFile findDataFileOrDieWrapper(String fileId){
     @GET
     @AuthRequired
     @Path("datafile/{fileId:.+}")
-    @Produces({"application/xml"})
+    @Produces({"application/xml","*/*"})
     public Response datafile(@Context ContainerRequestContext crc, @PathParam("fileId") String fileId, @QueryParam("gbrecs") boolean gbrecs, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) /*throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ {
 
         // check first if there's a trailing slash, and chop it: 

src/main/java/edu/harvard/iq/dataverse/api/Admin.java

@@ -65,6 +65,7 @@
 
 import java.io.InputStream;
 import java.io.StringReader;
+import java.nio.charset.StandardCharsets;
 import java.util.Map;
 import java.util.Map.Entry;
 import java.util.logging.Level;
@@ -1155,7 +1156,7 @@ public void write(OutputStream os) throws IOException,
                         os.write(",\n".getBytes());
                     }
 
-                    os.write(output.build().toString().getBytes("UTF8"));
+                    os.write(output.build().toString().getBytes(StandardCharsets.UTF_8));
 
                     if (!wroteObject) {
                         wroteObject = true;
@@ -1269,7 +1270,7 @@ public void write(OutputStream os) throws IOException,
                         os.write(",\n".getBytes());
                     }
 
-                    os.write(output.build().toString().getBytes("UTF8"));
+                    os.write(output.build().toString().getBytes(StandardCharsets.UTF_8));
 
                     if (!wroteObject) {
                         wroteObject = true;

src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java

@@ -113,6 +113,9 @@ public class Dataverses extends AbstractApiBean {
 
     @EJB
     SwordServiceBean swordService;
+
+    @EJB
+    PermissionServiceBean permissionService;
 
     @POST
     @AuthRequired
@@ -1653,4 +1656,25 @@ public Response linkDataverse(@Context ContainerRequestContext crc, @PathParam("
         }
     }
 
+    @GET
+    @AuthRequired
+    @Path("{identifier}/userPermissions")
+    public Response getUserPermissionsOnDataverse(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) {
+        Dataverse dataverse;
+        try {
+            dataverse = findDataverseOrDie(dvIdtf);
+        } catch (WrappedResponse wr) {
+            return wr.getResponse();
+        }
+        User requestUser = getRequestUser(crc);
+        JsonObjectBuilder jsonObjectBuilder = Json.createObjectBuilder();
+        jsonObjectBuilder.add("canAddDataverse", permissionService.userOn(requestUser, dataverse).has(Permission.AddDataverse));
+        jsonObjectBuilder.add("canAddDataset", permissionService.userOn(requestUser, dataverse).has(Permission.AddDataset));
+        jsonObjectBuilder.add("canViewUnpublishedDataverse", permissionService.userOn(requestUser, dataverse).has(Permission.ViewUnpublishedDataverse));
+        jsonObjectBuilder.add("canEditDataverse", permissionService.userOn(requestUser, dataverse).has(Permission.EditDataverse));
+        jsonObjectBuilder.add("canManageDataversePermissions", permissionService.userOn(requestUser, dataverse).has(Permission.ManageDataversePermissions));
+        jsonObjectBuilder.add("canPublishDataverse", permissionService.userOn(requestUser, dataverse).has(Permission.PublishDataverse));
+        jsonObjectBuilder.add("canDeleteDataverse", permissionService.userOn(requestUser, dataverse).has(Permission.DeleteDataverse));
+        return ok(jsonObjectBuilder);
+    }
 }

src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/PasswordEncryption.java

@@ -1,6 +1,6 @@
 package edu.harvard.iq.dataverse.authorization.providers.builtin;
 
-import java.io.UnsupportedEncodingException;
+import java.nio.charset.StandardCharsets;
 import java.security.MessageDigest;
 import java.security.NoSuchAlgorithmException;
 import org.apache.commons.lang3.RandomStringUtils;
@@ -36,13 +36,13 @@ public interface Algorithm {
         public String encrypt(String plainText) {
             try {
                 MessageDigest md = MessageDigest.getInstance("SHA");
-                md.update(plainText.getBytes("UTF-8"));
+                md.update(plainText.getBytes(StandardCharsets.UTF_8));
                 byte[] raw = md.digest();
                 //String hash = Base64.encodeToString(raw, true);
                 String hash = Base64.getEncoder().encodeToString(raw);
                 return hash;
 
-            } catch (NoSuchAlgorithmException | UnsupportedEncodingException e) {
+            } catch (NoSuchAlgorithmException e) {
                 throw new RuntimeException(e);
             }
         }

src/main/java/edu/harvard/iq/dataverse/dataaccess/TabularSubsetGenerator.java

@@ -20,30 +20,16 @@
 
 package edu.harvard.iq.dataverse.dataaccess;
 
-import edu.harvard.iq.dataverse.DataFile;
-import edu.harvard.iq.dataverse.datavariable.DataVariable;
-
-import java.io.BufferedInputStream;
-import java.io.BufferedOutputStream;
 import java.io.BufferedWriter;
 import java.io.File;
 import java.io.FileInputStream;
 import java.io.FileNotFoundException;
-import java.io.FileOutputStream;
 import java.io.FileWriter;
 import java.io.IOException;
 import java.io.InputStream;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Scanner;
-import java.util.Set;
-import java.math.BigDecimal;
-import java.math.MathContext;
-import java.math.RoundingMode;
-import java.nio.ByteBuffer;
-import java.nio.channels.FileChannel;
-import java.nio.file.Paths;
-import java.nio.file.StandardOpenOption;
 import java.util.logging.Logger;
 import java.util.regex.Matcher;