Skip to content

Commit

Permalink
Merge pull request #11035 from IQSS/s3-file-detect
Browse files Browse the repository at this point in the history
docs: file detection using first few bytes disabled on S3 direct upload
  • Loading branch information
ofahimIQSS authored Nov 19, 2024
2 parents 6d0afad + e5f8415 commit c814ee7
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 1 deletion.
3 changes: 2 additions & 1 deletion doc/sphinx-guides/source/api/native-api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3617,7 +3617,8 @@ The fully expanded example above (without environment variables) looks like this
Currently the following methods are used to detect file types:
- The file type detected by the browser (or sent via API).
- JHOVE: https://jhove.openpreservation.org
- Custom code that reads the first few bytes. As explained at :ref:`s3-direct-upload-features-disabled`, this method of file type detection is not utilized during direct upload to S3, since by nature of direct upload Dataverse never sees the contents of the file. However, this code is utilized when the "redetect" API is used.
- JHOVE: https://jhove.openpreservation.org . Note that the same applies about direct upload to S3 and the "redetect" API.
- The file extension (e.g. ".ipybn") is used, defined in a file called ``MimeTypeDetectionByFileExtension.properties``.
- The file name (e.g. "Dockerfile") is used, defined in a file called ``MimeTypeDetectionByFileName.properties``.
Expand Down
1 change: 1 addition & 0 deletions doc/sphinx-guides/source/developers/big-data-support.rst
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ Features that are Disabled if S3 Direct Upload is Enabled
The following features are disabled when S3 direct upload is enabled.

- Unzipping of zip files. (See :ref:`compressed-files`.)
- Detection of file type based on JHOVE and custom code that reads the first few bytes. (See :ref:`redetect-file-type`.)
- Extraction of metadata from FITS files. (See :ref:`fits`.)
- Creation of NcML auxiliary files (See :ref:`netcdf-and-hdf5`.)
- Extraction of a geospatial bounding box from NetCDF and HDF5 files (see :ref:`netcdf-and-hdf5`) unless :ref:`dataverse.netcdf.geo-extract-s3-direct-upload` is set to true.
Expand Down

0 comments on commit c814ee7

Please sign in to comment.