Create volume flow v2

[mzur]

Resolves #701

Plan:

• Create a new PendingVolume model
• Implement a CSV metadata parser that can recognize the CSV file and return a filled VolumeMetadata object.
• Create a validation rule for a VolumeMetadata object based on the existing Image/VideoMetadata rules.
• Implement the CSV parser and validation rule for videos, too
• Create an API endpoint to create a PendingVolume for a project (as a project admin). A new PendingVolume is associated to a media type, project and can optionally have an uploaded metadata file. Each user is allowed to create only one PendingVolume at a time for each project. When a metadata file is uploaded, the metadata file type is checked based on the configured metadata parsers. If one parser recognizes the file, it parses it and the parsed metadata object is validated.
• Update the metadata handling of the legacy API endpoint to use the new mechanisms
• Create an API endpoint to update a PendingVolume. The update triggers the creation of the new volume. Then, either the PendingVolume is deleted or it is actually updated (i.e. associated with the volume) so the annotation/file label import can proceed (see Create volume flow v2 #701 (comment)).
  • Implement the case without annotation/file label import
  • Implement the case with annotation/file label import
• Update CreateNewImagesOrVideos
  • Use metadata from the file associated with the volume
  • Ignore files that are not present in the metadata, ignore metadata that is not present in the files
• Update the VolumeMetadataController request to use the new metadata parsers (making no distinction between CSV and iFDO)
• Update CloneImagesOrVideos
• Update the clone volume feature for the new volume metadata
• Update the metadata file upload UI in the edit volume view
• Implement the ImportVolumeMetadata job
• Implement the new create volume UI, remove the old one
  • Add a way to continue or delete an existing pending volume for the project
  • Implement the UI without the import features
  • Implement the UI with pre-filled data from the metadata file (Create volume flow v2 #701 (comment))
  • Implement a warning if none of the selected files are included in the metadata
  • Implement the UI for the import features
• Ignore invalid label colors and label/user UUIDs during import
• Implement a way to extend available metadata parsers through modules/packages.
  • These should also provide file extensions MIME types for validation of the form request as well as the accept attribute of the HTML input elements.
• Remove iFDOv1 support
  • Remove iFDOv1 files associated with volumes manually (if there are any)
• Remove the ParsesMetadata trait. Update everything with "metadata" or "Ifdo" in it.
• Implement generic metadata download button in volume overview (instead of iFDO button)
• Keep algorithm to merge video metadata based on multiple files (or change it to override metadata?)
• Update the manual about metadata
  • Make sure to explain the new metadata merge behavior for videos (see UpdateVolumemetadata job)
  • Also make metadata parser modules extend the manual
• Implement a metadata_parser column for volumes that specifies which metadata parser class should be used for the metadata file
• Remove the yaml PHP extension (used by biigle/reports)
• Implement an iFDOv2 export (incl. annotations) in biigle/reports (remove v1 support) (Implement iFDOv2 reports reports#117)
  • Export the non-standard label and label tree UUIDs for each label (for matching during import) and the label color
  • Export the non-standard user UUID for each annotator (for matching during import) already done with the id property.
• Test the pre-filled volume form and the import with an iFDOv2
• Test the iFDOv2 import label and user matching with an iFDO report

---

Linked PRs

• Implement iFDOv2 reports reports#117
• Implement metadata parser metadata-ifdo#3

---

Upgrade instructions

• Run migrations
• Delete iFDO storage disk (with contents)
• Configure VOLUME_METADATA_STORAGE_DISK and VOLUME_PENDING_METADATA_STORAGE_DISK (with config)

[mzur]

Here are the tests that an iFDOv2 metadata parser should fulfill:

core/tests/php/Traits/ParsesMetadataTest.php

Lines 102 to 488 in 91ac92b

	public function testParseIfdo()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	image-set-items:
	myimage.jpg:
	IFDO;
	$expect = [
	'name' => 'myvolume',
	'url' => '',
	'handle' => '20.500.12085/d7546c4b-307f-4d42-8554-33236c577450',
	'media_type' => 'image',
	'uuid' => 'd7546c4b-307f-4d42-8554-33236c577450',
	'files' => [
	['filename'],
	['myimage.jpg'],
	],
	];
	$this->assertEquals($expect, $stub->parseIfdo($input));
	}
	public function testParseIfdoHeader()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	image-set-data-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450@data
	image-set-acquisition: photo
	image-latitude: 11.8581802
	image-longitude: -117.0214864
	image-meters-above-ground: 2
	image-area-square-meter: 5.0
	image-datetime: '2019-04-06 04:29:27.000000'
	image-depth: 2248.0
	image-camera-yaw-degrees: 20
	image-set-items:
	myimage.jpg:
	IFDO;
	$expect = [
	'name' => 'myvolume',
	'url' => 'https://hdl.handle.net/20.500.12085/d7546c4b-307f-4d42-8554-33236c577450@data',
	'handle' => '20.500.12085/d7546c4b-307f-4d42-8554-33236c577450',
	'media_type' => 'image',
	'uuid' => 'd7546c4b-307f-4d42-8554-33236c577450',
	'files' => [
	['filename', 'area', 'distance_to_ground', 'gps_altitude', 'lat', 'lng', 'taken_at', 'yaw'],
	['myimage.jpg', '5.0', '2', '-2248.0', '11.8581802', '-117.0214864', '2019-04-06 04:29:27.000000', '20'],
	],
	];
	$this->assertEquals($expect, $stub->parseIfdo($input));
	}
	public function testParseIfdoVideoType()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	image-acquisition: video
	image-set-items:
	myvideo.mp4:
	IFDO;
	$expect = [
	'name' => 'myvolume',
	'url' => '',
	'handle' => '20.500.12085/d7546c4b-307f-4d42-8554-33236c577450',
	'media_type' => 'video',
	'uuid' => 'd7546c4b-307f-4d42-8554-33236c577450',
	'files' => [
	['filename'],
	['myvideo.mp4'],
	],
	];
	$this->assertEquals($expect, $stub->parseIfdo($input));
	}
	public function testParseIfdoSlideIsImageType()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	image-set-acquisition: slide
	image-set-items:
	myimage.jpg:
	IFDO;
	$expect = [
	'name' => 'myvolume',
	'url' => '',
	'handle' => '20.500.12085/d7546c4b-307f-4d42-8554-33236c577450',
	'media_type' => 'image',
	'uuid' => 'd7546c4b-307f-4d42-8554-33236c577450',
	'files' => [
	['filename'],
	['myimage.jpg'],
	],
	];
	$this->assertEquals($expect, $stub->parseIfdo($input));
	}
	public function testParseIfdoItems()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	image-set-items:
	myimage.jpg:
	image-latitude: 11.8581802
	image-longitude: -117.0214864
	image-meters-above-ground: 2
	image-area-square-meter: 5.0
	image-datetime: '2019-04-06 04:29:27.000000'
	image-depth: 2248.0
	image-camera-yaw-degrees: 20
	IFDO;
	$expect = [
	'name' => 'myvolume',
	'url' => '',
	'handle' => '20.500.12085/d7546c4b-307f-4d42-8554-33236c577450',
	'media_type' => 'image',
	'uuid' => 'd7546c4b-307f-4d42-8554-33236c577450',
	'files' => [
	['filename', 'area', 'distance_to_ground', 'gps_altitude', 'lat', 'lng', 'taken_at', 'yaw'],
	['myimage.jpg', '5.0', '2', '-2248.0', '11.8581802', '-117.0214864', '2019-04-06 04:29:27.000000', '20'],
	],
	];
	$this->assertEquals($expect, $stub->parseIfdo($input));
	}
	public function testParseIfdoImageArrayItems()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	image-set-items:
	myimage.jpg:
	- image-latitude: 11.8581802
	image-longitude: -117.0214864
	image-meters-above-ground: 2
	image-area-square-meter: 5.0
	image-datetime: '2019-04-06 04:29:27.000000'
	image-depth: 2248.0
	image-camera-yaw-degrees: 20
	IFDO;
	$expect = [
	'name' => 'myvolume',
	'url' => '',
	'handle' => '20.500.12085/d7546c4b-307f-4d42-8554-33236c577450',
	'media_type' => 'image',
	'uuid' => 'd7546c4b-307f-4d42-8554-33236c577450',
	'files' => [
	['filename', 'area', 'distance_to_ground', 'gps_altitude', 'lat', 'lng', 'taken_at', 'yaw'],
	['myimage.jpg', '5.0', '2', '-2248.0', '11.8581802', '-117.0214864', '2019-04-06 04:29:27.000000', '20'],
	],
	];
	$this->assertEquals($expect, $stub->parseIfdo($input));
	}
	public function testParseIfdoItemsOverrideHeader()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	image-latitude: 11.8581802
	image-longitude: -117.0214864
	image-meters-above-ground: 2
	image-area-square-meter: 5.0
	image-datetime: '2019-04-06 04:29:27.000000'
	image-depth: 2248.0
	image-camera-yaw-degrees: 20
	image-set-items:
	myimage.jpg:
	image-meters-above-ground: 3
	image-area-square-meter: 5.1
	image-datetime: '2019-04-06 05:29:27.000000'
	IFDO;
	$expect = [
	'name' => 'myvolume',
	'url' => '',
	'handle' => '20.500.12085/d7546c4b-307f-4d42-8554-33236c577450',
	'media_type' => 'image',
	'uuid' => 'd7546c4b-307f-4d42-8554-33236c577450',
	'files' => [
	['filename', 'area', 'distance_to_ground', 'gps_altitude', 'lat', 'lng', 'taken_at', 'yaw'],
	['myimage.jpg', '5.1', '3', '-2248.0', '11.8581802', '-117.0214864', '2019-04-06 05:29:27.000000', '20'],
	],
	];
	$this->assertEquals($expect, $stub->parseIfdo($input));
	}
	public function testParseIfdoSubItemsOverrideDefaultsAndHeader()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	image-latitude: 11.8581802
	image-longitude: -117.0214864
	image-meters-above-ground: 2
	image-area-square-meter: 5.0
	image-datetime: '2019-04-06 04:29:27.000000'
	image-depth: 2248.0
	image-camera-yaw-degrees: 20
	image-acquisition: video
	image-set-items:
	myvideo.mp4:
	- image-meters-above-ground: 3
	image-area-square-meter: 5.1
	image-datetime: '2019-04-06 05:29:27.000000'
	- image-meters-above-ground: 4
	image-datetime: '2019-04-06 05:30:27.000000'
	IFDO;
	$expect = [
	'name' => 'myvolume',
	'url' => '',
	'handle' => '20.500.12085/d7546c4b-307f-4d42-8554-33236c577450',
	'media_type' => 'video',
	'uuid' => 'd7546c4b-307f-4d42-8554-33236c577450',
	'files' => [
	['filename', 'area', 'distance_to_ground', 'gps_altitude', 'lat', 'lng', 'taken_at', 'yaw'],
	['myvideo.mp4', '5.1', '3', '-2248.0', '11.8581802', '-117.0214864', '2019-04-06 05:29:27.000000', '20'],
	['myvideo.mp4', '5.1', '4', '-2248.0', '11.8581802', '-117.0214864', '2019-04-06 05:30:27.000000', '20'],
	],
	];
	$this->assertEquals($expect, $stub->parseIfdo($input));
	}
	public function testParseIfdoFile()
	{
	$stub = new ParsesMetadataStub;
	$path = __DIR__."/../../files/image-ifdo.yaml";
	$file = new UploadedFile($path, 'ifdo.yaml', 'application/yaml', null, true);
	$expect = [
	'name' => 'SO268 SO268-2_100-1_OFOS SO_CAM-1_Photo_OFOS',
	'url' => 'https://hdl.handle.net/20.500.12085/d7546c4b-307f-4d42-8554-33236c577450@data',
	'handle' => '20.500.12085/d7546c4b-307f-4d42-8554-33236c577450',
	'media_type' => 'image',
	'uuid' => 'd7546c4b-307f-4d42-8554-33236c577450',
	'files' => [
	['filename', 'area', 'distance_to_ground', 'gps_altitude', 'lat', 'lng', 'taken_at', 'yaw'],
	['SO268-2_100-1_OFOS_SO_CAM-1_20190406_042927.JPG', '5.0', '2', '-2248.0', '11.8581802', '-117.0214864', '2019-04-06 04:29:27.000000', '20'],
	['SO268-2_100-1_OFOS_SO_CAM-1_20190406_052726.JPG', '5.1', '2.1', '-4129.6', '11.8582192', '-117.0214286', '2019-04-06 05:27:26.000000', '21'],
	],
	];
	$this->assertEquals($expect, $stub->parseIfdoFile($file));
	}
	public function testParseIfdoNoHeader()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-items:
	myimage.jpg:
	IFDO;
	$this->expectException(Exception::class);
	$stub->parseIfdo($input);
	}
	public function testParseIfdoNoItems()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	IFDO;
	$expect = [
	'name' => 'myvolume',
	'url' => '',
	'handle' => '20.500.12085/d7546c4b-307f-4d42-8554-33236c577450',
	'media_type' => 'image',
	'uuid' => 'd7546c4b-307f-4d42-8554-33236c577450',
	'files' => [],
	];
	$this->assertEquals($expect, $stub->parseIfdo($input));
	}
	public function testParseIfdoNoName()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	IFDO;
	$this->expectException(Exception::class);
	$stub->parseIfdo($input);
	}
	public function testParseIfdoNoHandle()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	IFDO;
	$this->expectException(Exception::class);
	$stub->parseIfdo($input);
	}
	public function testParseIfdoNoUuid()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	IFDO;
	$this->expectException(Exception::class);
	$stub->parseIfdo($input);
	}
	public function testParseIfdoInvalidHandle()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: abc
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	IFDO;
	$this->expectException(Exception::class);
	$stub->parseIfdo($input);
	}
	public function testParseIfdoInvalidDataHandle()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:
	image-set-name: myvolume
	image-set-handle: 20.500.12085/d7546c4b-307f-4d42-8554-33236c577450
	image-set-uuid: d7546c4b-307f-4d42-8554-33236c577450
	image-set-data-handle: abc
	IFDO;
	$this->expectException(Exception::class);
	$stub->parseIfdo($input);
	}
	public function testParseIfdoInvalidYaml()
	{
	$stub = new ParsesMetadataStub;
	$input = <<<IFDO
	image-set-header:!!
	IFDO;
	$this->expectException(Exception::class);
	$stub->parseIfdo($input);
	}
	public function testParseIfdoNoYamlArray()
	{
	$stub = new ParsesMetadataStub;
	$this->expectException(Exception::class);
	$stub->parseIfdo('abc123');
	}
	}