Add AVIF Thumbnail Generation #40048

JanisPlayer · 2023-08-25T19:40:25Z

Summary

This pull request aims to enable the creation of preview thumbnails for AVIF files. This is currently a work-in-progress idea that others can contribute to. I've started by implementing a basic version that generates preview thumbnails using GD.

Currently, the preview generator generates a JPEG thumbnail, but a more efficient format like WebP would be better suited, as these images don't always require an alpha channel.

Accomplished:

AVIF preview thumbnails are generated upon file upload.
AVIF preview thumbnails are displayed in the thumbnail view.

Issues:

~~The Nextcloud Imaginary version doesn't yet support AVIF; this could potentially be improved. (Difficult to impossible to implement. Add AVIF Thumbnail Generation #40048 (comment))~~
~~Alpha channel detection doesn't work with Imagick and getImageAlphaChannel for AVIF.~~
For AVIF.php, it's better to use HEIC.php as a template, as it might be better than WebP.php, and it uses ProviderV2, which can lead to better optimizations for AVIF and the removal of unnecessary comments.
Since there are many AI apps that utilize the preview generator or the previews, the API should be restructured to allow app developers to pass output format and quality as parameters. However, app developers should also query which format the preview is using. Still, I would advise against JPEG previews due to quality settings; quality settings should ideally be queried on a per-image basis rather than globally, as it impacts recognition. However, this would require more complex database management. Therefore, it might be better to stick with a global quality setting or avoid using old JPEG previews for AI apps and always generate new ones. I have created a patch for Recognize that addresses the issues for testing with the app.

Content

This pull request is intended to enable the generation of preview thumbnails for AVIF files. It's still in its early stages and open for collaborative input.

I have implemented a basic functionality that currently generates preview thumbnails using DG. However, there is room for improvement, such as using a more suitable format like WebP or JPEG instead of PNG.

Experimental features have been added that allow the generation of all images as AVIF. To enable this, 'preview_format' => 'avif' must be set; if not, JPEG is used for AVIF due to performance reasons. Therefore, AVIF is an option for experienced users.

WebP is converted to WebP by default. However, it's still under consideration whether it might be better to handle WebP similarly to AVIF due to the performance required for WebP encoding and the existing support in Imaginary.

These features have been added as tests, and I have tried not to make significant code changes.

Accomplishments:

Implemented the creation of AVIF preview thumbnails upon file upload.
Enabled the display of AVIF preview thumbnails in the thumbnail view.

TODO

Experimental:

AVIF can use as preview format:
'preview_format' => 'avif', in the config all image use from now AVIF.
Without `'preview_format' the default format for AVIF is JPEG.
Don't use a res over 1080p is really AVIF. But you can make the quality on 50.
Change quality with occ config:app:set preview avif_quality --value="30"
The speed is set on the max (10).
WebP can use as preview format:
'preview_format' => 'webp', in the config all image use from now WebP.
Change quality with occ config:app:set preview webp_quality --value="30"

Checklist

Code is properly formatted.
Sign-off message is added to all commits.
Tests (unit, integration, API, and/or acceptance) are included.
Screenshots before/after for front-end changes.

Screenshots

Android File Browser
Android Media
Android Viewer

Documentation (manuals or wiki) has been updated or is not required.
Backports requested where applicable (e.g., critical bugfixes).

For those who want to test this out, I have a ZIP file with the changes for version 28.0.1 here.
You should just need to replace the files; I'm already using it myself.
However, it's not yet complete but demonstrates what's possible.
To revert everything back, you just need to replace the modified files with the original ones.
I've added the most important changes to the files, so if you encounter any errors, you can simply report them and include the log if there's an error included.

latest.zip (27.1.2)
latest.zip (28.0.1)

# cd /tmp/ && wget https://download.nextcloud.com/server/releases/latest.zip && unzip latest.zip #to reset
cd /tmp/ && wget https://github.com/nextcloud/server/files/14053549/latest.zip
unzip -j -o latest.zip 'nextcloud/lib/private/Preview/Generator.php' -d /var/www/nextcloud/lib/private/Preview/
unzip -j -o latest.zip 'nextcloud/lib/composer/composer/autoload_classmap.php' -d /var/www/nextcloud/lib/composer/composer/
unzip -j -o latest.zip 'nextcloud/lib/composer/composer/autoload_static.php' -d /var/www/nextcloud/lib/composer/composer/
unzip -j -o latest.zip 'nextcloud/lib/private/Collaboration/Reference/LinkReferenceProvider.php' -d /var/www/nextcloud/lib/private/Collaboration/Reference/
unzip -j -o latest.zip 'nextcloud/lib/private/Preview/AVIF.php' -d /var/www/nextcloud/lib/private/Preview/
unzip -j -o latest.zip 'nextcloud/lib/private/PreviewManager.php' -d /var/www/nextcloud/lib/private/
unzip -j -o latest.zip 'nextcloud/lib/private/legacy/OC_Image.php' -d /var/www/nextcloud/lib/private/legacy/
unzip -j -o latest.zip 'nextcloud/resources/config/mimetypemapping.dist.json' -d /var/www/nextcloud/resources/config/
# Apps
unzip -u -j -o latest.zip 'nextcloud/apps/photos/lib/AppInfo/Application.php' -d /var/www/nextcloud/apps/photos/lib/AppInfo/
unzip -u -j -o latest.zip 'nextcloud/apps/viewer/cypress/e2e/images/image.avif.cy.ts' -d /var/www/nextcloud/apps/viewer/cypress/e2e/images/
unzip -u -j -o latest.zip 'nextcloud/apps/viewer/js/*' -d /var/www/nextcloud/apps/viewer/js/
unzip -u -j -o latest.zip 'nextcloud/apps/viewer/src/models/images.js' -d /var/www/nextcloud/apps/viewer/src/models/
unzip -u -j -o latest.zip 'nextcloud/apps/memories/lib/AppInfo/Application.php' -d /var/www/nextcloud/apps/memories/lib/AppInfo/
wget https://raw.githubusercontent.com/nextcloud/recognize/926f922532c88d6dea3147a73e49b3aeea9c4b00/lib/Classifiers/Classifier.php -O /var/www/nextcloud/apps/recognize/lib/Classifiers/Classifier.php

sudo chown -R www-data:www-data /var/www/nextcloud/
service apache2 restart

Signed-off-by: JanisPlayer <[email protected]>

JanisPlayer · 2023-09-05T03:17:58Z

lib/private/Preview/AVIF.php

+	}
+
+	public function isAvailable(FileInfo $file): bool {
+		return (bool)(imagetypes() & IMG_AVIF);


Could add IMG_AVIF using define for older PHP versions below 8.1 or check for support. However, it's recommended to use a higher PHP version for Nextcloud, so I don't think it's a real issue.

lib/private/legacy/OC_Image.php

@@ -714,6 +714,20 @@
 					$this->logger->debug('OC_Image->loadFromFile, WBMP images not supported: ' . $imagePath, ['app' => 'core']);
 				}
 				break;
+			case IMAGETYPE_AVIF:


lib/private/legacy/OC_Image.php

Signed-off-by: JanisPlayer <[email protected]>

JanisPlayer · 2023-09-10T16:44:49Z

I conducted an AVIF benchmark with GD, and the results are as follows:

AVIF's speed needs to be pushed to the maximum.
At high resolutions, AVIF becomes a performance issue.

I considered adding AVIF and WebP because many users desire them. I would say it's possible, but with a limited resolution. Whether the images would actually be smaller and look better is a question.

One could also test if ImageMagick performs better in this test, but I doubt it, at least without acceleration options.

The next problem is, of course, browser support. With PHP 8.1, one could integrate WebP and AVIF into the generator and simply override the preview format with a system variable. However, this solution would be tailored to administrators who are knowledgeable about it.

I would suggest a solution involving Imaginary, similar to WebP. Here, I could add the libraries and customize the settings to make it work without hanging up the entire Nextcloud when an image is too large. However, there's an issue with this approach. I'm not a Go programmer, and I would need the Nextcloud version of the Docker. Some forks partially support AVIF.

If another developer optimizes Imaginary to prevent AVIF timeouts, then I think it could be feasible and would be less disruptive to the cloud.

Regarding the apps, they still need optimization, but I'm working on it. I want to keep this fork without additional changes for custom formats. As I mentioned before, I'm not entirely sure about WebP and AVIF, which can take up to 6 times longer than JPEG through GD. It might be a good idea for a developer to decide. I could add the option for testing purposes. However, I'm not sure if this option would make everyone happy. For me, displaying AVIF images in the gallery is already a significant step.

Adding to Nextcloud is easy, and there are different ways to do it. I'm considering switching to ImageMagick to convert AVIF images to JPEG, which requires fewer changes. Similarly, converting WebP images to JPEG or WebP instead of PNG. But I would need permission from a developer for this idea. However, I think WebP image previews are unnecessarily large, and if I do it for AVIF, I might as well do it for WebP.

Next, if things continue this way, we might add JPEG XL, and suddenly Nextcloud will become the most popular viewer. :D That's why I need feedback on the best implementation.

<?php
// Report all PHP errors
error_reporting(-1);
ini_set('error_reporting', E_ALL);

for ($i = 1; $i <= 10; $i++) {
    $startTime = microtime(true);
    $image = imagecreatefromjpeg('input.jpg');  // Read a JPG file
    $maxWidth = 1920;
    $maxHeight = 1920;
    $width = imagesx($image);
    $height = imagesy($image);

    // Calculate new dimensions while maintaining aspect ratio
    if ($maxWidth > 0 && $maxHeight > 0) {
        if ($width > $maxWidth || $height > $maxHeight) {
            $aspectRatio = $width / $height;
            if ($width > $height) {
                $newWidth = $maxWidth;
                $newHeight = $maxWidth / $aspectRatio;
            } else {
                $newHeight = $maxHeight;
                $newWidth = $maxHeight * $aspectRatio;
            }
            $image = imagescale($image, $newWidth, $newHeight);
        }
    }

    ob_start();

    if ($i <= 8) {
        // Iterationen 1-8: AVIF
        $res = imageavif($image, null, 90, 10);  // Save an AVIF file
    } elseif ($i == 9) {
        // Iteration 9: WebP
        $res = imagewebp($image, null, 90);  // Save a WebP file
    } else {
        // Iteration 10: JPEG
        $res = imagejpeg($image, null, 90);  // Save a JPEG file
    }
    $fileSize = ob_get_length(); // Get the file size in bytes
    ob_get_clean();
    imagedestroy($image);

    $endTime = microtime(true);
    $executionTime = ($endTime - $startTime) * 1000; // Convert to milliseconds

    if ($i <= 8) {
	echo "AVIF: ";
    } elseif ($i == 9) {
	echo "WebP: ";
    } else {
	echo "JPEG: ";
    }

    echo "Iteration $i: Time: " . round($executionTime, 2) . " milliseconds, File Size: $fileSize bytes<br>";
}
?>

Signed-off-by: JanisPlayer <[email protected]>

lib/private/legacy/OC_Image.php

@@ -714,6 +777,16 @@
 					$this->logger->debug('OC_Image->loadFromFile, WBMP images not supported: ' . $imagePath, ['app' => 'core']);
 				}
 				break;
+			case IMAGETYPE_AVIF:
+				if (imagetypes() & IMG_AVIF) {


lib/private/legacy/OC_Image.php

+					if (!$this->checkImageSize($imagePath)) {
+						return false;
+					}
+					$this->resource = @imagecreatefromavif($imagePath);


lib/private/legacy/OC_Image.php

+			case "image/avif":
+				/** @psalm-suppress InvalidScalarArgument */
+				imageinterlace($this->resource, (PHP_VERSION_ID >= 80100 ? true : 1));
+				$retVal = imageavif($this->resource, $filePath, $this->getAvifQuality(), 10);


lib/private/legacy/OC_Image.php

 		switch ($this->mimeType) {
 			case 'image/png':
 			case 'image/jpeg':
+			case 'image/webp':


lib/private/legacy/OC_Image.php

+			case "image/avif":
+				/** @psalm-suppress InvalidScalarArgument */
+				imageinterlace($this->resource, (PHP_VERSION_ID >= 80100 ? true : 1));
+				$res = imageavif($this->resource, null, $this->getAvifQuality(), 10);


_output() is maybe useless WebP have longer time to encode as PNG and JPEG, maybe remove it too by default for WebP. Signed-off-by: JanisPlayer <[email protected]>

lib/private/legacy/OC_Image.php

+					$imageType = IMAGETYPE_WEBP;
+					break;
+				case 'avif':
+					$imageType = IMAGETYPE_AVIF;


JanisPlayer · 2023-09-12T09:37:46Z

So, just to keep you updated on the progress:

I'm currently testing the whole setup. I also conducted a sort of survey where I explained to people that AVIF comes with certain limitations (performance, resolution, browser support) and is probably not even considered for experimental use but needs to be implemented differently.

That's why I wanted to ask you, where is Nextcloud's Imaginary? I'd like to add AVIF support to it, which could potentially solve many of these problems except for browser support. Ideally, I'd like to add AVIF only for decoding, allowing AVIF to be converted to JPEG or WebP using Imaginary. There are already AVIF support forks that enable decoding with Imaginary, as far as I've seen, so you'd only need to adapt and incorporate parts of those.

Regarding AVIF image support as a preview format, I see it this way: At most, it could be added with restrictions as an extra option, with a lot of warnings in the documentation explaining the limitations. Or, it could simply be implemented through Imaginary, which would be better suited for this purpose. The issue of browser support is also something to consider and mention.

But I think you can already see that I've added it in a way that doesn't require changing the entire codebase. I want to leave that to others who plan to do that with their PR. My focus for now is primarily on making AVIF images convert to JPEG, WebP, or PNG as previews and then be processed correctly in Nextcloud apps. I've already completed that part, but I'm not really satisfied with the current solution. However, as I mentioned before, I didn't want to make extensive changes to the entire generator for adding a new image format.

I've also made changes so that WebP images are no longer converted to PNG but to WebP preview images. However, this also consumes some resources, and PNG or JPEG might be better suited for this purpose. PNG is particularly suitable due to the alpha channel. Here, you could also add a test to check if the WebP image uses an alpha channel because PNG images are always large and consume a lot of storage.

Everything else, for me, is just experimenting and seeing what's possible.

Here are some specific questions for you:

What do you think of my solution of making as few code changes as possible?
Is it reasonable to encode AVIF as JPEG, or should we consider using WebP or PNG?
How do you feel about the change where WebP is converted to a WebP preview image from now on? Is it acceptable in terms of performance with DG and WebP browser support as a preview?
Did I miss any important Nextcloud apps?
What are your thoughts on the idea of experimental features?
AVIF or WebP through DG as a preview format when set in the config - does this make sense to you? I'm not entirely sure about AVIF, and even with WebP, Imaginary can already handle it, and WebP requires some performance.
Should I add AVIF in a similar way to how it's done in HEIC.php?
Are the license comments unnecessary or auto-generated and can they be removed?

Signed-off-by: JanisPlayer <[email protected]>

JanisPlayer · 2023-11-30T01:29:46Z

Small update:
I've been running this Nextcloud modification on my system for a while now.
Everything works wonderfully, there are no major errors that I've noticed, and the log also doesn't report any issues.
Currently, I'm using my Nextcloud with Imaginary and WebP previews, but I've also tested the experimental features with AVIF over GD, and there were no problems.
I chose GD because it was the simplest option, and ImageMagick was only used if GD lacked support. The only downside to GD is that it can't handle alpha channels in AVIF images.
The only issues I noticed were with thumbnails of HEIC, but this problem seems to be related to Imaginary and not my patch.
Users haven't reported any problems with the unofficial test patch under the pull request.
Now it's up to the real Nextcloud devs to decide what stays in the pull request and how it stays, whether it can be improved.
Once I know what stays or how it stays, I can create proper documentation and start initiating pull requests for the apps.
I improvised parts of this patch to accommodate user requests, so there is always room for improvement. For example, how often the format is called from the config and the entire call process.
I added AVIF completely and cleanly because this function is the most important part of the entire pull request.
For widely used apps, I had to take care of their support. It was relatively easy for photo apps and viewer apps, but for AI apps that needed the preview generator, I had to provide my own generator with GD.
So, there may be missing apps in this regard.
Other than that, AVIF doesn't cause any issues when displaying, and all patches are fulfilled.
Looking forward to your feedback on the new features so that I can finish the rest.

cromefire · 2023-12-23T20:39:08Z

Edge Browser don't work with AVIF as preview format, need a warning in the documentation which browser not supported.

Apparently that will be solved next month, so not sure if it's even worth it to add anymore: https://www.neowin.net/news/microsoft-edge-121-finally-brings-avif-support/

Edit: It happened and is here now. AVIF is also part of Baseline 2024: https://caniuse.com/avif

Signed-off-by: JanisPlayer <[email protected]>

JanisPlayer added 7 commits August 25, 2023 20:41

Create AVIF.php

3ca2e2d

Signed-off-by: JanisPlayer <[email protected]>

add AVIF

e4f0d9c

Signed-off-by: JanisPlayer <[email protected]>

add AVIF

4068ca3

Signed-off-by: JanisPlayer <[email protected]>

add AVIF

af2f4f6

Signed-off-by: JanisPlayer <[email protected]>

add AVIF

ddab01c

Signed-off-by: JanisPlayer <[email protected]>

add AVIF

aa3fb33

Signed-off-by: JanisPlayer <[email protected]>

add AVIF

497a641

Signed-off-by: JanisPlayer <[email protected]>

JanisPlayer mentioned this pull request Aug 25, 2023

Allow more preview formats #13552

Open

github-advanced-security bot found potential problems Aug 28, 2023

View reviewed changes

joshtrichards added feature: previews and thumbnails 2. developing Work in progress labels Sep 5, 2023

Change mimeType for AVIF previews to JPEG

55d6973

Signed-off-by: JanisPlayer <[email protected]>

szaimen added this to the Nextcloud 28 milestone Sep 10, 2023

Template for AVIF as preview format removed

59bb879

Signed-off-by: JanisPlayer <[email protected]>

JanisPlayer mentioned this pull request Sep 10, 2023

add avif and heif nextcloud/photos#2047

Draft

Experimental support for WebP and AVIF as a preview

276d473

Signed-off-by: JanisPlayer <[email protected]>

github-advanced-security bot found potential problems Sep 10, 2023

View reviewed changes

szaimen requested review from rullzer, ChristophWurst, J0WI, a team, ArtificialOwl, nfebe and sorbaugh and removed request for a team September 11, 2023 06:57

ChristophWurst removed their request for review September 11, 2023 06:57

experimental default format for AVIF is JPEG

0710c14

_output() is maybe useless WebP have longer time to encode as PNG and JPEG, maybe remove it too by default for WebP. Signed-off-by: JanisPlayer <[email protected]>

github-advanced-security bot found potential problems Sep 11, 2023

View reviewed changes

lib/private/legacy/OC_Image.php

$imageType = IMAGETYPE_WEBP;

break;

case 'avif':

$imageType = IMAGETYPE_AVIF;

Check failure

Code scanning / Psalm

UndefinedConstant

Const IMAGETYPE_AVIF is not defined

JanisPlayer added 4 commits October 18, 2023 02:46

Remove Licence

c622c48

Signed-off-by: JanisPlayer <[email protected]>

Add AVIF

e8b23b5

Signed-off-by: JanisPlayer <[email protected]>

Set default AVIF quality to 50

45b5596

Signed-off-by: JanisPlayer <[email protected]>

Merge branch 'master' into JanisPlayer-patch-1

b5fa45a

Signed-off-by: JanisPlayer <[email protected]>

JanisPlayer mentioned this pull request Oct 25, 2023

Ad a own Generator and can handle with WebP and AVIF preview. nextcloud/recognize#1008

Merged

skjnldsv mentioned this pull request Nov 1, 2023

chore: 28.0.0 beta 1 #41228

Merged

This was referenced Nov 6, 2023

28.0.0 beta 2 #41314

Merged

28.0.0 beta 3 #41379

Merged

This was referenced Nov 14, 2023

28.0.0 beta 4 #41477

Merged

28.0.0 RC1 #41621

Merged

blizzz modified the milestones: Nextcloud 28, Nextcloud 29 Nov 23, 2023

This was referenced Mar 12, 2024

29.0.0 beta 2 #44152

Merged

29.0.0 beta 3 #44186

Merged

29.0.0 beta 4 #44275

Merged

This was referenced Mar 20, 2024

29.0.0 beta 5 #44361

Merged

29.0.0 beta 6 #44449

Merged

skjnldsv mentioned this pull request Mar 28, 2024

29.0.0 RC1 #44552

Merged

81 tasks

skjnldsv modified the milestones: Nextcloud 29, Nextcloud 30 Mar 28, 2024

JanisPlayer added 2 commits April 28, 2024 22:36

Merge branch 'master' into JanisPlayer-patch-1

bf0aa68

Signed-off-by: JanisPlayer <[email protected]>

Add avif

e5ac220

Signed-off-by: JanisPlayer <[email protected]>

This was referenced Jul 30, 2024

30.0.0 beta 2 #46870

Merged

30.0.0 beta 3 #46946

Merged

This was referenced Aug 5, 2024

30.0.0 beta 4 #47028

Merged

30.0.0 beta 5 #47100

Merged

skjnldsv mentioned this pull request Aug 13, 2024

30.0.0 RC1 #47190

Merged

skjnldsv removed this from the Nextcloud 30 milestone Aug 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AVIF Thumbnail Generation #40048

Add AVIF Thumbnail Generation #40048

JanisPlayer commented Aug 25, 2023 •

edited

Loading

JanisPlayer Sep 5, 2023

JanisPlayer commented Sep 10, 2023

JanisPlayer commented Sep 12, 2023 •

edited

Loading

JanisPlayer commented Nov 30, 2023

cromefire commented Dec 23, 2023 •

edited

Loading

Add AVIF Thumbnail Generation #40048

Are you sure you want to change the base?

Add AVIF Thumbnail Generation #40048

Conversation

JanisPlayer commented Aug 25, 2023 • edited Loading

Summary

Accomplished:

Issues:

Content

Accomplishments:

TODO

Checklist

JanisPlayer Sep 5, 2023

Choose a reason for hiding this comment

JanisPlayer commented Sep 10, 2023

JanisPlayer commented Sep 12, 2023 • edited Loading

JanisPlayer commented Nov 30, 2023

cromefire commented Dec 23, 2023 • edited Loading

JanisPlayer commented Aug 25, 2023 •

edited

Loading

JanisPlayer commented Sep 12, 2023 •

edited

Loading

cromefire commented Dec 23, 2023 •

edited

Loading