Can I take and place some files related to plugins in my repos?

[bigcat88]

I wanted to provide plugins to Python community(probably for pillow_heif) and I need these files:

1. common_utils.cc/common_utils.h
2. heif_plugin.cc/heif_plugin.h
3. specific plugin related files like: encoder_x265.cc/encoder_x265.h

For me currently the easiest way is to just copy paste them from this repo, as I needed to slightly modify them(maybe I did something wrong, but without modifications they generate dependency on libheif)

Q1: Is it enough to just take them, as they all contain copyright at top of each file?

Q2: For binary wheels should I also provide LICENSES_bundle with such lines:

https://github.com/bigcat88/pillow_heif/blob/5859b29f9bda0271f593829fe870fdf85177b712/LICENSES_bundled.txt#L5-L9

as each plugin will contain these files taken from libheif repo?
Or can I skip it and mention only library for which the plugin is?

Example of plugin:

#define PY_SSIZE_T_CLEAN

#include "Python.h"
#include "encoder_x265.cc"

/* =========== Module =========== */

static int setup_module(PyObject* m) {
    return 0;
}

PyMODINIT_FUNC PyInit__pillow_heif_x265(void) {
    static PyModuleDef module_def = {
        PyModuleDef_HEAD_INIT,
        "_pillow_heif_x265", /* m_name */
        NULL,                /* m_doc */
        -1,                  /* m_size */
        NULL,                /* m_methods */
    };

    PyObject* m = PyModule_Create(&module_def);
    if (setup_module(m) < 0)
        return NULL;
    return m;
}

Of course I will anyway mention Libheif in each readme and description

[bradh]

I am not going to address the licensing question (better done by others).

However it looks like at least some of the files you are copying are internal API (i.e. subject to change in patch versions). That is a technical issue, and probably indicates something is wrong (could be your side, could be libheif). Can you say what you are trying to do? As in, why don't you upstream your changes? What are you trying to expose?

[bigcat88]

I tried locally to modify libheif to not generate dependency in plugins on libheif.dylib but with no success(I know that it is possible).

Will create a test repository today/tommorow and publish here a link to it, it will be faster than describing in words and need see tests of this in CI in different OS/compilers.

[bradh]

Please go up a level in the explanation. Why do you need plugins to not depend on libheif? Clearly they only work as part of libheif, so its not like you won't need libheif at runtime? In some ways, enforcing the dependency is better.

[bigcat88]

When building binary Python wheels, all the library files that are used are placed inside the wheels.
It is not entirely logical to bundle the libheif library in each plugin, since this library will already be in the main package that uses these plugins.

I can simply describe it with an example:

There is pillow_heif which contains libheif library as it uses it directly. It will also call heif_load_plugin for plugins.

Right now I’m making pillow_heif_x265 for test which contains only the x265 library.

If I do not remove the dependency on the libheif in pillow_heif_x265, in fact there will be 2 copies of the libheif library during installation, one will come from installation of pillow_heif and the second one that will come from pillow_heif_x265

[bigcat88]

How it is currently:

How I need for packaging:

[bradh]

OK, I think I got the need.

Can you perhaps build it all, and just use different namespace packages for the plugins?

Or just exclude libheif.1.dylib at packaging time?

[farindk]

The plugins should not depend on libheif. Actually, some plugins use some code of libheif, but these files are included explicitly and should not be visible from outside.
However, I noticed that the CMakeLists.txt still referenced the heif library which might have put the reference into the plugin.
We should be able to simply remove this:

diff --git a/libheif/plugins/CMakeLists.txt b/libheif/plugins/CMakeLists.txt
index ad2f074..af3ff13 100644
--- a/libheif/plugins/CMakeLists.txt
+++ b/libheif/plugins/CMakeLists.txt
@@ -19,7 +19,7 @@ macro(plugin_compilation name varName foundName optionName defineName)
                     HAVE_VISIBILITY)
             target_compile_definitions(heif-${name} PRIVATE PLUGIN_${defineName}=1)
             target_include_directories(heif-${name} PRIVATE ${PROJECT_SOURCE_DIR} ${libheif_BINARY_DIR} ${libheif_SOURCE_DIR} ${${varName}_INCLUDE_DIRS})
-            target_link_libraries(heif-${name} PRIVATE ${${varName}_LIBRARIES} heif)
+            target_link_libraries(heif-${name} PRIVATE ${${varName}_LIBRARIES})
 
             install(TARGETS heif-${name}
                     LIBRARY DESTINATION ${PLUGIN_DIRECTORY}

Can you check whether that change solves your issue?

[bigcat88]

[ 62%] Linking CXX shared module libheif-x265.so
ld: Undefined symbols:
  _heif_image_extend_padding_to_size, referenced from:
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
  _heif_image_get_bits_per_pixel_range, referenced from:
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
  _heif_image_get_chroma_format, referenced from:
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
  _heif_image_get_colorspace, referenced from:
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
  _heif_image_get_height, referenced from:
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
  _heif_image_get_nclx_color_profile, referenced from:
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
  _heif_image_get_plane_readonly, referenced from:
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
  _heif_image_get_width, referenced from:
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
  _heif_nclx_color_profile_free, referenced from:
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o
      x265_encode_image(void*, heif_image const*, heif_image_input_class) in encoder_x265.cc.o

[bigcat88]

First modification that I made to make it work for my demo:

Removed extern here:

libheif/libheif/heif_plugin.h

Lines 286 to 288 in 51da10c

	extern struct heif_error heif_error_ok;
	extern struct heif_error heif_error_unsupported_parameter;
	extern struct heif_error heif_error_invalid_parameter_value;

As these are already defined in heif_plugin.cc
Right now I am working on initial commit and hope to finish it and it will show that it works. You can wait till evening, after that we can just compare changes in files and you will say what we will do with it.

[bigcat88]

Here is the demo repo(it is temporary one, I will create it from scratch again when it becomes final with all CI stuff):
https://github.com/bigcat88/pillow_heif_x265

Here is the commit with modifications to make it work(tested currently only on macOS, soon will test and adjust to cover other OS):
bigcat88/pillow_heif_x265@5123dcf

It builds _pillow_heif_x265.cpython-312-darwin.so file which is Python C module created from encoder_x265.cc and libheif is able to load it and use.

As common_utils and encoder_x265 (and all other plugins) are not public I rather need to keep this files in repo(and make a CI to sync them automaticly), or make a CI to pull them from libheif repo during building.

[bigcat88]

On Windows using MingGW I was not able to build a plugin without a direct dependency on libheif.dll

I was thinking of returning to the idea that Bradh suggested, simply excluding the libheif dependency at the stage of building Python wheels, but I remembered that during the packaging of wheels, libraries are assigned random suffix in names, like this:

So going this way is not possible, as bundled libheif in main package will have a random name..

[farindk]

It appears that the linking process in Linux and Windows is different. On Linux, when libheif loads the plugin, the plugin has access to all functions of the parent libheif. In Windows, it apparently has not.

My proposal would be to split off all image-related functions that are necessary in the plugins into a static library (libheif-image.a or (libheif-plugin-support.a) that can then be linked into the plugins. The needed functions are probably small enough that a simple static library will do.