[question] Mysterious Package id Causing Missing prebuilt package Error

[marvin-the-mathematician]

What is your question?

The following is confusing the heck out of me and I wondered if you could shed some light on the situation for me. No worries if not, but I may be missing some fundamental perspective, which would be good to clear up.

I have bunch of pre-built package binaries in an upstream Conan remote. Most of them are standard third-party library builds and some are in-house library builds. I can see from my CI logs and from the view of my Conan remote in Artifactory that I have a particular in-house package binary in the remote. A summary of the CI job log is given below:

...
$ conan create ./ --profile:all=${PROFILE} --options:all="protobuf/*:shared=${BUILD_SHARED_LIBS}" --options:all="my_bindings/*:shared=${BUILD_SHARED_LIBS}" --options:all="my_engine/*:shared=${BUILD_SHARED_LIBS}"
======== Exporting recipe to the cache ========
RUN: git describe --tags
...
my_engine/15.0.0: Exported: my_engine/15.0.0#13e4307d2cfd9b2a27068ac15650b1c47f819413 (2024-08-01 08:00:28 UTC)
======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu20
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[options]
...
my_engine/*:shared=True
my_bindings/*:shared=True
protobuf/*:shared=True
[conf]
tools.cmake.cmaketoolchain:generator=Ninja
Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu20
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[options]
...
my_engine/*:shared=True
my_bindings/*:shared=True
protobuf/*:shared=True
[conf]
tools.cmake.cmaketoolchain:generator=Ninja
======== Computing dependency graph ========
======== Computing necessary packages ========
my_engine/15.0.0: Forced build from source
======== Installing packages ========
======== Launching test_package ========
======== Computing dependency graph ========
======== Computing necessary packages ========
======== Installing packages ========
======== Testing the package ========
======== Uploading to remote my_conan ========
-------- Checking integrity of cache packages --------
my_engine/15.0.0: Integrity checked: ok
my_engine/15.0.0:e8e84a8e45a9e90398807762d21da327c509cb00: Integrity checked: ok
-------- Checking server existing packages --------
my_engine/15.0.0: Checking which revisions exist in the remote server
my_engine/15.0.0: Recipe 'my_engine/15.0.0#13e4307d2cfd9b2a27068ac15650b1c47f819413' already in server, skipping upload
-------- Preparing artifacts for upload --------
my_engine/15.0.0:e8e84a8e45a9e90398807762d21da327c509cb00: Compressing conan_package.tgz
-------- Uploading artifacts --------
my_engine/15.0.0: Uploading package 'my_engine/15.0.0#13e4307d2cfd9b2a27068ac15650b1c47f819413:e8e84a8e45a9e90398807762d21da327c509cb00#65b740da783ed9a30ba8047dec72fdb1' (1.3MB)
Upload completed in 0s
-------- Upload summary --------
my_conan
  my_engine/15.0.0
    revisions
      13e4307d2cfd9b2a27068ac15650b1c47f819413 (Skipped, already in server)
        packages
          e8e84a8e45a9e90398807762d21da327c509cb00
            revisions
              65b740da783ed9a30ba8047dec72fdb1 (Uploaded)
...

In short it proves that the my_engine/15.0.0#13e4307d2cfd9b2a27068ac15650b1c47f819413:e8e84a8e45a9e90398807762d21da327c509cb00#65b740da783ed9a30ba8047dec72fdb1 package binary exists in the remote and describes the profile that is was built with.

My problem is on the consumption side. I am then trying to install the my_engine/15.0.0 package into my local conan cache. I am authenticated with the my_conan remote:

conan remote list
conancenter: https://center.conan.io [Verify SSL: True, Enabled: True]
my_conan: https://artifactory.corp.my.company/artifactory/api/conan/my-conan [Verify SSL: True, Enabled: True]

The profile I am using looks compatible with that described above:

conan profile show --profile:all=linux-amd64-gcc11-release
Host profile:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu20
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[options]
...
[conf]
tools.cmake.cmaketoolchain:generator=Ninja

Build profile:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu20
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[options]
...
[conf]
tools.cmake.cmaketoolchain:generator=Ninja

I can install the my_bindings/11.0.4 package successfully (and all of the other dependencies of the my_engine/15.0.0 package) using:

conan install \
  --remote my_conan \
  --requires=protobuf/3.21.12 \
  --profile:all=linux-amd64-gcc11-release \
  --options:all='protobuf/*:shared=True'
  
conan install \
  --remote my_conan \
  --requires=my_bindings/11.0.4 \
  --profile:all=linux-amd64-gcc11-release \
  --options:all='protobuf/*:shared=True' \
  --options:all='my_bindings/*:shared=True'

etc.

But when I then try to install the my_engine/15.0.0 package like so:

conan install \
  --remote my_conan \
  --requires=my_engine/15.0.0 \
  --profile:all=linux-amd64-gcc11-release \
  --options:all='protobuf/*:shared=True' \
  --options:all='my_bindings/*:shared=True' \
  --options:all='my_engine/*:shared=True'

I hit a Missing prebuilt package error! It reads:

======== Computing dependency graph ========
my_engine/15.0.0: Not found in local cache, looking in remotes...
my_engine/15.0.0: Checking remote: my_conan
my_engine/15.0.0: Downloaded recipe revision 13e4307d2cfd9b2a27068ac15650b1c47f819413
...
Graph root
    cli
Requirements
    ...
    my_engine/15.0.0#13e4307d2cfd9b2a27068ac15650b1c47f819413 - Downloaded (my_conan)
    my_bindings/11.0.4#74b329d08f7d30966a663a65b0e6caee498eec42 - Cache
    protobuf/3.21.12#cb77bd8fdc06016b8ea08d3ba2ce2004 - Cache
    ...
Test requirements
    ...
    gtest/1.14.0#7b032dca3e6e453b1002d6f01fa06b7d - Cache
Build requirements
    ...
Resolved version ranges
    ...

======== Computing necessary packages ========
my_engine/15.0.0: Compatible package ID 8890b0353360ccb4268afd9596a7a12940a71118 equal to the default package ID: Skipping it.
my_engine/15.0.0: Checking 3 compatible configurations
my_engine/15.0.0: Compatible configurations not found in cache, checking servers
my_engine/15.0.0: '8b8ea63d648da5559af6b6dc51f0c60cb7a1ee77': compiler.cppstd=20
my_engine/15.0.0: 'e16b5fe6bfa65de1e65d6cb5bdbf97d6f18e2c96': compiler.cppstd=23
my_engine/15.0.0: '4056111c8b446f57d4d1a9999b8791a619cbbe29': compiler.cppstd=gnu23
Requirements
    my_engine/15.0.0#13e4307d2cfd9b2a27068ac15650b1c47f819413:8890b0353360ccb4268afd9596a7a12940a71118 - Missing
    my_bindings/11.0.4#74b329d08f7d30966a663a65b0e6caee498eec42:c9c4927e04c62075a00c1151435a90cb343b4d69#58633e1202b4e06b3370e655f9eec52f - Cache
    protobuf/3.21.12#cb77bd8fdc06016b8ea08d3ba2ce2004:d46e24ed63986c35aec0d547bfd2fcf13be6303f#8ebc20a178c2e2c74b421e688b5f135e - Cache
Test requirements
Build requirements
Skipped binaries
    ..., protobuf/3.21.12,
ERROR: Missing binary: my_engine/15.0.0:8890b0353360ccb4268afd9596a7a12940a71118

My question is, where did the mysterious 8890b0353360ccb4268afd9596a7a12940a71118 package id come from? the conan install command clearly located the my_engine/15.0.0#13e4307d2cfd9b2a27068ac15650b1c47f819413 recipe in the remote, but there is no package associated with that recipe with the package id 8890b0353360ccb4268afd9596a7a12940a71118! And that seems to be the problem since the package id that I would expect to locate is e8e84a8e45a9e90398807762d21da327c509cb00.

I know that was a long-winded description of a very specific problem, but if I am missing something silly I would really appreciate an explanation of where I am going wrong here (I have already spent a long time trying to understand it myself and am still none the wiser).

P.S. The same error is also occurring in a docker container that very closely resembles the docker container in which the my_engine/15.0.0 was built. I am seeing the same mysterious 8890b0353360ccb4268afd9596a7a12940a71118 package id in the docker build error logs too:

..
1.521 ======== Computing necessary packages ========
1.826 my_engine/15.0.0: Compatible package ID 8890b0353360ccb4268afd9596a7a12940a71118 equal to the default package ID: Skipping it.
1.826 my_engine/15.0.0: Checking 3 compatible configurations
1.827 my_engine/15.0.0: Compatible configurations not found in cache, checking servers
1.827 my_engine/15.0.0: '8b8ea63d648da5559af6b6dc51f0c60cb7a1ee77': compiler.cppstd=20
1.864 my_engine/15.0.0: 'e16b5fe6bfa65de1e65d6cb5bdbf97d6f18e2c96': compiler.cppstd=23
1.906 my_engine/15.0.0: '4056111c8b446f57d4d1a9999b8791a619cbbe29': compiler.cppstd=gnu23
1.949 Requirements
1.949     my_engine/15.0.0#13e4307d2cfd9b2a27068ac15650b1c47f819413:8890b0353360ccb4268afd9596a7a12940a71118 - Missing
1.949     ...
1.951 ERROR: Missing binary: my_engine/15.0.0:8890b0353360ccb4268afd9596a7a12940a71118

So not specific to my local setup.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[marvin-the-mathematician]

So I think I might have spotted the issue...

Having read the docs on how-the-package-id-is-computed (https://docs.conan.io/2/reference/binary_model/package_id.html#how-the-package-id-is-computed) and realised that it depends on the exact versions of the package dependencies that appear in the conaninfo.txt file for the package it looks like I have managed (in some way) to build the my_engine/15.0.0 package in CI with reference to a dependent recipe revision that is no longer the latest revision of the recipe. Presumably that means that the resolution performed by conan install afterwards cannot locate that package any longer (because only the latest recipe revisions are considered). This may be because some layer in my build image was taken from the cache. Indeed, there is nothing in my dockerfiles at present that would lead docker build to believe that it needed to reinstall the package using the latest revision of the recipe (since only the package name and the package version number appear in the docker file).

This is all getting a little difficult to manage with respect to dockerised build containers in CI. Presumably one wants to represent the fact that the recipe revision is as important as the package version by including it in any conan install invocations that take place inside docker build invocations. I am unsure that I have fully grasped the nature of the issue quite yet, but I am getting there (I think).

In the mean time, if you have any general "best-practice" guidelines in this area (docker files and dependency management with Conan in CI) that you have adopted over time I would be very interested to hear them since at present I am shooting myself in the foot on a daily basis ;-)

[marvin-the-mathematician]

So managing all of the dependencies in my docker files using package references that do include the package recipe revision like so:

RUN conan remote login my_conan $CONAN_REMOTE_USERNAME --password $CONAN_REMOTE_PASSWORD \
    && conan install \
      --remote my_conan \
      --requires=my_engine/15.0.0#c9b50f9e416918f18fc02ee66903b2c7a4f2bc1e \
      --profile:all=linux-amd64-gcc11-release \
      --options:all='protobuf/*:shared=True' \
      --options:all='my_bindings/*:shared=True' \
      --options:all='my_engine/*:shared=True'
...

does indeed solve my problem. But it seems like an anti-pattern to me. I clearly need to read the docs more thoroughly until I understand the details of how to build a dependency graph properly without pinning everything down! But I can at least install what I need to install and build what I need to build for now, which is good. Closing.

[memsharded]

Hi @marvin-the-mathematician

Thanks for your question, sorry that we couldn't respond to it more quickly.

As a hint, Conan has now the conan graph explain command, that if given the same arguments as you did for your conan install can explain what are the differences between the existing binaries and the requested binaries.

Please give it a try the next time it happens and let us know.

It is likely that what you are seeing is the effect of shared libraries linking static or header-only libraries. In those cases, an "embedding" of the transitive dependencies binaries is happening, which means that if some of the dependencies change, it needs a new binary of the consumer of those dependencies.

So it is not as much a problem of docker or not docker, but about process and CI, when something changes upstream, it is often necessary to re-build the packages affected by those new upstream versions or revisions. We are in the process of documenting this in conan-io/docs#3799 if you are interested.

Please don't hesitate to create new tickets if you have any further question regarding package_id computation. Thanks for the feedback!