[question] requiring "unstable" builds for a subset of dependencies (potential bug)

[stackfun]

What is your question?

Summary:

It’s common for teams in my organization to require “unstable” builds for a subset of dependencies. What’s best practice for this? Our current solution seems to have encountered a conan bug or limitation.

Our current setup:

Our organization made a decision to not use the unstable/stable channel names in our private conan server because it seems to be discouraged now. Instead, we have separate “release” and “unstable” repositories, Team’s CI builds publish to the “unstable” repositories, and promote to “release” in a different workflow.
The remotes are configured such that the “release” repository is first in the list, the “unstable” last.
In CI builds, teams publish an unstable alias, like “my_package/(2.0.0-unstable)” which is an alias to the latest unstable 2.0.0 revision. This unstable alias does not get promoted to the “release” repository.

How to reproduce:

> conan install my_package/2.0.0" --update       # succeeds, gets latest from release repo
> conan install my_package/(2.0.0-unstable)" --update     # fails with error below

my_package/2.0.0: Retrieving from predefined remote 'release'
my_package/2.0.0: Trying with 'release'...
ERROR: my_package/2.0.0 was not found in remote 'release'

> conan remove "my_package/*" -f     # ugly workaround
> conan install my_package/(2.0.0-unstable)" --update   # now succeeds, get both alias and the actual package from unstable repo

What I think is happening:
When we call conan install from an empty cache, this works as expected - always picks up the latest package from the unstable repo even if the release repo is in the remotes list. If the conanfile.py from the “release” repo is in the cache, conan seems to always try to install from the release repo even when specifying the unstable alias.
It seems to be exhibiting similar behavior as conan info --update

Adding -r unstable works as well for this example, but won’t work for our full conanfile.py setup since we only want to install a subset of dependencies from the unstable repo.

Workaround:

• Clearing the cache worked, but this definitely isn’t ideal.
• Another workaround I can think of right now is to use channel names… publish the same package twice to the “unstable” repo, one with the “stable” and “unstable” channel name then only promote the package from the “stable” channel from the “unstable” repo to the “stable” repo. It seems redundant and thought using channel names like this was not considered good practice anymore.

Additional questions:

• Why is Conan retrieving from “predefined remote” even with the --update flag? Perhaps this is a bug in how aliases are handled?
• Is there going to be a difference in how this works in Conan 2.0?
• Are there other changes we can make in our conanfiles or publishing process?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @stackfun

I am trying to understand the flow:

In CI builds, teams publish an unstable alias, like “my_package/(2.0.0-unstable)” which is an alias to the latest unstable 2.0.0 revision. This unstable alias does not get promoted to the “release” repository.

Does this mean that all the recipes are changed to depend on it like requires = "my_package/(2.0.0-unstable)"? It is not clear what is the advantage of using such alias, and to which packages versions will be them resolving to.

Have you considered a different approach, like having the unstable repo first, then the stable one. And not changing any user, channel or version reference, the packages are unstable because they are in the unstable repo. In this way, packages that exist in unstable are used, and those who are not unstable, they are used from the stable one, isn't this the desired behavior?

A heads up: Conan 2.0 does not keep any track of remotes anymore. So the "predefined remote" concept is not a thing. Furthermore, it is possible that the unstable versions will be easily defined by their recipe revisions, and using lockfiles will be way easier to manage more deterministically.

[stackfun]

Thanks for the quick response! I'll add some more details to our organization's workflow.

We have "component" teams work closely together, and one component may depend on another.
And then we have "product" teams which pick up all the components from the "release" aka "stable" repo. There are multiple products, and each product depends on a different subset of components.

Most times, component teams want to pick up the "stable" builds of their dependencies. However, sometimes features require changes in multiple components, and component teams have to work together. It'll be ideal if the downstream component can temporarily pick up "unstable" package for only the team(s) they're working with for the feature, and their build will automatically pick up the latest only for this package. Changing the remote list order won't be ideal for them since there are other components they integrate with, and they'd prefer stable builds for those.

We could ask each product teams to use lockfiles, or specify a stricter version range, but this requires a lot of communication. Due to the way our organization is structured, communication between component teams is excellent (in the same timezone, same division), but communication between components and products is not (different timezones, different company division, etc).

We used to publish packages to Perforce, so teams are used to manage dependencies with clientspec paths like //depot/unstable/package_a/2.0.0/... and then change the path to //depot/stable/package_a/2.0.0/... when their integration is complete. So component teams are looking for something very similar in Conan. They're not familiar with Conan lockfiles yet, but I think it would be a great option if there is an easy way to only update a specific dependency from the "unstable" repo. I proposed the requires = "my_package/(2.0.0-unstable)" alias approach primarily because it resembles something the engineers are familiar with. Once they're done with the integration, they can revert to requires = "my_package/[^2.0.0]" and the dependency will be picked up from the "stable" remote again.

A heads up: Conan 2.0 does not keep any track of remotes anymore.

I presume tracking is going away only in the Conan cache? I see there's a metadata.json file with remote names in Conan 1.x.

[memsharded]

Most times, component teams want to pick up the "stable" builds of their dependencies. However, sometimes features require changes in multiple components, and component teams have to work together. It'll be ideal if the downstream component can temporarily pick up "unstable" package for only the team(s) they're working with for the feature, and their build will automatically pick up the latest only for this package. Changing the remote list order won't be ideal for them since there are other components they integrate with, and they'd prefer stable builds for those.

But the "unstable" repo should only be available for the "components" team, right? But not for the "products" team.

We could ask each product teams to use lockfiles, or specify a stricter version range, but this requires a lot of communication. Due to the way our organization is structured, communication between component teams is excellent (in the same timezone, same division), but communication between components and products is not (different timezones, different company division, etc).

Yes, definitely, this would be for "components" teams only, not for product team. The "product" team should always have stable things in their available repo.

They're not familiar with Conan lockfiles yet, but I think it would be a great option if there is an easy way to only update a specific dependency from the "unstable" repo. I proposed the requires = "my_package/(2.0.0-unstable)" alias approach primarily because it resembles something the engineers are familiar with. Once they're done with the integration, they can revert to requires = "my_package/[^2.0.0]" and the dependency will be picked up from the "stable" remote again.

Yes, this is the way it works (mostly in 2.0) to be fully transparent/automatic (not requiring changes in recipes):

• Dev does some changes to a package, that creates a new recipe revision: pkga/1.0#rev1. During this creating they can easily capture a "conan.lock" lockfile (user custom filename, like pkgaunstable.lock)
• That specific revision is uploaded to the "unstable" repo
• Users wanting to force to use that specific recipe revision can just use the conan install --lockfile=pkgunstable.lock (depending on how locks are managed, it can require a --lockfile-partial meaning that not all dependencies are defined in the lockfile, only some of them. This will resolve the right revision in the "unstable" repo, as it doesn't exist anywhere else.
• By default, yes, the unstable repo should be latest, so it resolves things with higher priority in the stable repo
• It is possible to do merges of lockfiles, and to create pkgaunstable.lock that also include down to the consuming product

I presume tracking is going away only in the Conan cache? I see there's a metadata.json file with remote names in Conan 1.x.

Yes, the metadata.json file does not exist anymore in 2.0. That removes the tracking to the remote (the whole conan cache is different in 2.0, including the paths, to be always short, and the cache in 2.0 will be able to manage multiple revisions too)

[stackfun]

But the "unstable" repo should only be available for the "components" team, right? But not for the "products" team.

The "unstable" repo is used by both component and product teams. Right now, it's just an oral agreement that product teams only pick up components from the stable repo but we'll definitely consider creating additional repos.

Thanks about all the information on lockfiles - I also reviewed the conan tribe proposal and discussion were also super helpful.

I'm sold on you recommendation, and tested it out in Conan 2 and it worked great. However, had some difficulties with Conan 1.x, like generating the lockfile with just the "unstable" dependency, and merging lockfiles (requires the same graph topology). Unfortunately, our org is stuck on Conan 1.x for a few quarters while we wait on recipe authors to migrate their recipes to be compatible with Conan 2. Do you have a different or slightly modified recommendation for Conan 1.x?

I can't thank you and your team enough and can't wait to migrate to Conan 2!

[memsharded]

However, had some difficulties with Conan 1.x, like generating the lockfile with just the "unstable" dependency, and merging lockfiles (requires the same graph topology). Unfortunately, our org is stuck on Conan 1.x for a few quarters while we wait on recipe authors to migrate their recipes to be compatible with Conan 2. Do you have a different or slightly modified recommendation for Conan 1.x?

Yes, these are some of the known limitations of lockfiles in 1.X. We are aware they can be quite inconvenient, and the truth is that there is no great alternative (using lockfiles).

Do you have a different or slightly modified recommendation for Conan 1.x?

Well, lockfiles are an orthogonal way to lock versions and revisions, without having to modify recipes. But if devs already have to modify requires to explicitly define the specific stable/unstable variants, then that is indeed basically the approach. There could be different variants of it though:

• Creating different channels (or even using conan copy in 1.X, not great, but might do the work until 2.0), and then changing the downstream requires to use the "unstable" channel. Is not great, but if requires change is done explicitly then it might be doable.
• Avoiding the channel change with an alias. Similar to the above, but avoiding the conan copy
• Using revisions and include revisions explicitly in the requires. This is a bit less intuitive than using a different package name, but as an advantage it doesn't require any renaming (conan copy), creation, or extra alias packages.

I am thinking of a new possible feature: controlling version-ranges pre-releases with some global configuration. In any case, this will take some time, most likely 2.X, I will file a ticket for it.

[stackfun]

leaning towards approach 2 for now, thanks again @memsharded!