[question] how to use lockfiles together with build-requires?

[mjb6]

Hello,
I took your lockfile ci example and modified the profile to contain a build_requires package with my cross-compilers.
When running the lockfile ci example, I get:

libd/0.1@user/testing: Forced build from source
ERROR: Build-require 'toolchain' cannot be found in lockfile

How can I include the build_requires into the lockfiles?

Some more info for reference:
I'm using conan 1.55.0

Here is my lockfile-ci-example profile:

$ cat conan-examples/features/lockfiles/ci/tmp/.conan/profiles/
default
[settings]
os=Linux
os_build=Linux
arch=armv8
arch_build=x86_64
compiler=gcc
compiler.version=9
compiler.libcxx=libstdc++11
build_type=Release
[options]
[build_requires]
toolchain/[>=202212151335]@
[env]

The toolchain package is installed locally in the cache and is used for the builds.

It seems that the error appears in the simulated "product pipeline" when performing the builds detected by calculate build order, when running this command: conan install libd/0.1 --build=libd/0.1 --lockfile=locks/app1_release.lock --lockfile-out=locks/app1_release_updated.lock"

The app1_release.lock looks like this and misses to lock the dependency to the toolchain package.

cat app1_release.lock
{
 "graph_lock": {
  "nodes": {
   "0": {
    "ref": "libb/0.2@user/testing",
    "options": "",
    "package_id": "39b0b5fad32aaf2a462d27d4282d12c08bd6c818",
    "prev": "0",
    "requires": [
     "1"
    ],
    "context": "host"
   },
   "1": {
    "ref": "liba/0.1@user/testing",
    "options": "",
    "package_id": "4024617540c4f240a6a5e8911b0de9ef38a11a72",
    "prev": "0",
    "context": "host"
   },
   "4": {
    "ref": "app1/0.1@user/testing",
    "options": "",
    "package_id": "15d2c695ed8d421c0d8932501fc654c8083e6582",
    "requires": [
     "5"
    ],
    "context": "host"
   },
   "5": {
    "ref": "libd/0.1@user/testing",
    "options": "",
    "package_id": "b03c813b34cfab7a095fd903f7e8df2114e2b858",
    "requires": [
     "0",
     "6"
    ],
    "context": "host"
   },
   "6": {
    "ref": "libc/0.1@user/testing",
    "options": "",
    "package_id": "39b0b5fad32aaf2a462d27d4282d12c08bd6c818",
    "prev": "0",
    "requires": [
     "1"
    ],
    "context": "host"
   }
  },
  "revisions_enabled": false
 },
 "version": "0.4",
 "profile_host": "[settings]\narch=armv8\narch_build=x86_64\nbuild_type=Release\ncompiler=gcc\ncompiler.libcxx=libstdc++11\ncompiler.version=9\nos=Linux\nos_build=Linux\n[options]\n[build_requires]\n*: toolchain/[>=202212151335]\n[env]\n"

What I tried out so far:
When changing:

run("conan lock create --reference=app1/0.1@user/testing --lockfile=locks/app1_base.lock  --lockfile-out=locks/app1_release.lock")

To not use --reference but instead point to the app1/conanfile.py, then magically, the lockfile contains the lock of the toolchain package, but the issue "ERROR: Build-require 'toolchain' cannot be found in lockfile" remains.

$ cat locks/app1_release.lock
{
 "graph_lock": {
  "nodes": {
   "0": {
    "ref": "libb/0.2@user/testing",
    "options": "",
    "package_id": "39b0b5fad32aaf2a462d27d4282d12c08bd6c818",
    "prev": "0",
    "requires": [
     "1"
    ],
    "context": "host"
   },
   "1": {
    "ref": "liba/0.1@user/testing",
    "options": "",
    "package_id": "4024617540c4f240a6a5e8911b0de9ef38a11a72",
    "prev": "0",
    "context": "host"
   },
   "5": {
    "ref": "libd/0.1@user/testing",
    "options": "",
    "package_id": "b03c813b34cfab7a095fd903f7e8df2114e2b858",
    "requires": [
     "0",
     "6"
    ],
    "context": "host"
   },
   "6": {
    "ref": "libc/0.1@user/testing",
    "options": "",
    "package_id": "39b0b5fad32aaf2a462d27d4282d12c08bd6c818",
    "prev": "0",
    "requires": [
     "1"
    ],
    "context": "host"
   },
   "7": {
    "options": "",
    "requires": [
     "5"
    ],
    "build_requires": [
     "8"
    ],
    "path": "../app1/conanfile.py",
    "context": "host"
   },
   "8": {
    "ref": "toolchain/202212151335",
    "options": "",
    "package_id": "5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9",
    "prev": "0",
    "context": "host"
   }
  },
  "revisions_enabled": false
 },
 "version": "0.4",
 "profile_host": "[settings]\narch=armv8\narch_build=x86_64\nbuild_type=Release\ncompiler=gcc\ncompiler.libcxx=libstdc++11\ncompiler.version=9\nos=Linux\nos_build=Linux\n[options]\n[build_requires]\n*: toolchain/[>=202212151335]\n[env]\n"

Is the difference between using --reference and directly referring conanfile.py intended?

Can you please give me a hint how to solve this?
Thanks and best regards

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[mjb6]

Hello
in the meantime, I tried following things:

• in the profile, I renamed build_requires to tool_requires
• I explicitly set --profile:host with my cross compile profile and --profile:build for my toolchain package that contains the compiler.

The resulting lockfile contains again the dependency to the toolchain, this time with context: build
But only the App references the toolchain, the libs do not reference it.

...
    "6": {
     "ref": "toolchain/202212200812#bcb2b9856af6f2fd34eada5eb04c0b1b",
     "options": "",
     "package_id": "5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9",
     "prev": "4579392e4424fd711d31452a127791a1",
     "context": "build"
    }
   },
   "revisions_enabled": true
...

I still get the error. I would appreciate any help :)

@memsharded: during my experiments, I might have discovered another minor bugs. Is it worth creating it as separate bug ticket? Here's a summary:

• there seems to be a dependency to the default profile when running conan lock create --base, even though the docs say: "It does not store the profile, and it does not capture the package". You can reproduce it by creating a broken default profile (e.g that contains the setting invalid=profile) and run conan lock create --base. It will fail with ERROR: 'settings.invalid' doesn't exist. In this case creation of the base lock file only works when specifying another profile
• and the different behavior between conan lock create --reference and conan lock create <path-to-conanfile> (as described above)

[connorsmacd]

Duplicate of #10544?

[mjb6]

Duplicate of #10544?

Thanks for the link and your hint about using 2.0 lockfiles. I will give it a try if 2.0 resovles this issue as well. I think it's not entirely a duplicate cause I dont have the dependency as both requires and build_requires but only as build_requires (which is defined in the profile).

cheers

[mjb6]

I do not face this issue with conan 2.0.
To try it out, I used a modified version of examples 2 repo: conan-io/examples2@main...mjb6:examples2:main

[BoilerTom90]

@mjb6

I'd like to add a couple comments and/or questions on this issue as I'm experiencing a similar problem when trying to use lock files with conan 1.x. I'm the one that created issue: #13162

• First of all, there's a lot of talk about conan 2.0 (from a consumer perspective), but I tried that yesterday, and the issue I ran into is a handful of the packages/recipes we're trying to use are not compliant with 2.0. The whole ecosystem needs to support conan 2.0 before it's a viable option.

• build_requires or tool_requires: Currently, in our conanfile.txt, there's a build_requires section. Yet, in some of the 1.x documentation, I see references to tool_requires. Which is it supposed to be? I'm using conan.cmake to create the conanfile as part of our cmake/devops pipeline, and it outputs build_requires not tool_requires

• in our generated conan.base.lock and conan.lock files, I see the "build_requires" sections output:

{
    "build_requires": [
     "32",
     "33",
     "34"
    ],
}

When I look at where some of these references are listed in the lock files, I wonder if the context value is correct. The context value is "host" throughout the entire file. Should the context value be set to build or host?

   "32": {
    "ref": "nasm/2.15.05#799d63b1672a337584b09635b0f22fc1",
    "options": "",
    "package_id": "01edd76db8e16db9b38c3cca44ec466a9444c388",
    "prev": "bc150d26d6dde7a726c4b39a62cecab8",
    "context": "host"
   },

[mjb6]

Hello @BoilerTom90, I switched from build_requires to tool_requires just to see if the issue persists, this was kind of a desperate move..
In the docs, I found about the difference:

The tool requirement was formerly named “build requirement” and has been renamed to highlight that the usage of this kind of requirement must be for “tools” exclusively, not being valid for libraries to express a “private” require or other meanings.
https://docs.conan.io/1/devtools/build_requires.html#build-requires

I started looking into the conan 2 docs today and tried a few things. To me it also does not look ready, I was e.g. missing some features that are not marked as deprecated like https://docs.conan.io/1/reference/env_vars.html#conan-user-home
Neverthless, I hope a stable 2.0 release happens soon, in order to get rid of the architectural burdens that complicate the usage of v1

[BoilerTom90]

@mjb6

When you switched from build_requires to tool_requires, did it make a difference?

I was also thinking about hand-modifying the lock file and changing the value of the context attribute from host to build for the build_requires entries to see if that helped. Thoughts?

-- Update --
I hand-modified my conan.lock file to change the context attribute from host to build for the build dependencies. Then, I did a conan install specifying the lockfile option, and it worked. I surmise the issue with conan is the conan lock create command doesn't properly set the context attribute for build_requires entries.

Tom