[bug] [lockfiles] Build-require XXX cannot be found in lockfile

[jgsogo]

I've prepared a reproducible example, didn´t have time to investigate the underlying issue:

Environment, compiler, profiles,...:

docker run -it conanio/gcc11-ubuntu16.04:1.44.0

conanfile.txt

[requires]
magnum/2020.06
libalsa/1.2.5.1

Commands

Create base lockfile (note the --build):

conan lock create --name=name --version=version --base --lockfile-out=base.lock --build --profile:host=default --profile:build=default conanfile.txt 

Specialize it for a given configuration:

conan lock create --profile:host=default --profile:build=default --lockfile=base.lock --lockfile-out=lockfile.json --name=name --version=version conanfile.txt --build --update

And then, try to install-build everything needed:

conan install --lockfile=lockfile.json conanfile.txt  --build=missing

...
ERROR: Build-require 'automake' cannot be found in lockfile

---

Error is not enough to know the recipe that build-requires this automake and fails. My guess is that there is some build-require that also build-requires automake, and this transitive expansion is not done while creating the lockfile with --build.

---

I could probably work around it if I build all the build-requires first and then remove the --build from the command line, but it would really make the process more complex.

[Hopobcn]

Hi, I tried to create a small test to reproduce this error (I think it also affects me) but I think in my case i'm hitting another error (might be related?):

tool/1.2.3

from conans import ConanFile, tools

class ToolConan(ConanFile):
    name = "tool"
    version = "1.2.3"
    settings = "os", "compiler", "build_type", "arch"

lib/1.2.3 with a build_requires

from conans import ConanFile, tools

class LibConan(ConanFile):
    name = "lib"
    version = "1.2.3"
    settings = "os", "compiler", "build_type", "arch"

    def build_requires(self):
        self.build_requires("tool/1.2.3")

and a consumer conanfile.txt

[requires]
lib/1.2.3

After exporting recipes I follow with the typical commands to create base & specialized lockfiles:

% conan lock create --name=lib --version=1.2.3 --base --lockfile-out=base.lock --profile:host=apple --profile:build=apple conanfile.txt
Requirements
    lib/1.2.3 from local cache - Cache
Packages
    lib/1.2.3:a6d9ecd51d9301cf890b4d4bfe8a9ed7e68b81f7 - Missing

% conan lock create --name=lib --version=1.2.3 --lockfile=base.lock --lockfile-out=specific.lock --profile:host=apple --profile:build=apple conanfile.txt
Requirements
    lib/1.2.3 from local cache - Cache
Packages
    lib/1.2.3:a6d9ecd51d9301cf890b4d4bfe8a9ed7e68b81f7 - Missing

And in the install phase I get an exception:

% conan install --lockfile=specific.lock --build=missing conanfile.txt
Using lockfile: '/Users/pfarre/Downloads/bug/consumer/specific.lock'
Configuration (profile_host):
[settings]
arch=armv8
arch_build=armv8
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=13.0
os=Macos
os_build=Macos
[options]
[build_requires]
[env]

Configuration (profile_build):
[settings]
arch=armv8
arch_build=armv8
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=13.0
os=Macos
os_build=Macos
[options]
[build_requires]
[env]

Traceback (most recent call last):
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/command.py", line 2232, in run
    method(args[0][1:])
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/command.py", line 523, in install
    info = self._conan.install(path=args.path_or_reference,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/conan_api.py", line 93, in wrapper
    return f(api, *args, **kwargs)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/conan_api.py", line 606, in install
    deps_install(app=self.app,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/manager.py", line 58, in deps_install
    deps_graph = graph_manager.load_graph(ref_or_path, create_reference, graph_info, build_modes,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 125, in load_graph
    deps_graph = self._resolve_graph(root_node, profile_host, profile_build, graph_lock,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 287, in _resolve_graph
    deps_graph = self._load_graph(root_node, check_updates, update,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 408, in _load_graph
    self._recurse_build_requires(graph, builder, check_updates, update, build_mode,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 343, in _recurse_build_requires
    package_build_requires = self._get_recipe_build_requires(node.conanfile, default_context)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 308, in _get_recipe_build_requires
    conanfile.build_requires = _RecipeBuildRequires(conanfile, default_context)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 31, in __init__
    self.add(build_require, context=self._default_context)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 35, in add
    build_require = ConanFileReference.loads(build_require)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/model/ref.py", line 221, in loads
    name, version, user, channel, revision = get_reference_fields(text)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/model/ref.py", line 44, in get_reference_fields
    if "#" in arg_reference:
TypeError: argument of type 'method' is not iterable

ERROR: argument of type 'method' is not iterable

---

Interestingly if I generate the base lockfile providing --build missing it also explodes:

 % conan lock create --name=lib --version=1.2.3 --base --lockfile-out=base.lock --profile:host=apple --profile:build=apple conanfile.txt --build missing  
Traceback (most recent call last):
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/command.py", line 2232, in run
    method(args[0][1:])
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/command.py", line 2046, in lock
    self._conan.lock_create(path=args.path,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/conan_api.py", line 93, in wrapper
    return f(api, *args, **kwargs)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/conan_api.py", line 1487, in lock_create
    deps_graph = self.app.graph_manager.load_graph(ref_or_path, None, graph_info, build, update,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 125, in load_graph
    deps_graph = self._resolve_graph(root_node, profile_host, profile_build, graph_lock,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 287, in _resolve_graph
    deps_graph = self._load_graph(root_node, check_updates, update,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 408, in _load_graph
    self._recurse_build_requires(graph, builder, check_updates, update, build_mode,
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 343, in _recurse_build_requires
    package_build_requires = self._get_recipe_build_requires(node.conanfile, default_context)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 308, in _get_recipe_build_requires
    conanfile.build_requires = _RecipeBuildRequires(conanfile, default_context)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 31, in __init__
    self.add(build_require, context=self._default_context)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/client/graph/graph_manager.py", line 35, in add
    build_require = ConanFileReference.loads(build_require)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/model/ref.py", line 221, in loads
    name, version, user, channel, revision = get_reference_fields(text)
  File "/Users/pfarre/Library/Python/3.8/lib/python/site-packages/conans/model/ref.py", line 44, in get_reference_fields
    if "#" in arg_reference:
TypeError: argument of type 'method' is not iterable

ERROR: argument of type 'method' is not iterable

[Hopobcn]

Sorry in my previous comment I had a bug in the reproduciton example. This should be good:

tool/conanfile.py

from conans import ConanFile, tools

class ToolConan(ConanFile):
    name = "tool"
    version = "1.2.3"
    settings = "os", "compiler", "build_type", "arch"

lib/conanfile.py

from conans import ConanFile, tools

class LibConan(ConanFile):
    name = "lib"
    version = "1.2.3"
    settings = "os", "compiler", "build_type", "arch"
    
    def build_requirements(self):
        self.build_requires("tool/1.2.3")

consumer/conanfile.txt

[requires]
lib/1.2.3

Reproduction script

#!/bin/bash

set -x

# cleanup
conan remove "*" -f
rm -rf consumer/*.lock

pushd tool
conan export .
popd

pushd lib
conan export .
popd

pushd consumer
conan lock create --name=lib --version=1.2.3 --base               --lockfile-out=base.lock     conanfile.txt
conan lock create --name=lib --version=1.2.3 --lockfile=base.lock --lockfile-out=specific.lock conanfile.txt
conan install --lockfile=specific.lock conanfile.txt  --build=missing
popd

Output:

./issue.sh 
+ conan remove '*' -f
+ rm -rf consumer/base.lock consumer/specific.lock
+ pushd tool
+ conan export .
Exporting package recipe
tool/1.2.3: A new conanfile.py version was exported
tool/1.2.3: Folder: /home/_/.conan/data/tool/1.2.3/_/_/export
tool/1.2.3: Using the exported files summary hash as the recipe revision: f8591cbf34a6ad681e4d417ea4d475e7 
tool/1.2.3: Exported revision: f8591cbf34a6ad681e4d417ea4d475e7
+ popd
+ pushd lib
+ conan export .
Exporting package recipe
lib/1.2.3: A new conanfile.py version was exported
lib/1.2.3: Folder: /home/_/.conan/data/lib/1.2.3/_/_/export
lib/1.2.3: Using the exported files summary hash as the recipe revision: 70220297d788b5f659da04b3bc98cbf2 
lib/1.2.3: Exported revision: 70220297d788b5f659da04b3bc98cbf2
+ popd
+ pushd consumer
+ conan lock create --name=lib --version=1.2.3 --base --lockfile-out=base.lock conanfile.txt
Requirements
    lib/1.2.3 from local cache - Cache
Packages
    lib/1.2.3:a9a721f0cdf549f029c26fd0adcdb5221bfe7301 - Missing

Generated lockfile: /path-to/consumer/base.lock
+ conan lock create --name=lib --version=1.2.3 --lockfile=base.lock --lockfile-out=specific.lock conanfile.txt
Requirements
    lib/1.2.3 from local cache - Cache
Packages
    lib/1.2.3:a9a721f0cdf549f029c26fd0adcdb5221bfe7301 - Missing

Generated lockfile: /path-to/consumer/specific.lock
+ conan install --lockfile=specific.lock conanfile.txt --build=missing
Using lockfile: '/path-to/consumer/specific.lock'
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=7
os=Linux
os.distro=Ubuntu
os.distro.version=18.04
os_build=Linux
[options]
[build_requires]
[env]

ERROR: Build-require 'tool' cannot be found in lockfile

And the lockfile contents looke like this:

base.lock

{
 "graph_lock": {
  "nodes": {
   "0": {
    "requires": [
     "1"
    ],
    "path": "conanfile.txt",
    "context": "host"
   },
   "1": {
    "ref": "lib/1.2.3#81f8e2817a295385bf957d9f698a966b",
    "context": "host"
   }
  },
  "revisions_enabled": true
 },
 "version": "0.4"
}

specific.lock

{
 "graph_lock": {
  "nodes": {
   "0": {
    "options": "",
    "requires": [
     "1"
    ],
    "path": "conanfile.txt",
    "context": "host"
   },
   "1": {
    "ref": "lib/1.2.3#81f8e2817a295385bf957d9f698a966b",
    "options": "",
    "package_id": "a9a721f0cdf549f029c26fd0adcdb5221bfe7301",
    "context": "host"
   }
  },
  "revisions_enabled": true
 },
 "version": "0.4",
 "profile_host": "[settings]\narch=x86_64\narch_build=x86_64\nbuild_type=Release\ncompiler=gcc\ncompiler.libcxx=libstdc++11\ncompiler.version=7\nos=Linux\nos.distro=Ubuntu\nos.distro.version=18.04\nos_build=Linux\n[options]\n[build_requires]\n[env]\n"
}

What is the reason why conan is not incorporating the build_requires into the build?

Unfortunately this isue makes implementing something like https://docs.conan.io/en/latest/versioning/lockfiles/ci.html# borderline impossible if build_requires are present into the mix.

[SSE4]

@Hopobcn as I understand, it includes build requires if you run conan lock create --build ....
but it's not a silver bullet, and you may hit #10544 (as there could be same package in two context, which is legal, e.g. in case of protobuf)

[Hopobcn]

Yes maybe my example is not suitable to reproduce this bug. Because if you don't use --build missing maybe you're literally saying that you don't want to build them, therefore no build requires are present, and this ends up with Build-require xxx cannot be found in lockfile.

[memsharded]

This referred to old Conan 1.X lockfiles. Lockfiles have been fully revamped in Conan 2.0, I am closing this ticket as outdated, please create new tickets referring to the new lockfiles for any further question or issue. Thanks!