Nix.fetchGit surprisingly uses export-ignore git attribute

[AllanBlanchard]

Problem

The function fetchGit uses the .gitattributes to determine whether some file should be included. Consequently, if a part of the repository is excluded (say for example that manuals are not packaged into the archive that git must generate), then it is not checked out by Nix.

I find this behavior really surprising, and I really hesitated to report it as a bug. Basically, when I ask for a git repository, I expect to get more or less the equivalent of a clone of the repository (or ref, or whatever that is related to git) and not the archive that would be generated from this repository. Is this behavior intended?

Checklist

• checked latest Nix manual - builtins - fetchGit
• checked [open documentation issues and pull requests] for possible duplicates

Proposal

• clarify whether the behavior is intended or not
  • if not: add a field useExport that would determine whether the attribute should be considered?
• document depending on the previous decision
  • if is intended, indicate something like "Please note that the command behaves as if Nix was generating an archive using git archive, thus taking in account .gitattributes" ?

[thufschmitt]

Do you have a simple repro for that (or even a pointer to the bit of the code that does that)?

From what I can see, we're using git ls-files to know which files to add to the store. Looking at the manpage it shouldn't respect export-ignore, but maybe I'm missing something?

[AllanBlanchard]

Yes, here is a script:

mkdir repro ; cd repro
echo a > a
echo b > b
echo "b export-ignore" > .gitattributes
cat << EOF > p.nix
{ stdenv } :
stdenv.mkDerivation rec {
  name = "my_p" ;
  src = fetchGit { url=./. ; };
  buildPhase = "ls b" ;
  installPhase = "touch \$out";
}
EOF
git init .
git add a b .gitattributes p.nix
git commit -m "commit"
nix-build -E "with import <nixpkgs> {}; callPackage ./p.nix {}"

which returns for the nix command:

this derivation will be built:
  /nix/store/pim19lqszaj304pkfai466s5s4dcr542-my_p.drv
building '/nix/store/pim19lqszaj304pkfai466s5s4dcr542-my_p.drv'...
unpacking sources
unpacking source archive /nix/store/l2zr0q52x7br95ndk863l45s0rrhjwgy-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
ls: cannot access 'b': No such file or directory
error: builder for '/nix/store/pim19lqszaj304pkfai466s5s4dcr542-my_p.drv' failed with exit code 2;
       last 8 log lines:
       > unpacking sources
       > unpacking source archive /nix/store/l2zr0q52x7br95ndk863l45s0rrhjwgy-source
       > source root is source
       > patching sources
       > configuring
       > no configure script, doing nothing
       > building
       > ls: cannot access 'b': No such file or directory
       For full logs, run 'nix log /nix/store/pim19lqszaj304pkfai466s5s4dcr542-my_p.drv'.

Interestingly, I just realized that if one makes the tree dirty, then b becomes available:

echo x >> a
nix-build -E "with import <nixpkgs> {}; callPackage ./p.nix {}"

Results in:

warning: Git tree '/tmp/repro' is dirty
this derivation will be built:
  /nix/store/khl5qv9vnk2gvk33ba68zizkj458nh3g-my_p.drv
building '/nix/store/khl5qv9vnk2gvk33ba68zizkj458nh3g-my_p.drv'...
unpacking sources
unpacking source archive /nix/store/iz3nbxaf8dm4qs639xqppmbd23jzjqma-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
b
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/id6km0m45bavr3ghl8if9hs52mfj7k2l-my_p
strip is /nix/store/61zfi5pmhb0d91422f186x26v7b52y5k-gcc-wrapper-11.3.0/bin/strip
patching script interpreter paths in /nix/store/id6km0m45bavr3ghl8if9hs52mfj7k2l-my_p
checking for references to /build/ in /nix/store/id6km0m45bavr3ghl8if9hs52mfj7k2l-my_p...
/nix/store/id6km0m45bavr3ghl8if9hs52mfj7k2l-my_p

Maybe some context might be useful. I use Nix in a continuous integration, thus from the reprository extracted by GitLab, I run Nix to build the project and some other components that are ignored when creating a build archive (for example manuals). Thus with this current behavior, since the manuals are ignored during export, they are not available for the CI targets that build the manuals.

[jbboehr]

Yeah, been running into this issue too. It's standard in composer-land to export-ignore all the dotfiles and tests to keep the vendor directory smaller, but it causes my CI setup to fail since the tests don't get exported. But only when the git tree is clean...

[zebreus]

fetchGit currently uses git archive for most cases. However, when also fetching submodules, it basically clones the repository and deletes the .git folder. When no reference or revision is specified for a local repository it uses git ls-files and copies all listed files. All three paths have slightly different behavior, which is annoying.

So you can also enable submodules in the example for b to become available:

mkdir repro ; cd repro
echo a > a
echo b > b
echo "b export-ignore" > .gitattributes
cat << EOF > p.nix
{ stdenv } :
stdenv.mkDerivation rec {
  name = "my_p" ;
  src = fetchGit { url=./. ; submodules=true; };
  buildPhase = "ls b" ;
  installPhase = "touch \$out";
}
EOF
git init .
git add a b .gitattributes p.nix
git commit -m "commit"
nix-build -E "with import <nixpkgs> {}; callPackage ./p.nix {}"

In my opinion, using git archive is a more suitable default behavior than git clone. That way, fetching from git URLs (git+https://github.com/some/repo) produces the same result as fetching from GitHub URLs like github:some/repo. Changing the current default would probably break some things. I think it would be nice to have the option to select what fetchGit does. The current behavior should definitely be better documented.