dep init failing for repositories with sub-directories

[ghost]

dep master 83e2b6f

Steps to reproduce:

1. mkdir $GOPATH/src/foo
2. cd $GOPATH/src/foo

cat > foo.go <<EOL
package foo
import (
        "github.com/prometheus/client_golang/prometheus"
)
func main() {
}
EOL

4. dep init

Expected output:
$? == 0

Actual output:

Searching GOPATH for projects...
Following dependencies were not found in GOPATH. Dep will use the most recent versions of these projects.
  github.com/prometheus/client_golang
safe write of manifest and lock: second attempt failed: cannot rename /tmp/dep666066408/vendor to /home/ser/workspace/src/foo/vendor: read /tmp/dep666066408/vendor/github.com/prometheus/procfs/fixtures/self: is a directory

More Info
I believe it is because of the import of a subdirectory; I haven't encountered this error when importing a 3-segment path.

[sdboyer]

hi, welcome, thanks for the issue report!

I'd guess the problem there is the symlink - .../prometheus/procfs/self is a symlink to .../prometheus/procfs/26231. If this just started happening, it's likely because we changed/got something wrong with our recent deduplication of copy/rename funcs.

ping @ibrasho 😄

[sdboyer]

note that i can't replicate the bug locally, though - what OS are you running this on?

[ibrasho]

This seems to work with latest dep. Could you try updating github.com/golang/dep?

@sdboyer We should add a command or flag to print the current version of dep (at least the commit hash of HEAD for the build). This will help in triaging issues.

[sdboyer]

@ibrasho it would help those things, but it has other costs 😢 #209

[sdboyer]

(@serussell also noted the version of dep used at the top - from an hour ago)

[ibrasho]

oh 😞 .. We'll have to wait until we are able to version then.
And I didn't see the commit hash 🤦‍♂️ ...

dep build from that commit is not reproducing this issue for me on a macOS 10.12.

[ghost]

@ibrasho @sdboyer OS Linux 4.10.13-1-ARCH

[ghost]

I'm also getting this error on ensures, sometimes:

➜  dep ensure
grouped write of manifest, lock and vendor: second attempt failed: cannot rename /tmp/dep846302803/vendor to /home/ser/workspace/src/bitbucket.org/OWNER/PROJECT/vendor: read /tmp/dep846302803/vendor/github.com/prometheus/procfs/fixtures/self: is a directory

It is an odd error, isn't it? It doesn't make any sense to me. Are both of you on OSX?

Incidentally, go version go1.8 linux/amd64.

[ibrasho]

I managed to reproduce this on archlinux. I'll try to debug to see what's causing the issue.

@sdboyer it seems like os.Open() (in fs.copyFile within fs.go L245) returns an error when passed a symlink to a directory on archlinux. I tried this on macOS and Amazon Linux and had no issue, so I'll assume it's only happening on Arch Linux.

Any ideas on how to tackle this?

[sdboyer]

@ibrasho my oh my, that's fun.

I don't have handy access to an Arch instance (or time to tinker and set one up, sadly), but judging solely from the error above, it would appear that calls to os.Open() on Arch are being performed on the referent file, rather than the symlink itself. My approach to this would be:

1. Verify if this is actually what's happening by calling various os.Open() on a symlink to a file, and see if you get the symlink or the file
   1. If you get the file, then I think we've found a lang bug - file one and we'll figure out what to do from there
2. If you get the symlink, then maybe switch to os.OpenFile() and experiment with different flags to see if you can get it to target the symlink instead of the dir. Hopefully you'll find some way of making this call that will work on Arch...

[ibrasho]

Reproduced this on macOS after some tinkering. This bug only occurs when os.Rename() fails and it fallback to fs.renameByCopy.

os.Open() (and os.OpenFile()) returns the target of the symlink. (I'm not able to find options to make open(2) return the symlink itself.

Anyhow, I found a solution and submitted a PR.

[ibrasho]

@serussell Could you update dep and try again now that #657 is merged?

[ghost]

Fixed! Thanks! Awesome analysis & turn-around. You guys rock.