Could not introduce package, as its subpackage does not contain usable Go code

[ChimeraCoder]

We're migrating Veneur (github.com/stripe/veneur) to dep, but getting an error on master (current: 08b7908a70f4c76fd8a84635415a477811605e39).

The error appears similar to #387 - it complains that the subpackage stathat.com/c/consistent doesn't contain usable Go code, which... well, isn't true, either for the vendored version in the repository or for the version in my GOPATH.

This happens on two different machines with completely different Go setups, so it doesn't look like a local issue.

What version of Go (go version) and dep (git describe --tags) are you using?

go version go1.8.3, both darwin/amd64 and linux/amdx64

v0.1.0-286-g0a63c47

What dep command did you run?

$ dep init -v
Root project is "github.com/stripe/veneur"
 15 transitively valid internal packages
 26 external packages imported from 18 projects
(0)   ✓ select (root)
(1)     ? attempt github.com/golang/protobuf with 1 pkgs; 1 versions to try
(1)         try github.com/golang/protobuf@master
(1)     ✓ select github.com/golang/protobuf@master w/1 pkgs
(2)     ? attempt gopkg.in/yaml.v2 with 1 pkgs; 1 versions to try
(2)         try gopkg.in/yaml.v2@v2
(2)     ✓ select gopkg.in/yaml.v2@v2 w/1 pkgs
(3)     ? attempt golang.org/x/sys with 1 pkgs; 1 versions to try
(3)         try golang.org/x/sys@master
(3)     ✓ select golang.org/x/sys@master w/1 pkgs
(4)     ? attempt stathat.com/c/consistent with 1 pkgs; 1 versions to try
(4)         try stathat.com/c/consistent@master
(5)     ✗   stathat.com/c/consistent at master has problem subpkg(s):
(5)             stathat.com/c/consistent has err (*pkgtree.LocalImportsError); required by (root).
(4)       ← no more versions of stathat.com/c/consistent to try; begin backtrack
(3)     ← backtrack: no more versions of golang.org/x/sys to try
(2)     ← backtrack: no more versions of gopkg.in/yaml.v2 to try
(1)     ← backtrack: no more versions of github.com/golang/protobuf to try
  ✗ solving failed

Solver wall times by segment:
     b-list-versions: 7.651384284s
         b-list-pkgs: 748.013299ms
              b-gmal: 480.572702ms
             satisfy:   6.358833ms
         select-root:   4.396338ms
            unselect:    571.824µs
         select-atom:    454.343µs
            new-atom:      359.7µs
           backtrack:     26.956µs
  b-deduce-proj-root:      10.74µs
               other:      9.173µs
     b-source-exists:       7.85µs

  TOTAL: 8.892166042s

No versions of stathat.com/c/consistent met constraints:
        master: Could not introduce stathat.com/c/consistent@master, as its subpackage stathat.com/c/consistent does not contain usable Go code (*pkgtree.LocalImportsError).. (Package is required by (root).)

What did you expect to see?

¯_(ツ)_/¯

What did you see instead?

No versions of stathat.com/c/consistent met constraints:
        master: Could not introduce stathat.com/c/consistent@master, as its subpackage stathat.com/c/consistent does not contain usable Go code (*pkgtree.LocalImportsError).. (Package is required by (root).)

[rhwlo]

It looks like this may be stemming from something similar to #705: in this case, https://github.com/stathat/consistent/blob/master/example_test.go has a relative import that could be the culprit.

[sdboyer]

hi hi! awesome that you're giving dep a shot with veneur 😄

so, tl;dr - if you can change this to import "." or import "stathat.com/c/consistent", either of which i believe will work, your problem should go away.

longer explanation is that you're hitting a few interrelated things, and it's being compounded by us needing to improve our failure feedback - very much a known pain point, but difficult, and something i haven't yet found time for. these issues are:

1. dep simply doesn't allow relative imports. this is one of the few places where we restrict a case that the toolchain itself allows. we disallow them only because
   1. the toolchain already frowns heavily on them
   2. it's worse for our case, as we start venturing into dot dot hell territory when trying to prove that the import does not escape the tree of the project
2. we have a spot to handle import comments, but we don't actually do it yet. (i'm gonna open a follow-up on this, it's a good help-wanted, possibly even good-first-pr for someone)
3. this particular case is interesting, because we should be able to deduce from the combination of import comment (staticcheck.com/c/consistent) and the import statement (../consistent) that we're necessarily within the tree...
4. but neither 2 nor 3 should actually matter, as that relative import is in a test file, and thus we should ignore it for the purposes of determining importability. but, we don't separate test imports from non-test imports in for these purposes in the way we should. (upspin revealed another way that we did this wrong - Avoid erroneous import cycle loop detection in ToReachMap() arising from tests #420. that one's much trickier, though, and i still haven't had a chance to fix it.)

idk if you have access to fix consistent yourself, but the fourth item there should be reasonably solvable. let's consider fixing that the goal for this particular issue. 😄

[carolynvs]

Should we document in the FAQ that relative imports aren't supported?

[sdboyer]

@carolynvs yes, probably worth doing 😄

[ChimeraCoder]

Aha - that would do it! Thanks @sdboyer. We don't have write access to consistent, but I've opened a patch, and worst-case scenario, we could just operate off of a forked or modified version in the meantime.

Regarding (2), I could probably take a stab at that. Feel free to assign me the ticket when you write it up.

[sdboyer]

@ChimeraCoder cool cool - #902 is that ticket. (GH limits first-class ticket assignment to org members, unfortunately 😞)

Reflecting on the proper fix for this a little more, it may end up getting into all the same issues as #420, which ultimately amount to the tangle of #291 that I've been deferring. So, removing the help-wanted tag and assigning to me, as that's a morass I need to wade through.

[carolynvs]

Added #910 for the FAQ entry.

[jlovero-isp]

I see this issue is still open, so I hope it's okay to ask a related question. I opened an issue with the go-git repo and linked this issue, as you can see above.

I've searched through the go-git code for relative imports and I cannot find any. Is there another reason besides relative imports why one might get the Could not introduce package, as its subpackage does not contain usable Go code error when attempting to add a package?

If you would like me to open a separate issue for my question, I'm happy to do so.

Thanks.

[sdboyer]

@jlovero-isp yeah, if you could open a separate issue for that, that'd be ideal. thanks!

[mcandre]

Hello, I actually get the same does not contain usable Go code error from dep when attempting to resolve Go's x tools. Here's my Gopkg.toml:

required = ["golang.org/x/tools"]

[[constraint]]
  name = "golang.org/x/tools"
  branch = "release-branch.go1.9"

[prune]
  go-tests = true
  unused-packages = true

I get the same error behavior for golang.org/x/crypto as well.

[rsc]

@mcandre, @sdboyer explained to me that constraint.name is a project (repository root, which may not be an actual import path) but required is a list of packages (project directories with Go code; actual import paths). Dep should probably diagnose this better but the workaround for now is to put a real package import path in required.

[sdboyer]

@mcandre yep, this can definitely be confusing (and we're working on some changes that may help to clear up the semantics, but as our docs indicate, namerefers to the project/source root , while required refers to an actual package.