Support for subpath

[tnozicka]

Inheriting package from dependency's vendor

As a user I am using the same package as one of my dependencies which vendors that package. That is rightfully resulting in a conflict which can be usually solved by pinning that package to a specific version that works for both. The fun comes when the dependency modifies files in their vendor so there is no repo with such commit id to pin. dep needs to have a way to allow user to specify to use the package directly from vendor folder of that dependency. Or allow something like a subpath with repo URLs.

I know it might be hard to understand so here is a concrete example on public and very large Go project:
OpenShift vendors k8s.io/kubernetes and also carries patches for Kubernetes (in vendor/k8s.io/kubernetes). My project using OpenShift uses also Kubernetes but I have no way of specifying version of common dependency (k8s.io/kubernetes) because there is no repo having the correct files except a subpath github.com/openshift/origin/vendor/k8s.io/kubernetes

If you will agree that this is a valid use case the implementations might be as simple as having a syntax for it in Gopkg.toml and creating a symlink in vendor.

I would create the symlink myself but dep overrides vendor/ every time and it might also interfere with other parts of the tool. And it won't also create vendor/ until all the dependencies succeed.

Staging

Staging is fairly similar to what's described above. And should be solved by something like subpath. I will go right for an example:

In Kubernetes there is a folder staging/
This directory is the staging area for packages that have been split to their own repository. The content here will be periodically published to respective top-level k8s.io repositories.
I would like to be able to specify my dependencies from there; say my include k8s.io/apimachinery/pkg/... should point to github.com/kubernetes/kubernetes/staging/src/k8s.io/apimachinery/pkg/...

[tnozicka]

It might be better to move that package from the nested vendor to the top level instead of a symlink to avoid having that object in two binary versions (meaning the types won't be the same) - see https://github.com/Masterminds/glide/blame/245caced2b16358b1c5e267691b17e9ee9952127/docs/vendor.md#L23-L26
and https://github.com/mattfarina/golang-broken-vendor/blame/6f31003510e9528f0b9b468ac75f6b084ad3a201/Readme.md#L30

[sdboyer]

so, i get the motivation here, i do - but unfortunately, this is a non-starter. there are several (long-settled) design rules that doing this would violate:

1. the unit of dependency is the repo/project.
2. vendor/ is disposable, containing nothing unique or valuable.

without these rules, dep would pretty much fall apart.

it's unfortunate that OpenShift isn't following the second rule, maintaining custom patches solely in their vendor tree. they run the risk of just being generally un-importable, for anyone, because that vendor dir must be preserved, which produces issues like this. i know that we'd be happy to help them work towards converting to a structure that's compliant with these rules.

wrt k8s, their staging directory is not intended for public import, as i understand it. we're working with them directly to hammer these things out, but it's possible that those directories may be moved beneath an internal dir, codifying that intention. if you're relying on them, they should be imported from the canonical, separate repository instead.

[tnozicka]

@sdboyer

I give up on having patches in vendor thing and I have actually sent a PR to move it to staging as Kubernetes and symlink it.

But let's talk about staging. Why would supporting subpath be bad? The unit of dependency is still a repository you just don't use it all. It also seems like it could be easily implemented by creating a symlink in vendor directory.

The issue with the separate repo is that it's hard to find corresponding revisions across tens of repositories when you want some particular commit of the main repository and all corresponding repositories at the same and tested revision.

[sdboyer]

The issue with the separate repo is that it's hard to find corresponding revisions across tens of repositories when you want some particular commit of the main repository and all corresponding repositories at the same and tested revision.

of course! this is why monorepo-style projects are fundamentally useful in the first place. the problem is trying to have your cake and eat it too - wanting to have an internal part of the project masquerade as an external part.

But let's talk about staging. Why would supporting subpath be bad? The unit of dependency is still a repository you just don't use it all. It also seems like it could be easily implemented by creating a symlink in vendor directory.

heh, you're asking the same questions here as @sttts in #1000 and #638 😄

maybe. and i'm open to being convinced - but any argument that convinces me will have to focus on how we establish boundaries around it. the happy path isn't enough - i need to be sure that we understand the consequences. so, lemme jot down some thoughts about how i can see this panning out.

i can imagine it working decently well for the repository in which this substitution happens, but it gets trickier for the dependers on that repository. say that we let k8s.io/kubernetes redefine k8s.io/apimachinery as living as k8s.io/kubernetes/staging/apimachinery. when k8s.io/kubernetes is the current project, this is easy enough to deal with - symlink vendor/k8s.io/apimachinery to ../../../staging/apimachinary.

when the current project is a depender on k8s.io/kubernetes that does not, itself, import k8s.io/apimachinery, it's a tad more complicated, but not horribly so - we symlink vendor/k8s.io/apimachinery to ../kubernetes/staging/apimachinary.

but, when the depender also directly or transitively imports k8s.io/apimachinery through some path that does not run through k8s.io/kubernetes, we have to decide how to deal that. conflicts on source fields elsewhere cause solving to fail, and i don't see a way that could be different here. this suggests that, if something imports k8s.io/apimachinery, and also k8s.io/kubernetes, it will simply have to use k8s.io/kubernetes's version of k8s.io/apimachinery in staging, otherwise solving will fail. at a certain point, it starts being kinda pointless (from a dep mgmt perspective) to have k8s.io/apimachinery exist as a standalone repository at all. and it seems rather fraught with gotchas for users.

still, thinking through this a bit has helped me here, actually - the idea of an "alternate root" is something that i've already been ruminating on in the context of monorepos, and a two-birds-one-stone solution is starting to form in my mind.