[Request] Include Release & Docker build for Raspberry Pi (ARMv6/7 Architecture)

[shbatm]

Not a bug. This is a general feature request to include an Arm version of the Docker build (via Travis-CI).

I was able to build the image myself successfully using the steps in this Gist (leaving here for anyone else who might want to reference), but it would be nice to have a pre-built Docker image for running on a Raspberry Pi.

Thanks!

[JoelSpeed]

Totally in agreement, we should add this. If anyone wants to pick this up feel free!

[kfox1111]

v8 too please. I've switched almost entirely to aarch64.

[Sheshagiri]

I'm in for this. Let me know.

[karlskewes]

I've added the armv6 and arm64 build lines to the Makefile and the resulting binaries work on the different platforms (I have RPi and Rock64 SBC's) (at least oauth2_proxy --version).

Building arm or arm64 images on an amd64 host is slightly more complicated as the build machine needs to have qemu-user-static to cross compile if we follow someone like Flannel's build process.
Then we need to extend the Makefile and Dockerfile's (Dockerfile.{amd64|armv6|arm64}) out with all this.

An alternative is GoReleaser which seems to produce multiple docker images for each arch easily and should require the same build steps no matter what arch the building host is. goreleaser --snapshot --rm-dist or similar.
I've just recently used GoReleaser for something else.

I'd like to run oauth2_proxy on my multi-arch kubernetes cluster so I can have a go at a PR depending on what way we want to go.

[JoelSpeed]

I'd personally be tempted to go for the Flannel approach and have separate Dockerfiles for the arm builds.

I'd be tempted to leave the amd64 Dockerfile just as Dockerfile as this is likely to be the most frequently used Dockerfile, do we think that would confuse people?

What was the reason for choosing armv6? I understand arm versions have some backwards compatibility but I'm not an arm user myself. I notice that Kubernetes use armv7 for their arm releases

[karlskewes]

I couldn't comment whether it's better to build for v7 or v6.
Golang download has armv6 but not armv7 - https://golang.org/dl/

Sounds good to leave amd64 Dockerfile as is.
If we end up being able to do multi-arch docker images then we can duplicate the Dockerfile with extension at that point.
I can PR the binary Makefile changes as that is straight forward?
Docker as stage 2?

Docker image building:

1. Building by users "on arm" - docker build -f Dockerfile.arm64 .
   Our current Dockerfile relies on golang/dep however that project doesn't build arm/arm64 binaries - dep releases.
   There is this issue for adding support for arm.
   It looks like it might be possible to extend to arm with a one liner change in dep's Makefile
   Investigating this now.
   I've built dep binary for arm and am going through seeing if I can then build oauth2_proxy with it.
   It may dictate whether we build armv6 or armv7.

2. Building by users "on amd64" or for releases.
   This is where the qemu process will come in.

[kfox1111]

As far as I know, only the rpi zero and origional rpi are armv6 and pretty much everything else is v7 these days.

[karlskewes]

Thanks for merging binaries!

Do we want to keep this issue open for tracking the second part; arm docker images?
I'm tempted to wait and see what happens with the dep PR linked above before doing any more work with Dockerfiles though.

[JoelSpeed]

So I don't think the dep thing is a problem.

The Dockerfiles we want in the repo are for building releases and pushing to Quay right? People won't be building Dockerfiles on their arm machines

So given we use multi stage builds, we can make our first stage of that amd64 (as it is now) and then just copy the arm binary into an arm based base image for distribution

Have I missed something?

[karlskewes]

Okay.
Yeah I agree it's probably a bit of an oddball case to build docker images on arm platform. Most people will have access to amd64 to build if they want/need.
Building on arm is only blocked by my dep PR which is a minor patch that a motivated person could apply in any case.
Leaving this out of scope for now might be an option?
Anyone else in this thread wish to comment?

Regarding your suggestion about copying in an arm binary, I tried just that by modifying a Dockerfile and building on my amd64 laptop however the end Docker image ends up showing arch: amd64 in the manifest when you inspect it.

$ docker inspect bitnami-labs/sealed-secrets-controller:arm64-latest | grep Arch
        "Architecture": "amd64",

If try setting the second image to FROM arm32v6/alpine:3.8 then I think the apk add stage or general container start fails.
The failure message is not very helpful but web search suggests it's an ELF issue.
This led me to dep and qemu and contrasts with my experience with GoReleaser which magically creates all the different arch images somehow.

So.... I will have a go at the qemu stuff to get us building arm docker images on amd64. :)

[kfox1111]

Have a look at this pr that the helm folks are working on. I think its based on how k8s cross compiles and builds the images:
helm/helm#5039

It might be helpful.

[karlskewes]

Thanks for the link, matches my recent learnings. Similar approach to flannel. Will have a go at it soon.

[JoelSpeed]

I think this can be closed by #39, can those interested double check?