Requires root access, or the root-equivalent 'docker' group

[ids1024]

It seems like tensorman is expected to be run by a user in the docker group.

This group is considered fairly problematic if handy, and isn't supported on Red Hat due to security concerns, supposing this post is still accurate. The problem is that being a member of the docker group can be used to gain root access to the host system.

If I'm not mistaken, this makes tensorman unsuitable for use in applications where a non-administrator user should be able to use TensorFlow, which presumably is desirable for some organizations. Even for a user in sudoers, having a means to acquire root access without entering a password is problematic.

We inherit this issue from Docker, but if the goal is to wrap it and make it easier to work with, it would be good to address this. I see a few possible solutions:

• Do nothing, but make sure to document this potential concern clearly
• A daemon that runs as a user in the docker group, separate from the CLI
• Instead of a daemon, an SUID binary
• Use Docker's experimental rootless mode

The daemon or SUID binary would be a pain to deal with, and raise their own challenges to keep secure. Rootless mode is very attractive, and would allow everything to run under the user's account, just like Rustup. I'm not sure if the Tensorflow images would run well without issue it, since is is experimental and somewhat limited.

[ids1024]

Another solution, which seems attractive at first glance: Podman. It runs without a daemon and prioritizes support for rootless mode, while being compatible with docker images and providing a docker-compatible CLI. So if it works well with the official tensorflow images, it should allow for an experience closes to Rustup, without any security concerns.

Plus, the name matches nicely with "Tensorman".

It's in the repos for Fedora and CentOS, but not Debian/Ubuntu. So we would have to package it.

Edit: For example, you can run podman run -it tensorflow/tensorflow:latest python.

[ids1024]

It looks like Podman is included in the Universe repository for Ubuntu 20.10 Groovy, so perhaps if it works without issue, tensorman can use Podman in Pop!_OS 20.10.

Requirements for this:

• Call the podman command instead of docker. In principle, nothing more is needed, since Podman aims to provide a compatible CLI.
• Perhaps have an option to use docker instead. I'd probably provide at least an undocumented command-line argument for setting the docker/podman command name to use.
• Test to make sure everything works as expected when running in Podman. (I assume GPU acceleration in a container wouldn't require root?)
• Images that have been installed system-wide by Docker won't be found by Podman. Should we have a command for migrating? I suppose it would be easy to retrieve a list of installed images from Docker, and have Podman download the same ones.

[mmstick]

It may be worth it to backport podman to 20.04 because we'll be supporting 20.04 for quite a while.

[ids1024]

Tensorman can use podman in place of docker using #24 with --docker-cmd podman. Except for GPU support (which is rather essential).

• Tensorman passes --gpus=all, which doesn't work in podman, and passes it results in an error. Just not passing this when using Podman solves this.
• nvidia-container-toolkit has an OCI container hook, which allows accessing GPUs in podman, including rootless podman containers.
  • But without changing the configuration, loading the hook causes rootless podman to crash. And with the configuration change, rootful podman can't access Nvidia GPUs (Running nvidia-container-runtime with podman is blowing up. NVIDIA/nvidia-container-runtime#85)