Re: [Lava-users] docker-compose with lava

I've continued to work on using the official docker containers with docker-compose.

If you have docker and docker-compose installed, you can clone https://github.com/danrue/lava-docker-compose and run "make". You should end up with LAVA running at http://localhost, with a qemu worker and qemu device which will run a successful health-check automatically.

I tried to keep everything as simple and as upstream native as possible. As the containers improve, this example repository will simplify. For example, currently it builds a new lava-server container (using upstream lava-server as a base) to work around one bug that's fixed but not yet released, and to add the ability to do the initial provisioning.

The dispatcher container runs directly from the released container.

There are two additional containers in use. An official postgres container is used for the lava-server database, which has a nice interface and semantics as you would expect. Lastly, an nginx container is included to serve health-check images. The initial image files are retrieved with a Makefile rule.

In a separate branch (named beaglebone-black), I've started adding support for beaglebone-black. As expected, this required a lot more work in the dispatcher container, and it is still not quite working. It is also lab-specific.

My goal here is to try to develop a reference implementation of deploying the official LAVA docker containers to help people get started quickly and without adding any additional layers of complexity/abstraction (there are enough already!)

Dan

On Wed, Jan 02, 2019 at 04:51:25PM -0600, Dan Rue wrote:

To follow up on my original post which I guess didn't garner much attention, here's a working docker compose config which uses the latest production lava containers, and launches a single qemu worker.

https://github.com/danrue/lava.home.therub.org/tree/b17ed0f8a660f9a8c31f892d...

Running 'docker-compose up' then brings up a postgres container, a lava master container, and a dispatcher container. A few settings files are mounted into the master container, as well as devices and health-checks. The dispatcher itself must be run in privileged mode in order to use (at least) /dev/net/tun.

Once up, run something like the following to add an admin user: docker-compose exec server lava-server manage users add --staff --superuser --email dan.rue@linaro.org --passwd foo drue

Then, you may go into the admin interface to add the devices.

The only real gap that I can see is that the docker containers have the architecture hard coded in the container name. This should use a manifest[1] instead so that running docker on ARM "just works". A lot of work has been done to make architecture transparent when running docker (for the benefit of ARM users like us) - we should use it.

Next step - I spoke with Matt Hart about the idea of having lava be more ephemeral. If you don't actually need your historical runs, you could have lava re-configure itself every time on start-up, based on the contents of devices/ and health-checks/. I'm not sure the best way to do that - perhaps as an additional setup script that's supported by entrypoint.sh, or perhaps as an outside thing using lavacli. Or perhaps it can be done with docker exec and lava-server manage...

Thanks! Dan

[1] https://docs.docker.com/edge/engine/reference/commandline/manifest/

On Mon, Dec 10, 2018 at 04:58:23PM -0600, Dan Rue wrote:

...

I started playing with the official lava images today and wanted to share my work in progress, in case others are doing something similar or have feedback. My goal is to deploy a lava lab locally. My architecture is a single host (for now) that will host both the lava server and one dispatcher. Once it's all working, I'll start deploying a qemu worker followed by some actual boards (hopefully).

So far, I have the following docker-compose.yml:

version: '3' services: database: image: postgres:9.6 environment: POSTGRES_USER: lavaserver POSTGRES_PASSWORD: mysecretpassword PGDATA: /var/lib/postgresql/data/pgdata volumes: - ${PWD}/pgdata:/var/lib/postgresql/data/pgdata server: image: lavasoftware/amd64-lava-server:2018.11 ports: - 80:80 volumes: - ${PWD}/etc/lava-server/settings.conf:/etc/lava-server/settings.conf - ${PWD}/etc/lava-server/instance.conf:/etc/lava-server/instance.conf depends_on: - database dispatcher: image: lavasoftware/amd64-lava-dispatcher:2018.11 environment: - "DISPATCHER_HOSTNAME=--hostname=dispatcher.lava.therub.org" - "LOGGER_URL=tcp://server:5555" - "MASTER_URL=tcp://server:5556"

With that file, settings.conf, and instance.conf in place, I run 'mkdir pgdata; docker-compose up' and the 3 containers come up and start talking to each other. The only thing exposed to the outside world is lava-server's port 80 at the host's IP, which gives the lava homepage as expected. The first time they come up, the database isn't up fast enough (it has to initialize the first time) and lava-server fails to connect. If you cancel and run again it will connect the second time.

A few things to note here. First, it doesn't seem like a persistent DB volume is possible with the existing lava-server container, because the DB is initialized at container build time rather than run-time, so there's not really a way to mount in a volume for the data. Anyway, postgres already solves this. In fact, I found their container documentation and entrypoint interface to be well done, so it may be a nice model to follow: https://hub.docker.com/_/postgres/

The server mostly works as listed above. I copied settings.conf and instance.conf out of the original container and into ./etc/lava-server/ and modified as needed.

The dispatcher then runs and points to the server.

It's notable that docker-compose by default sets up a docker network, allowing references to "database", "server", "dispatcher" to resolve within the containers.

Once up, I ran the following to create my superuser:

docker-compose exec server lava-server manage users add --staff --superuser --email dan.rue@linaro.org --passwd foo drue

Now, for things I've run into and surprises:

• When I used a local database, I could log in. With the database in a separate container, I can't. Not sure why yet.
• I have the dreaded CSRF problem, which is unlikely to be related to docker, but the two vars in settings.conf didn't seem to help. (I'm terminating https outside of the container context, and then proxying into the container over http)
• I was surprised there were no :latest containers published
• I was surprised the containers were renamed to include the architecture name was in the container name. My understanding is that's the 'old' way to do it. The better way is to transparently detect arch using manifests. Again, see postgres/ as an example.
• my pgdata/ directory gets chown'd when I run postgres container. I see the container has some support for running under a different uid, which I might try.
• If the entrypoint of server supported some variables like LAVA_DB_PASSWORD, LAVA_DB_SERVER, SESSION_COOKIE_SECURE, etc, I wouldn't need to mount in things like instance.conf, settings.conf.

I pushed my config used here to https://github.com/danrue/lava.home.therub.org. Git clone and then run 'docker-compose up' should just work.

Anyway, thanks for the official images! They're a great start and will hopefully really simplify deploying lava. My next step is to debug some of the issues I mentioned above, and then start looking at dispatcher config (hopefully it's just a local volume mount).

Dan