On Wed, 2 Jan 2019 at 22:51, Dan Rue dan.rue@linaro.org wrote:
To follow up on my original post which I guess didn't garner much attention, here's a working docker compose config which uses the latest production lava containers, and launches a single qemu worker.
https://github.com/danrue/lava.home.therub.org/tree/b17ed0f8a660f9a8c31f892d...
Running 'docker-compose up' then brings up a postgres container, a lava master container, and a dispatcher container. A few settings files are mounted into the master container, as well as devices and health-checks. The dispatcher itself must be run in privileged mode in order to use (at least) /dev/net/tun.
Is a tun device absolutely essential? It would be much better to not need privileged mode at all. In the Harston lab, we only really need tun to be able to support hacking sessions. netdevice: user is enough to get network usage within QEMU working.
Once up, run something like the following to add an admin user: docker-compose exec server lava-server manage users add --staff --superuser --email dan.rue@linaro.org --passwd foo drue
Then, you may go into the admin interface to add the devices.
Adding devices can be scripted using lava-server manage.
The only real gap that I can see is that the docker containers have the architecture hard coded in the container name. This should use a manifest[1] instead so that running docker on ARM "just works". A lot of work has been done to make architecture transparent when running docker (for the benefit of ARM users like us) - we should use it.
Next step - I spoke with Matt Hart about the idea of having lava be more ephemeral. If you don't actually need your historical runs, you could have lava re-configure itself every time on start-up, based on the contents of devices/ and health-checks/. I'm not sure the best way to do that - perhaps as an additional setup script that's supported by entrypoint.sh, or perhaps as an outside thing using lavacli. Or perhaps it can be done with docker exec and lava-server manage...
Thanks! Dan
[1] https://docs.docker.com/edge/engine/reference/commandline/manifest/
On Mon, Dec 10, 2018 at 04:58:23PM -0600, Dan Rue wrote:
I started playing with the official lava images today and wanted to share my work in progress, in case others are doing something similar or have feedback. My goal is to deploy a lava lab locally. My architecture is a single host (for now) that will host both the lava server and one dispatcher. Once it's all working, I'll start deploying a qemu worker followed by some actual boards (hopefully).
So far, I have the following docker-compose.yml:
version: '3' services: database: image: postgres:9.6 environment: POSTGRES_USER: lavaserver POSTGRES_PASSWORD: mysecretpassword PGDATA: /var/lib/postgresql/data/pgdata volumes: - ${PWD}/pgdata:/var/lib/postgresql/data/pgdata server: image: lavasoftware/amd64-lava-server:2018.11 ports: - 80:80 volumes: - ${PWD}/etc/lava-server/settings.conf:/etc/lava-server/settings.conf - ${PWD}/etc/lava-server/instance.conf:/etc/lava-server/instance.conf depends_on: - database dispatcher: image: lavasoftware/amd64-lava-dispatcher:2018.11 environment: - "DISPATCHER_HOSTNAME=--hostname=dispatcher.lava.therub.org" - "LOGGER_URL=tcp://server:5555" - "MASTER_URL=tcp://server:5556"
With that file, settings.conf, and instance.conf in place, I run 'mkdir pgdata; docker-compose up' and the 3 containers come up and start talking to each other. The only thing exposed to the outside world is lava-server's port 80 at the host's IP, which gives the lava homepage as expected. The first time they come up, the database isn't up fast enough (it has to initialize the first time) and lava-server fails to connect. If you cancel and run again it will connect the second time.
A few things to note here. First, it doesn't seem like a persistent DB volume is possible with the existing lava-server container, because the DB is initialized at container build time rather than run-time, so there's not really a way to mount in a volume for the data. Anyway, postgres already solves this. In fact, I found their container documentation and entrypoint interface to be well done, so it may be a nice model to follow: https://hub.docker.com/_/postgres/
The server mostly works as listed above. I copied settings.conf and instance.conf out of the original container and into ./etc/lava-server/ and modified as needed.
The dispatcher then runs and points to the server.
It's notable that docker-compose by default sets up a docker network, allowing references to "database", "server", "dispatcher" to resolve within the containers.
Once up, I ran the following to create my superuser:
docker-compose exec server lava-server manage users add --staff --superuser --email dan.rue@linaro.org --passwd foo drueNow, for things I've run into and surprises:
- When I used a local database, I could log in. With the database in a separate container, I can't. Not sure why yet.
- I have the dreaded CSRF problem, which is unlikely to be related to docker, but the two vars in settings.conf didn't seem to help. (I'm terminating https outside of the container context, and then proxying into the container over http)
- I was surprised there were no :latest containers published
- I was surprised the containers were renamed to include the architecture name was in the container name. My understanding is that's the 'old' way to do it. The better way is to transparently detect arch using manifests. Again, see postgres/ as an example.
- my pgdata/ directory gets chown'd when I run postgres container. I see the container has some support for running under a different uid, which I might try.
- If the entrypoint of server supported some variables like LAVA_DB_PASSWORD, LAVA_DB_SERVER, SESSION_COOKIE_SECURE, etc, I wouldn't need to mount in things like instance.conf, settings.conf.
I pushed my config used here to https://github.com/danrue/lava.home.therub.org. Git clone and then run 'docker-compose up' should just work.
Anyway, thanks for the official images! They're a great start and will hopefully really simplify deploying lava. My next step is to debug some of the issues I mentioned above, and then start looking at dispatcher config (hopefully it's just a local volume mount).
Dan
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users