On Mon, Jan 28, 2019 at 09:24:26AM +0000, Milosz Wasilewski wrote:
On Fri, 25 Jan 2019 at 23:00, Dan Rue dan.rue@linaro.org wrote:
More updates! Described below, referenced links go to source:
- beaglebone-black is now working for me [1]
- ser2net containerized [2]
- LAVA upgrade process is documented [3]
- Squid container added; nginx images hack removed [4]
The beaglebone-black branch represents what's now an actual working docker-compose environment for my bbb, using a recent u-boot (this turned out to be the hardest part - totally unrelated to docker). I ended up running NFS and TFTP on the host and mounting the paths into the dispatcher. I'd like to containerize those still, but NFS is a bit difficult in particular and I just wanted to see things work.
The beaglebone-black branch is back to using the dispatcher without rebuilding it. I did this by breaking ser2net into its own container that can be found at danrue/ser2net and used as follows:
version: '3.4' services: ser2net: image: danrue/ser2net:3.5 volumes: - ./ser2net/ser2net.conf:/etc/ser2net.conf devices: - /dev/serial/by-id/usb-Silicon_Labs_CP2102_USB_to_UART_Bridge_Controller_0001-if00-port0The best part is running something like this to spy on the serial port during testing:
docker-compose exec dispatcher telnet ser2net 5001The LAVA upgrade has been documented in the README, but it's simple enough I'll reproduce it here:
1. Stop containers. 2. Back up pgsql from its docker volume sudo tar cvzf lava-server-pgdata-$(date +%Y%m%d).tgz /var/lib/docker/volumes/lava-server-pgdata 3. Change e.g. `lavasoftware/amd64-lava-server:2018.11` to `lavasoftware/amd64-lava-server:2019.01` andIs the content of /var/lib/lava-server/default/media/job-output/ also preserved in this scenario? If not, this dir should also probably be mapped into a volume so it is moved between migrated versions.
Oh, good catch. Fixed with a docker volume @ https://github.com/danrue/lava-docker-compose/blob/master/docker-compose.yml...
Dan
milosz
`lavasoftware/amd64-lava-dispatcher:2018.11` to `lavasoftware/amd64-lava-dispatcher:2019.01` in docker-compose.yml. 4. Change the FROM line if any containers are being rebuilt, such as ./server-docker/Dockerfile 5. Start containers.Please note the implication there. Since the containers have no stateful data, we don't care about 'upgrading' their contents. We can just stop the old ones and start the new ones. A downgrade is simple: restore the DB and start the old containers. One issue that I ran into (where I use https) is the following error:
lava_server | ERROR 2019-01-25 21:11:08,479 exception Invalid HTTP_HOST header: 'lava.therub.org'. You may need to add 'lava.therub.org'I added the following line to /etc/lava-server/settings.conf to fix:
"ALLOWED_HOSTS": ["lava.therub.org", "127.0.0.1"],Other than that, no changes were necessary to use 2019.01 LAVA!
Finally, I removed the hacky 'images' container that was hosting health check images behind nginx, and replaced it with a general purpose squid container, backed by a docker volume. Note that in a real production type environment, both approaches might be necessary. I do like the idea of health check images being statically hosted as they are in validation.l.o.
The sources can be found at https://github.com/danrue/lava-docker-compose/, and I also "tweeted" about some of what I've learned at https://twitter.com/mndrue/status/1088627889426350080.
Thanks again for reading this far! Dan
[1] https://github.com/danrue/lava-docker-compose/tree/beaglebone-black [2] https://github.com/danrue/lava-docker-compose/commit/af1f62b22dbfd60757ec8a7... [3] https://github.com/danrue/lava-docker-compose/commit/76d69783a3ea7ace6d85538... [4] https://github.com/danrue/lava-docker-compose/commit/bb37d9cb42d490b61c9c7ff...
On Fri, Jan 11, 2019 at 03:38:01PM -0600, Dan Rue wrote:
I've continued to work on using the official docker containers with docker-compose.
If you have docker and docker-compose installed, you can clone https://github.com/danrue/lava-docker-compose and run "make". You should end up with LAVA running at http://localhost, with a qemu worker and qemu device which will run a successful health-check automatically.
I tried to keep everything as simple and as upstream native as possible. As the containers improve, this example repository will simplify. For example, currently it builds a new lava-server container (using upstream lava-server as a base) to work around one bug that's fixed but not yet released, and to add the ability to do the initial provisioning.
The dispatcher container runs directly from the released container.
There are two additional containers in use. An official postgres container is used for the lava-server database, which has a nice interface and semantics as you would expect. Lastly, an nginx container is included to serve health-check images. The initial image files are retrieved with a Makefile rule.
In a separate branch (named beaglebone-black), I've started adding support for beaglebone-black. As expected, this required a lot more work in the dispatcher container, and it is still not quite working. It is also lab-specific.
My goal here is to try to develop a reference implementation of deploying the official LAVA docker containers to help people get started quickly and without adding any additional layers of complexity/abstraction (there are enough already!)
Dan
On Wed, Jan 02, 2019 at 04:51:25PM -0600, Dan Rue wrote:
To follow up on my original post which I guess didn't garner much attention, here's a working docker compose config which uses the latest production lava containers, and launches a single qemu worker.
https://github.com/danrue/lava.home.therub.org/tree/b17ed0f8a660f9a8c31f892d...
Running 'docker-compose up' then brings up a postgres container, a lava master container, and a dispatcher container. A few settings files are mounted into the master container, as well as devices and health-checks. The dispatcher itself must be run in privileged mode in order to use (at least) /dev/net/tun.
Once up, run something like the following to add an admin user: docker-compose exec server lava-server manage users add --staff --superuser --email dan.rue@linaro.org --passwd foo drue
Then, you may go into the admin interface to add the devices.
The only real gap that I can see is that the docker containers have the architecture hard coded in the container name. This should use a manifest[1] instead so that running docker on ARM "just works". A lot of work has been done to make architecture transparent when running docker (for the benefit of ARM users like us) - we should use it.
Next step - I spoke with Matt Hart about the idea of having lava be more ephemeral. If you don't actually need your historical runs, you could have lava re-configure itself every time on start-up, based on the contents of devices/ and health-checks/. I'm not sure the best way to do that - perhaps as an additional setup script that's supported by entrypoint.sh, or perhaps as an outside thing using lavacli. Or perhaps it can be done with docker exec and lava-server manage...
Thanks! Dan
[1] https://docs.docker.com/edge/engine/reference/commandline/manifest/
On Mon, Dec 10, 2018 at 04:58:23PM -0600, Dan Rue wrote:
I started playing with the official lava images today and wanted to share my work in progress, in case others are doing something similar or have feedback. My goal is to deploy a lava lab locally. My architecture is a single host (for now) that will host both the lava server and one dispatcher. Once it's all working, I'll start deploying a qemu worker followed by some actual boards (hopefully).
So far, I have the following docker-compose.yml:
version: '3' services: database: image: postgres:9.6 environment: POSTGRES_USER: lavaserver POSTGRES_PASSWORD: mysecretpassword PGDATA: /var/lib/postgresql/data/pgdata volumes: - ${PWD}/pgdata:/var/lib/postgresql/data/pgdata server: image: lavasoftware/amd64-lava-server:2018.11 ports: - 80:80 volumes: - ${PWD}/etc/lava-server/settings.conf:/etc/lava-server/settings.conf - ${PWD}/etc/lava-server/instance.conf:/etc/lava-server/instance.conf depends_on: - database dispatcher: image: lavasoftware/amd64-lava-dispatcher:2018.11 environment: - "DISPATCHER_HOSTNAME=--hostname=dispatcher.lava.therub.org" - "LOGGER_URL=tcp://server:5555" - "MASTER_URL=tcp://server:5556"
With that file, settings.conf, and instance.conf in place, I run 'mkdir pgdata; docker-compose up' and the 3 containers come up and start talking to each other. The only thing exposed to the outside world is lava-server's port 80 at the host's IP, which gives the lava homepage as expected. The first time they come up, the database isn't up fast enough (it has to initialize the first time) and lava-server fails to connect. If you cancel and run again it will connect the second time.
A few things to note here. First, it doesn't seem like a persistent DB volume is possible with the existing lava-server container, because the DB is initialized at container build time rather than run-time, so there's not really a way to mount in a volume for the data. Anyway, postgres already solves this. In fact, I found their container documentation and entrypoint interface to be well done, so it may be a nice model to follow: https://hub.docker.com/_/postgres/
The server mostly works as listed above. I copied settings.conf and instance.conf out of the original container and into ./etc/lava-server/ and modified as needed.
The dispatcher then runs and points to the server.
It's notable that docker-compose by default sets up a docker network, allowing references to "database", "server", "dispatcher" to resolve within the containers.
Once up, I ran the following to create my superuser:
docker-compose exec server lava-server manage users add --staff --superuser --email dan.rue@linaro.org --passwd foo drueNow, for things I've run into and surprises:
- When I used a local database, I could log in. With the database in a separate container, I can't. Not sure why yet.
- I have the dreaded CSRF problem, which is unlikely to be related to docker, but the two vars in settings.conf didn't seem to help. (I'm terminating https outside of the container context, and then proxying into the container over http)
- I was surprised there were no :latest containers published
- I was surprised the containers were renamed to include the architecture name was in the container name. My understanding is that's the 'old' way to do it. The better way is to transparently detect arch using manifests. Again, see postgres/ as an example.
- my pgdata/ directory gets chown'd when I run postgres container. I see the container has some support for running under a different uid, which I might try.
- If the entrypoint of server supported some variables like LAVA_DB_PASSWORD, LAVA_DB_SERVER, SESSION_COOKIE_SECURE, etc, I wouldn't need to mount in things like instance.conf, settings.conf.
I pushed my config used here to https://github.com/danrue/lava.home.therub.org. Git clone and then run 'docker-compose up' should just work.
Anyway, thanks for the official images! They're a great start and will hopefully really simplify deploying lava. My next step is to debug some of the issues I mentioned above, and then start looking at dispatcher config (hopefully it's just a local volume mount).
Dan
-- Linaro - Kernel Validation
-- Linaro - Kernel Validation
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users