I did this earlier today 😢 Oh well at least I don't have to commit it so 👍

@GrahamcOfBorg test matomo

https://gist.github.com/Kiwi/48e8160134ed08164923d32594dd3298

I'm wondering if it might not be better to have it more like in this gist (but I can't figure out how to keep it DRY right now so probably not like that but to have 2 VMs) because the way I did it originally I didn't realize was only creating 1VM and which is then reused between the different versions...which will cause problems. I've already been encountering them. :|

I also have a setup now that I've gone through the install of that other PR I had to show that nothing anyone was worried about had happened... (which is good, since I've been using it for a while...)

@Kiwi I think the 1VM thing is how NixOS tests work and shouldn't be a problem because persistent storage is wiped between runs? Not sure though. Only disadvantage I see is that nix-build nixos/tests/matomo.nix runs both tests in parallel and it's impossible to distinguish which is which.

One possibility would be something like this: https://gist.github.com/mmilata/a83d34417522f61c2dde33778e7a2065 (and add the beta to all-tests.nix: mmilata@1a9216a#diff-8e2944fa8b75f34d13428287028e1323)

Not sure if it's better or even different wrt VM reuse. Disadvantage is that I don't know how to run the matomo-beta test from command line:D

With the test file we have you can do

nix-build nixos/tests/matomo.nix -A matomo-beta.driver
./result/bin/nixos-run-vms

That will launch a VM with the environment the tests run (in this case the beta, but matomo.driver works as well). The VM is named machine.

ls /tmp/vm-state-machine/
machine.qcow2  monitor  shell  xchg

It's... a persistent state machine. :) If I test the installation process on the beta one and then do the same on the stable one it will anger matomo for trying to downgrade it. (I'm trying to add more tests than we have right now) It also means I can't test both at the same time, and have to take manual steps that I'll forget to do to make sure I'm doing everything I need to do for accurate test results.

With something like the one in my gist they don't share state

nix-build nixos/tests/test-matomo.nix -A matomo-beta.driver -A matomo-stable.driver
./result/bin/nixos-run-vms
./result-2/bin/nixos-run-vms

ls /tmp/vm-state-matomo_*
/tmp/vm-state-matomo_beta:
matomo_beta.qcow2  monitor  shell  xchg

/tmp/vm-state-matomo_stable:
matomo_stable.qcow2  monitor  shell  xchg

like so 😃