This might of course cause some issues if the remote-store test starts testing stuff that's not available with the latest stable release. But it's not the case atm and it seems fair to keep things that way

Can you add this as a Hydra job? That (rather than Github actions) is the canonical test suite for Nix. It's also easier for developers since you can run a test using nix-build.

Can you add this as a Hydra job? That (rather than Github actions) is the canonical test suite for Nix. It's also easier for developers since you can run a test using nix-build.

I've actually just rewritten it to run as part of the standard testsuite (peeking the reference nix daemon from an environment variable that's set in flake.nix). Having it as a hydra job would be even nicer, but I don't think I could easily reuse the existing tests in that case (or could I?), while here I just have a test that sources remote-store.sh with an overriden nix-daemon which means that we have a reasonably good coverage at a low price

I like the approach Eelco suggested. It allows for more combinations to be tested, and to reuse the whole test suite.

reasonably good coverage at a low price

The price is always low when tests run on CI. You don't need to test compatibility when iterating on a feature, but you can still do it with nix-build or a flake command when you're working on the protocol.

It'd look like this:

old daemon

• Allow specifying the old nix package. This is your OUTER_NIX. Perhaps you could call it call it NIX_DAEMON_PACKAGE
  • The normal package does not specify this variable
  • In the checks we add the normal package with this variable as an override
• The test suite always uses NIX_DAEMON_PACKAGE's daemon when set, so the whole test suite will be run against the old daemon
• Test cases that require a new daemon are in a conditional based on the daemon version or protocol version

old client

• Similar to "old daemon", but instead of overriding based on the current normal package, we import an older Nix revision using flakes and then set the daemon to be the current package

version combinations

• Derivations per old version approach allows multiple version combinations to be tested
• Most of the value is in having one older version to run against. It doesn't even have to be old, just older. We can retain relevant versions to run against as development progresses
• We should probably check against Nix 2.3 too but that can be a separate issue. Backporting test suite changes seems perfectly acceptable to me
• Mistakes in the test suite can be corrected on the maintenance branches
• Irrelevant failures can be patched away if necessary (still beats not testing anything at all)

The price is always low when tests run on CI. You don't need to test compatibility when iterating on a feature, but you can still do it with nix-build or a flake command when you're working on the protocol.

Yes sorry, I didn't mean “low price” as “doesn't require too much computation” but rather as “easy to implement and maintain”.

I like the approach Eelco suggested. It allows for more combinations to be tested, and to reuse the whole test suite.

Well, my point is that I don't think we can (easily) reuse the test-suite because it's currently tied to the actual build process − the test setup points PATH to the directory that has been defined by the configure script, make installcheck depends on building a Nix plugin with hardcoded include paths to src/ etc…

In practice it means that we can't take an arbitrary nix and run the test suite against it − hence the solution of just adding one extra test with a hack to get an external nix-daemon.

I agree that I'd be better in the long run to indeed have it as a matrix of hydra checks, but that would be a longer term goal as it would first require un-coupling the install-testsuite from the build process (and do so without breaking the development cycle which − at least for me − relies a lot on being able to run make install && make installcheck in a row).

I've updated this a little to

1. Make it actually work in case there's a schema change (in the previous version, some commands bypassing the daemon were sneaking in the middle, triggering a db migration so that the daemon was unable to read the database for the rest of the test)
2. Add a test for the db migration (fixing Test the db migration #4227 )

@edolstra is there anything you'd like to see on this?

Yeah @edolstra very much looking forward to daemon protocol code changes being less scary after this!

Catching and fixing some things in #4588 only makes me want this more!

I've rebased this on top of master and reworked a bit the way it works:

1. It is now possible to run the installcheck independently from the rest of the build, by reusing an external Nix
2. It's also possible to use a different Nix version for the client and the daemon in installcheck. In that case, the db migration between the two versions will also be checked
3. There's a new installTests flake test that runs the install tests with
   1. The current Nix version both as client and daemon
   2. The current Nix version as client and an older one as daemon

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-9/12357/1