Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/testing: add deprecation notice for Perl VM tests #79335

Merged
merged 3 commits into from Feb 10, 2020

Conversation

flokli
Copy link
Contributor

@flokli flokli commented Feb 6, 2020

Most VM tests have been migrated to use the python test driver
(introduced in #71684), the migration is tracked in #72828 (which also
thankfully uncovered and fixed many currently broken tests)

While increasing the acceptance and adoption of NixOS integration tests
by using a more popular language, there was also nobody willing to do
larger refactors in the currently very convoluted test infrastructure.

We plan to remove the perl infrastructure between the 20.03 and 20.09
release, to be able to do these refactorings.

Some people might be using Perl tests in their internal CI, so print a
warning for 20.03, and give users time to move to the python testing
infrastructure.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@flokli flokli added this to the 20.03 milestone Feb 6, 2020
Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the intent of this PR.

Just concerned that the warning message may get lost as nixos tests are particularly noisy. Using nix build it might be useful, but nix-build will definitely blanket the warning with noise.

@ofborg ofborg bot added the 6.topic: nixos label Feb 6, 2020
@flokli
Copy link
Contributor Author

flokli commented Feb 6, 2020 via email

@worldofpeace
Copy link
Contributor

@worldofpeace I assume you're adding a general explanation to the release notes as well?

There's one thing that's being forgot here aside from the deprecation notice and it's nixosTest uses nixos/lib/testing.nix.

@worldofpeace
Copy link
Contributor

@flokli I believe I will be adding that as a highlight.

@fpletz
Copy link
Member

fpletz commented Feb 7, 2020

While in principle being in favour of this, I have a question about your refactoring plans: Will the current Python driver API (both Nix and Python) stay compatible? If we advocate to move away from the Perl test infra, we should have something ready that will still be compatible for at least the next release in half a year.

If there a big changes planned, keeping the Perl stuff around could be worthwhile and a manageable technical debt because both test drivers could be cleanly separated with the Perl driver in maintenance mode.

If the Python API is considered stable, awesome! 👍

@flokli
Copy link
Contributor Author

flokli commented Feb 7, 2020 via email

worldofpeace and others added 3 commits February 9, 2020 23:53
This still referred to the legacy make-test.nix (Perl)
Most VM tests have been migrated to use the python test driver
(introduced in NixOS#71684), the migration is tracked in NixOS#72828 (which also
thankfully uncovered and fixed many currently broken tests)

While increasing the acceptance and adoption of NixOS integration tests
by using a more popular language, there was also nobody willing to do
larger refactors in the currently very convoluted test infrastructure.

We plan to remove the perl infrastructure between the 20.03 and 20.09
release, to be able to do these refactorings.

Some people might be using Perl tests in their internal CI, so print a
warning for 20.03, and give users time to move to the python testing
infrastructure.
@worldofpeace
Copy link
Contributor

This PR does update nixosTest to use the python driver, and you can't use the nixpkgs function to access the perl one anymore. @flokli noted in IRC that he didn't think that would be problematic.

@flokli
Copy link
Contributor Author

flokli commented Feb 10, 2020

Well, nixosTest seems to be quite newer the import ./make-test.nix pattern, which can still be used to execute perl tests.

I don't think it makes any sense to introduce a nixosTestPerl at least 😆

@worldofpeace
Copy link
Contributor

Well, nixosTest seems to be quite newer the import ./make-test.nix pattern, which can still be used to execute perl tests.

I don't think it makes any sense to introduce a nixosTestPerl at least laughing

I think similarly that having anything like a nixosTestPerl would be pointless if it's just going to be removed in 20.09.

@worldofpeace worldofpeace merged commit aa4ba50 into NixOS:master Feb 10, 2020
@worldofpeace worldofpeace mentioned this pull request Feb 23, 2020
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants