simp_le: 0.9.0 -> 0.16.0 #71291

picnoir · 2019-10-17T16:09:42Z

Motivation for this change

As of today, let's encrypt dropped ACME V1 support. We need to update simp_le to 0.16.0 to support ACME V2.

Things done

Notify maintainers

cc @arianvp @flokli

picnoir · 2019-10-17T16:10:46Z

Note: we need to update the let's encrypt service as well.

See discussion in #70966 #62283

arianvp · 2019-10-17T16:26:59Z

@GrahamcOfBorg test acme

picnoir · 2019-10-17T16:28:42Z

Trying to update acme as well.

There's something wrong with the acme-ci integration test setup:

    workers = ['primary'] if not config.option.numprocesses\
E   AttributeError: 'Namespace' object has no attribute 'numprocesses'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!

Coming from there: https://github.com/certbot/certbot/blob/master/certbot-ci/certbot_integration_tests/conftest.py#L85

Investigating.

picnoir · 2019-10-17T16:29:07Z

Should I disable this test suite (acme-ci) altogether?

picnoir · 2019-10-17T17:00:46Z

I have to go offline for today.

Since this is a somehow critical situation (letsencrypt is broken), I disabled the certbot tests altogether.

In order to re-enable them, we have to pass the --numprocesses 1 flag to pytest.

I am not familiar enough with the nixos python infrastructure to know how to do it and did not manage to track down were this override should happen.

I'll have a second look to this issue that tomorrow. In the meantime, feel free to either merge this PR with the disabled tests or edit it to pass the appropriate flag to the test runner.

WilliButz

As mentioned in the changelog, this version does not work out of the box with the previous configuration, so the module needs an update as well.

Adding the required flag did the trick for me, but I'd like someone else who is more familiar with the whole acme module to do a review before merging :)

diff --git a/nixos/modules/security/acme.nix b/nixos/modules/security/acme.nix
index b321c04e574..e35ea0c7b92 100644
--- a/nixos/modules/security/acme.nix
+++ b/nixos/modules/security/acme.nix
@@ -69,9 +69,9 @@ let
       plugins = mkOption {
         type = types.listOf (types.enum [
           "cert.der" "cert.pem" "chain.pem" "external.sh"
-          "fullchain.pem" "full.pem" "key.der" "key.pem" "account_key.json"
+          "fullchain.pem" "full.pem" "key.der" "key.pem" "account_key.json" "account_reg.json"
         ]);
-        default = [ "fullchain.pem" "full.pem" "key.pem" "account_key.json" ];
+        default = [ "fullchain.pem" "full.pem" "key.pem" "account_key.json" "account_reg.json" ];
         description = ''
           Plugins to enable. With default settings simp_le will
           store public certificate bundle in <filename>fullchain.pem</filename>,

picnoir · 2019-10-18T06:06:27Z

Thanks @WilliButz ,

Just updated with your requested change.

The nixos test still fails though:

acmeStandalone# [   19.777060] simp_le[737]: Traceback (most recent call last):
acmeStandalone# [   19.779146] simp_le[737]:   File "/nix/store/dj30chq7dlnh8aphj1bq79b4hjlmj47p-python3.7-urllib3-1.24.3/lib/python3.7/site-packages/urllib3/connection.py", line 159, in _new_conn
acmeStandalone# [   19.787745] simp_le[737]:     (self._dns_host, self.port), self.timeout, **extra_kw)
acmeStandalone# [   19.789733] simp_le[737]:   File "/nix/store/dj30chq7dlnh8aphj1bq79b4hjlmj47p-python3.7-urllib3-1.24.3/lib/python3.7/site-packages/urllib3/util/connection.py", line 57, in create_connection
acmeStandalone# [   19.797023] simp_le[737]:     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
acmeStandalone# [   19.804943] simp_le[737]:   File "/nix/store/8lhmyjarm73453f2mdz0xli9w8sy0wvh-python3-3.7.4/lib/python3.7/socket.py", line 748, in getaddrinfo
acmeStandalone# [   19.812701] simp_le[737]:     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
acmeStandalone# [   19.819733] simp_le[737]: socket.gaierror: [Errno -2] Name or service not known
acmeStandalone# [   19.821687] simp_le[737]: During handling of the above exception, another exception occurred:
acmeStandalone# [   19.828823] simp_le[737]: Traceback (most recent call last):
acmeStandalone# [   19.830757] simp_le[737]:   File "/nix/store/dj30chq7dlnh8aphj1bq79b4hjlmj47p-python3.7-urllib3-1.24.3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
acmeStandalone# [   19.838018] simp_le[737]:     chunked=chunked)
acmeStandalone# [   19.844801] simp_le[737]:   File "/nix/store/dj30chq7dlnh8aphj1bq79b4hjlmj47p-python3.7-urllib3-1.24.3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
acmeStandalone# [   19.852721] simp_le[737]:     self._validate_conn(conn)
acmeStandalone# [   19.854093] simp_le[737]:   File "/nix/store/dj30chq7dlnh8aphj1bq79b4hjlmj47p-python3.7-urllib3-1.24.3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
acmeStandalone# [   19.860752] simp_le[737]:     conn.connect()
acmeStandalone# [   19.861986] simp_le[737]:   File "/nix/store/dj30chq7dlnh8aphj1bq79b4hjlmj47p-python3.7-urllib3-1.24.3/lib/python3.7/site-packages/urllib3/connection.py", line 301, in connect
acmeStandalone# [   19.869857] simp_le[737]:     conn = self._new_conn()
acmeStandalone# [   19.871853] simp_le[737]:   File "/nix/store/dj30chq7dlnh8aphj1bq79b4hjlmj47p-python3.7-urllib3-1.24.3/lib/python3.7/site-packages/urllib3/connection.py", line 168, in _new_conn
acmeStandalone# [   19.883110] simp_le[737]:     self, "Failed to establish a new connection: %s" % e)
acmeStandalone# [   19.890968] simp_le[737]: urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fcc238c6f50>: Failed to establish a new connection: [Errno -2] Name or service not known
acmeStandalone# [   19.899746] simp_le[737]: During handling of the above exception, another exception occurred:
acmeStandalone# [   19.901795] simp_le[737]: Traceback (most recent call last):
acmeStandalone# [   19.904480] simp_le[737]:   File "/nix/store/k27rv5djq90f93rl28b08aj493061lcz-python3.7-requests-2.22.0/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
acmeStandalone# [   19.914962] simp_le[737]:     timeout=timeout
acmeStandalone# [   19.917055] simp_le[737]:   File "/nix/store/dj30chq7dlnh8aphj1bq79b4hjlmj47p-python3.7-urllib3-1.24.3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
acmeStandalone# [   19.923495] simp_le[737]:     _stacktrace=sys.exc_info()[2])
acmeStandalone# [   19.926365] simp_le[737]:   File "/nix/store/dj30chq7dlnh8aphj1bq79b4hjlmj47p-python3.7-urllib3-1.24.3/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment
acmeStandalone# [   19.930136] simp_le[737]:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
acmeStandalone# [   19.933354] simp_le[737]: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fcc238c6f50>: Failed to establish a new connection: [Errno -2] Name or service not known'))
acmeStandalone# [   19.946894] simp_le[737]: During handling of the above exception, another exception occurred:
acmeStandalone# [   19.950181] simp_le[737]: Traceback (most recent call last):
acmeStandalone# [   19.954652] simp_le[737]:   File "/nix/store/ddgffgv61bpirahy4sxdg53703izd45j-simp_le-client-0.16.0/lib/python3.7/site-packages/simp_le.py", line 1565, in main
acmeStandalone# [   19.960103] simp_le[737]:     return main_with_exceptions(cli_args)
acmeStandalone# [   19.964639] simp_le[737]:   File "/nix/store/ddgffgv61bpirahy4sxdg53703izd45j-simp_le-client-0.16.0/lib/python3.7/site-packages/simp_le.py", line 1549, in main_with_exceptions
acmeStandalone# [   19.973666] simp_le[737]:     persist_new_data(args, existing_data)
acmeStandalone# Job for acme-standalone.com.service failed because the control process exited with error code.
acmeStandalone# See "systemctl status acme-standalone.com.service" and "journalctl -xe" for details.[   19.984113]
acmeStandalone# simp_le[737]:   File "/nix/store/ddgffgv61bpirahy4sxdg53703izd45j-simp_le-client-0.16.0/lib/python3.7/site-packages/simp_le.py", line 1408, in persist_new_data
acmeStandalone# [   19.988593] simp_le[737]:     args, existing_data.account_key, existing_data.account_reg)
acmeStandalone# [   19.992414] simp_le[737]:   File "/nix/store/ddgffgv61bpirahy4sxdg53703izd45j-simp_le-client-0.16.0/lib/python3.7/site-packages/simp_le.py", line 1341, in registered_client
acmeStandalone# [   19.996954] simp_le[737]:     directory = messages.Directory.from_json(net.get(args.server).json())
acmeStandalone# [   20.003102] simp_le[737]:   File "/nix/store/5703sd0w9h9gk3ii51fc6kkwhbadaiv8-python3.7-acme-0.39.0/lib/python3.7/site-packages/acme/client.py", line 1161, in get
acmeStandalone# [   20.008666] simp_le[737]:     self._send_request('GET', url, **kwargs), content_type=content_type)
acmeStandalone# [   20.012744] systemd[1]: acme-standalone.com.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

Will look into it in the next ~4 hours. In the meantime, if somebody familiar with this whole area understands what's happening, feel free to edit this PR.

picnoir · 2019-10-18T08:25:12Z

Updated nixos tests to use acme-V02 endpoints.

We are currently using an outdated boulder version for the tests; which is not compatible with the acme-v02 protocol.

We probably want to:

Extract boulder from the nixos test and declare it as a regular nixpkgs pkg.
Update the boulder derivation to release-2019-10-13

I won't have enough spare time to do that today, would someone be willing to share the migration effort here?

adisbladis · 2019-10-18T10:24:49Z

@NinjaTrappeur I took a few minutes and packaged up Boulder in #71324

manveru · 2019-10-18T10:40:12Z

FWIW, i'm using this with the fix from @WilliButz and it works fine :)

flokli · 2019-10-18T10:46:28Z

FWIW, i'm using this with the fix ***@***.*** and it works fine :)

What's the fix? Can you link to it? ;-)

manveru · 2019-10-18T10:54:34Z

@flokli I made a branch at manveru@956169b

picnoir · 2019-10-18T10:56:50Z

@flokli @manveru It's included in this PR right? Am I missing something?

pkgs/tools/admin/simp_le/default.nix

jtojnar · 2019-10-18T11:28:47Z

pkgs/tools/admin/certbot/default.nix

    josepy
    parsedatetime
    psutil
    pyRFC3339
    pyopenssl
+    pytest


Shouldn't this go to checkInputs?

fpletz

Thanks!

c0bw3b · 2019-10-24T09:25:08Z

@GrahamcOfBorg build boulder certbot pebble simp_le

c0bw3b · 2019-10-24T09:26:25Z

(nitpick)
boulder and pebble could be better located under pkgs/tools/security rather than pkgs/tools/admin

picnoir · 2019-10-24T12:58:53Z

I'm kind of puzzled by those ofborg errors. I tried to reproduce that locally on my machine and did not manage to trigger any of those.

boulder and pebble could be better located under pkgs/tools/security rather than pkgs/tools/admin

certbot is currently stored in pkgs/tools/admin, I stored both boulder and pebble there out of consistency.

This PR adds both the pebble and the boulder derivation to pkgs. Since we do not provide any service other than the acme test for both of those, I think we should drop the one we decide not to use (be it pebble or boulder).

I'll wait for azlig's decision - is this pebble-based test good enough or should we rollback and fix the boulder one - to drop the useless commit.

flokli · 2019-10-24T14:44:45Z

@NinjaTrappeur would you mind filing a backport PR?

picnoir · 2019-10-24T20:14:08Z

Sure.

Backporting ACME v2 fix (#71291) to 19.09

Izorkin · 2019-10-29T09:46:20Z

Error build certbot in master branch:

Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing setuptools-build-hook
Using setuptoolsBuildPhase
Using setuptoolsShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/i7aiyqdq7m24ki60rp7dd1f6qyzngccf-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/windows-installer/template.nsi
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
applying patch /nix/store/9zxgqy1y6mxr89bb5m7n32rns64y9qnp-0001-pebble_artifacts-hardcode-pebble-location.patch
patching file certbot-ci/certbot_integration_tests/utils/pebble_artifacts.py
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Executing setuptoolsBuildPhase
Traceback (most recent call last):
  File "nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 63, in <module>
    if StrictVersion(setuptools_version) >= StrictVersion('36.2'):
  File "/nix/store/zdh16dcvjw99ybam59zd2ijb6bx138j0-python3-3.7.5/lib/python3.7/distutils/version.py", line 40, in __init__
    self.parse(vstring)
  File "/nix/store/zdh16dcvjw99ybam59zd2ijb6bx138j0-python3-3.7.5/lib/python3.7/distutils/version.py", line 137, in parse
    raise ValueError("invalid version number '%s'" % vstring)
ValueError: invalid version number '41.4.0.post20191022'

picnoir · 2019-10-29T10:08:37Z

Reproduced on x86_64.

Not sure where this is coming from though :/ ~~Searching for "41.4.0" in nixpkgs does not yields any result.~~

~~Looks like we'll need to bisect that.~~

501314c might be related. @FRidh I'm not familiar with this part of the toolchain. Any idea about what could happen here?

Izorkin · 2019-10-29T11:03:14Z

With setuptools v41.2.0 same error.

picnoir · 2019-10-29T11:06:02Z

It does build on top of 0a4373a.

We should probably bisect from there.

Can you open a proper issue for tracking this?

Izorkin · 2019-10-29T12:29:38Z

In becfba9 - error build
in dc84a7d - normal build
dc84a7d...NixOS:becfba9512c68b013ac62f5adb0ffc2d96c5132e
No ideas where to look for a error.

FRidh · 2019-10-29T13:00:03Z

The failure happens since we updated setuptools. If I am correct it is due to pypa/setuptools#1847. Upstream packages (in this case certbot) needs to fix this.

Izorkin · 2019-10-29T13:02:16Z

@FRidh with setuptools v41.2.0 same error.

flokli · 2019-10-29T13:07:39Z

Let's please not clutter this PR, but instead open a new issue, and properly start git bisect-ing the breakage.

FRidh · 2019-10-29T13:11:28Z

fixed certbot in 2db400d

flokli · 2019-10-29T15:26:26Z

@FRidh does this fix need to be backported to 19.09 too?

picnoir · 2019-10-29T16:00:59Z

@FRidh does this fix need to be backported to 19.09 too?

I don't think so, 19.09 still uses setuptools `41.2.0`. We are spamming notifications by sending messages to this PR. We should open a new issue and ping the appropriate people if needed in the future.

ofborg bot requested review from makefu and gebner October 17, 2019 16:22

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Oct 17, 2019

ofborg bot requested a review from domenkozar October 17, 2019 16:45

ofborg bot added 10.rebuild-darwin: 1-10 and removed 10.rebuild-darwin: 0 labels Oct 17, 2019

picnoir force-pushed the nin-update-acme branch from 34c7826 to 00949e5 Compare October 17, 2019 16:56

c0bw3b added the 9.needs: port to stable A PR needs a backport to the stable release. label Oct 17, 2019

WilliButz requested changes Oct 17, 2019

View reviewed changes

picnoir force-pushed the nin-update-acme branch from 00949e5 to 66a3765 Compare October 18, 2019 06:03

ofborg bot added 6.topic: nixos 8.has: module (update) labels Oct 18, 2019

picnoir force-pushed the nin-update-acme branch from 66a3765 to 2ac7594 Compare October 18, 2019 08:22

picnoir mentioned this pull request Oct 18, 2019

nginx's enableACME fails because of ACMEv1 EOL #70966

Closed

jtojnar reviewed Oct 18, 2019

View reviewed changes

pkgs/tools/admin/simp_le/default.nix Outdated Show resolved Hide resolved

picnoir force-pushed the nin-update-acme branch from 2ac7594 to eeb275d Compare October 18, 2019 11:20

jtojnar reviewed Oct 18, 2019

View reviewed changes

flokli requested review from WilliButz and aszlig October 24, 2019 06:02

fpletz approved these changes Oct 24, 2019

View reviewed changes

fpletz removed their assignment Oct 24, 2019

aszlig approved these changes Oct 24, 2019

View reviewed changes

flokli merged commit dc84a7d into NixOS:master Oct 24, 2019

picnoir mentioned this pull request Oct 24, 2019

Backporting ACME v2 fix (#71291) to 19.09 #71953

Merged

c0bw3b mentioned this pull request Oct 25, 2019

certbot: 0.31.0 -> ... -> 0.35.0 #61203

Closed

10 tasks

flokli added a commit that referenced this pull request Oct 25, 2019

Merge pull request #71953 from NinjaTrappeur/nin-backport-acme

6aab37b

Backporting ACME v2 fix (#71291) to 19.09

thelegy mentioned this pull request Oct 27, 2019

Acme staging still uses the ACMEv1 #72067

Closed

picnoir mentioned this pull request Oct 29, 2019

nixos/acme: Custom ACME endpoint #72007

Merged

10 tasks

picnoir deleted the nin-update-acme branch October 29, 2019 16:01

flokli mentioned this pull request Jan 19, 2020

ACME v2 [support for wildcard certificates] #35168

Closed

2 tasks

kirelagin mentioned this pull request Jan 23, 2020

[SDAO-59] Repin nixpkgs a little bit serokell/nixpkgs#25

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

simp_le: 0.9.0 -> 0.16.0 #71291

simp_le: 0.9.0 -> 0.16.0 #71291

picnoir commented Oct 17, 2019 •

edited

picnoir commented Oct 17, 2019 •

edited

arianvp commented Oct 17, 2019

picnoir commented Oct 17, 2019

picnoir commented Oct 17, 2019

picnoir commented Oct 17, 2019

WilliButz left a comment

picnoir commented Oct 18, 2019 •

edited

picnoir commented Oct 18, 2019 •

edited

adisbladis commented Oct 18, 2019

manveru commented Oct 18, 2019

flokli commented Oct 18, 2019 via email

manveru commented Oct 18, 2019

picnoir commented Oct 18, 2019

jtojnar Oct 18, 2019

fpletz left a comment

c0bw3b commented Oct 24, 2019

c0bw3b commented Oct 24, 2019

picnoir commented Oct 24, 2019

flokli commented Oct 24, 2019

picnoir commented Oct 24, 2019

Izorkin commented Oct 29, 2019

picnoir commented Oct 29, 2019 •

edited

Izorkin commented Oct 29, 2019

picnoir commented Oct 29, 2019 •

edited

Izorkin commented Oct 29, 2019

FRidh commented Oct 29, 2019 •

edited

Izorkin commented Oct 29, 2019

flokli commented Oct 29, 2019

FRidh commented Oct 29, 2019

flokli commented Oct 29, 2019

picnoir commented Oct 29, 2019 via email

simp_le: 0.9.0 -> 0.16.0 #71291

simp_le: 0.9.0 -> 0.16.0 #71291

Conversation

picnoir commented Oct 17, 2019 • edited

Motivation for this change

Things done

Notify maintainers

picnoir commented Oct 17, 2019 • edited

arianvp commented Oct 17, 2019

picnoir commented Oct 17, 2019

picnoir commented Oct 17, 2019

picnoir commented Oct 17, 2019

WilliButz left a comment

Choose a reason for hiding this comment

picnoir commented Oct 18, 2019 • edited

picnoir commented Oct 18, 2019 • edited

adisbladis commented Oct 18, 2019

manveru commented Oct 18, 2019

flokli commented Oct 18, 2019 via email

manveru commented Oct 18, 2019

picnoir commented Oct 18, 2019

jtojnar Oct 18, 2019

Choose a reason for hiding this comment

fpletz left a comment

Choose a reason for hiding this comment

c0bw3b commented Oct 24, 2019

c0bw3b commented Oct 24, 2019

picnoir commented Oct 24, 2019

flokli commented Oct 24, 2019

picnoir commented Oct 24, 2019

Izorkin commented Oct 29, 2019

picnoir commented Oct 29, 2019 • edited

Izorkin commented Oct 29, 2019

picnoir commented Oct 29, 2019 • edited

Izorkin commented Oct 29, 2019

FRidh commented Oct 29, 2019 • edited

Izorkin commented Oct 29, 2019

flokli commented Oct 29, 2019

FRidh commented Oct 29, 2019

flokli commented Oct 29, 2019

picnoir commented Oct 29, 2019 via email

picnoir commented Oct 17, 2019 •

edited

picnoir commented Oct 17, 2019 •

edited

picnoir commented Oct 18, 2019 •

edited

picnoir commented Oct 18, 2019 •

edited

picnoir commented Oct 29, 2019 •

edited

picnoir commented Oct 29, 2019 •

edited

FRidh commented Oct 29, 2019 •

edited