pyopenssl: 19.1.0 -> 20.0.0 #105454

raboof · 2020-11-30T14:15:29Z

Motivation for this change

This is needed to build 'master' of mitmproxy,
verified that this works with this change.

Things done

This is needed to build 'master' of mitmproxy, verified that this works with this change.

FRidh · 2020-12-01T13:52:53Z

I missed this one for in #105368, so it needs to go in next time.

raboof · 2020-12-01T14:41:56Z

I missed this one for in #105368, so it needs to go in next time.

No rush, anything I can do to help it get in next time?

FRidh · 2020-12-02T08:07:36Z

No rush, anything I can do to help it get in next time?

I suppose we need some way to indicate which packages should be collected for a next batch upgrade. I suppose I could make a project for it. You should be able to add your PR to it then.

FRidh · 2020-12-02T08:16:31Z

cc @jonringer I created the "Python batch upgrade" project. Hopefully that way we can keep track of the PR's to include.

jonringer · 2020-12-02T17:09:33Z

Really wish the python ecosystem wasn't constantly tripping over itself

FRidh · 2020-12-08T04:29:33Z

Let's try how it works out. If its no good we revert.

makefu · 2020-12-13T08:35:52Z

@FRidh @raboof the update broke platformio on master, however i am unsure why:

..
running install tests
============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
OpenSSL: b'OpenSSL 1.1.1h  22 Sep 2020'
cryptography: 3.2.1
rootdir: /build/pyOpenSSL-20.0.0, configfile: setup.cfg, testpaths: tests
plugins: flaky-3.7.0
collected 525 items / 8 deselected / 517 selected                              

tests/test_crypto.py ................................................... [  9%]
........................................................................ [ 23%]
........................................................................ [ 37%]
.........................................................F.............. [ 51%]
...............                                                          [ 54%]
tests/test_debug.py .                                                    [ 54%]
tests/test_rand.py ....                                                  [ 55%]
tests/test_ssl.py ...................................................... [ 65%]
........................................................................ [ 79%]
........................................ss.............s................ [ 93%]
...............................                                          [ 99%]
tests/test_util.py .                                                     [100%]

=================================== FAILURES ===================================
__________________ TestX509StoreContext.test_verify_with_time __________________

self = <tests.test_crypto.TestX509StoreContext object at 0xf60239d0>

    def test_verify_with_time(self):
        """
        `verify_certificate` raises error when the verification time is
        set at notAfter.
        """
        store = X509Store()
        store.add_cert(self.root_cert)
        store.add_cert(self.intermediate_cert)

        expire_time = self.intermediate_server_cert.get_notAfter()
        expire_datetime = datetime.strptime(
            expire_time.decode("utf-8"), "%Y%m%d%H%M%SZ"
        )
>       store.set_time(expire_datetime)

tests/test_crypto.py:4111:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <OpenSSL.crypto.X509Store object at 0xf6023b98>
vfy_time = datetime.datetime(2047, 12, 20, 17, 11, 20)

    def set_time(self, vfy_time):
        """
        Set the time against which the certificates are verified.

        Normally the current time is used.

        .. note::

          For example, you can determine if a certificate was valid at a given
          time.

        .. versionadded:: 17.0.0

        :param datetime vfy_time: The verification time to set on this store.
        :return: ``None`` if the verification time was successfully set.
        """
        param = _lib.X509_VERIFY_PARAM_new()
        param = _ffi.gc(param, _lib.X509_VERIFY_PARAM_free)

>       _lib.X509_VERIFY_PARAM_set_time(
            param, calendar.timegm(vfy_time.timetuple())
        )
E       OverflowError: integer 2460474680 does not fit '32-bit int'

/nix/store/8z8f06f2m5j99g8jip6wk1s7fl1gjhl4-python3.8-pyOpenSSL-20.0.0/lib/python3.8/site-packages/OpenSSL/crypto.py:1679: OverflowError
=============================== warnings summary ===============================
../../nix/store/a5f5xkh9jbclv1yqq7j7rj49wivkvrmd-python3.8-pytest-6.1.2/lib/python3.8/site-packages/_pytest/config/__init__.py:1230
  /nix/store/a5f5xkh9jbclv1yqq7j7rj49wivkvrmd-python3.8-pytest-6.1.2/lib/python3.8/site-packages/_pytest/config/__init__.py:1230: PytestConfigWarning: Unknown config option: strict

    self._warn_or_fail_if_strict("Unknown config option: {}\n".format(key))

tests/test_crypto.py:39
  /build/pyOpenSSL-20.0.0/tests/test_crypto.py:39: DeprecationWarning: PKCS#7 support in pyOpenSSL is deprecated. You should use the APIs in cryptography.
    from OpenSSL.crypto import PKCS7, load_pkcs7_data

tests/test_crypto.py:40
  /build/pyOpenSSL-20.0.0/tests/test_crypto.py:40: DeprecationWarning: PKCS#12 support in pyOpenSSL is deprecated. You should use the APIs in cryptography.
    from OpenSSL.crypto import PKCS12, load_pkcs12

tests/test_ssl.py::TestContext::test_set_cipher_list[hello world:AES128-SHA1]
  /build/pyOpenSSL-20.0.0/tests/test_ssl.py:493: DeprecationWarning: str for cipher_list is no longer accepted, use bytes
    context.set_cipher_list(cipher_string)

tests/test_ssl.py::TestConnection::test_client_set_session
  /build/pyOpenSSL-20.0.0/tests/test_ssl.py:2637: DeprecationWarning: str for buf is no longer accepted, use bytes
    ctx.set_session_id("unity-test")

-- Docs: https://docs.pytest.org/en/stable/warnings.html
===Flaky Test Report===

test_gmtime_adj_notBefore passed 1 out of the required 1 times. Success!
test_gmtime_adj_notAfter passed 1 out of the required 1 times. Success!
test_set_cipher_list_no_cipher_match passed 1 out of the required 1 times. Success!

===End Flaky Test Report===
=========================== short test summary info ============================
FAILED tests/test_crypto.py::TestX509StoreContext::test_verify_with_time - Ov...
===== 1 failed, 513 passed, 3 skipped, 8 deselected, 5 warnings in 11.38s ======
builder for '/nix/store/hnvlbssv2ysv5k4z5qgqhfadllqlbb3a-python3.8-pyOpenSSL-20.0.0.drv' failed with exit code 1
...

# hydra-check --channel master platformio                     
Build Status for platformio.x86_64-linux on master
✖ (Dependency failed) platformio from 2020-12-11 - https://hydra.nixos.org/build/132653659

Last Builds:
✔ platformio from 2020-12-08 - https://hydra.nixos.org/build/132477376
...

platformio builds an chrootenv for the package

Mic92 · 2020-12-13T08:47:19Z

Maybe also pyopenssl upstream could help here. They seem to be responsive. I just could not get tests set up on pyopenssl master.

raboof · 2020-12-13T09:18:43Z

Ouch. Looking into this a bit, though it's not my comfort zone.

First observations: 2047, 12, 20, 17, 11, 20 is indeed 2460474680 since epoch, which is indeed larger than fits in a signed 32-bit int (2^31=2147483648)

raboof · 2020-12-13T09:41:43Z

Ah, this is the i686 version of the package failing, x86_64 is OK

Can I test this locally without rebuilding the world? I tried nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs> {}).lib.systems.examples.gnu32' -A python3Packages.pyopenssl but it looks like it's now building gcc :)

Found in NixOS#105454 (comment) Upstream issue pyca/pyopenssl#974

Mic92 · 2020-12-13T10:00:20Z

Ah, this is the i686 version of the package failing, x86_64 is OK

Can I test this locally without rebuilding the world? I tried nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs> {}).lib.systems.examples.gnu32' -A python3Packages.pyopenssl but it looks like it's now building gcc :)

Does it work with pkgs.pkgsi686Linux?

Mic92 · 2020-12-13T10:01:58Z

I wonder why it uses the 32 bit version at all? I don't see any 32-bit usage in the nix code.

makefu · 2020-12-13T10:03:02Z

good catch!

# nix-build -A pkgs.pkgsi686Linux.python3.pkgs.pyopenssl
...
===== 1 failed, 513 passed, 3 skipped, 8 deselected, 5 warnings in 12.41s ======
builder for '/nix/store/hnvlbssv2ysv5k4z5qgqhfadllqlbb3a-python3.8-pyOpenSSL-20.0.0.drv' failed with exit code 1
error: build of '/nix/store/hnvlbssv2ysv5k4z5qgqhfadllqlbb3a-python3.8-pyOpenSSL-20.0.0.drv' failed

makefu · 2020-12-13T10:05:01Z

@raboof thanks for making an effort and even reporting the issue upstream!

FRidh · 2020-12-13T10:09:21Z

I wonder why it uses the 32 bit version at all? I don't see any 32-bit usage in the nix code.

build-fhs-userenv includes 32-bit versions by default when multiPkgs is passed in.

On x86_64 they are added to targetPkgs and in addition their 32bit
versions are also installed.

Mic92 · 2020-12-13T10:20:48Z

@makefu maybe this can be disabled for platformio actually. I don't see why multi-lib is needed.

makefu · 2020-12-13T10:42:56Z

@Mic92 I tried flashing two ESP8266 with esphome and everything seems to work (at least for these build environments)

makefu · 2020-12-13T10:44:09Z

diff --git a/pkgs/development/arduino/platformio/chrootenv.nix b/pkgs/development/arduino/platformio/chrootenv.nix
index 72384c0994a..b9b17012ed2 100644
--- a/pkgs/development/arduino/platformio/chrootenv.nix
+++ b/pkgs/development/arduino/platformio/chrootenv.nix
@@ -23,7 +23,6 @@ in buildFHSUserEnv {
   name = "platformio";
 
   targetPkgs = pio-pkgs;
-  multiPkgs = pio-pkgs;
 
   meta = with lib; {
     description = "An open source ecosystem for IoT development";

Mic92 · 2020-12-13T10:59:12Z

multiPkgs

Would you mind open this as a PR? I might also ask upstream if there are aware of any 32-bit based toolchains...

raboof · 2020-12-13T11:06:57Z

(in parallel I PR'ed skipping this particular test on i686 in #106810 - I think both changes are useful improvements)

makefu · 2020-12-13T11:45:19Z

@Mic92 i created a post in the community discourse page: https://community.platformio.org/t/any-i686-library-dependencies-for-platformio-core/17769

multiPkgs makes available 32-bits versions of dependencies on 64-bits systems, but we're not aware of any toolchains that require this. As discussed in NixOS#105454 (comment) See also https://community.platformio.org/t/any-i686-library-dependencies-for-platformio-core/17769 Co-Authored-By: makefu <github@syntax-fehler.de>

Found in #105454 (comment) Upstream issue pyca/pyopenssl#974

raboof requested review from FRidh and jonringer as code owners November 30, 2020 14:15

raboof changed the base branch from master to staging November 30, 2020 14:17

pyopenssl: 19.1.0 -> 20.0.0

c4318a3

This is needed to build 'master' of mitmproxy, verified that this works with this change.

raboof force-pushed the pyopenssl-19.1.0-to-20.0.0 branch from 9e37b91 to c4318a3 Compare November 30, 2020 14:18

ofborg bot added 6.topic: python 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Nov 30, 2020

FRidh added this to the 21.03 milestone Dec 1, 2020

FRidh added this to To do in Python batch upgrade Dec 2, 2020

FRidh merged commit 13456d6 into NixOS:staging Dec 8, 2020

Python batch upgrade automation moved this from To do to Done Dec 8, 2020

raboof mentioned this pull request Dec 13, 2020

test_verify_with_time test fails on i686 pyca/pyopenssl#974

Open

raboof added a commit to raboof/nixpkgs that referenced this pull request Dec 13, 2020

pyopenssl: skip test that fails on i686

242e3f3

Found in NixOS#105454 (comment) Upstream issue pyca/pyopenssl#974

raboof mentioned this pull request Dec 13, 2020

pyopenssl: skip test that fails on i686 #106810

Merged

10 tasks

raboof mentioned this pull request Dec 15, 2020

platformio: don't use multiPkgs in chrootenv #106957

Closed

10 tasks

FRidh pushed a commit that referenced this pull request Dec 15, 2020

pyopenssl: skip test that fails on i686

40b1f0d

Found in #105454 (comment) Upstream issue pyca/pyopenssl#974

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pyopenssl: 19.1.0 -> 20.0.0 #105454

pyopenssl: 19.1.0 -> 20.0.0 #105454

raboof commented Nov 30, 2020

FRidh commented Dec 1, 2020

raboof commented Dec 1, 2020

FRidh commented Dec 2, 2020 •

edited

FRidh commented Dec 2, 2020

jonringer commented Dec 2, 2020

FRidh commented Dec 8, 2020

makefu commented Dec 13, 2020

Mic92 commented Dec 13, 2020

raboof commented Dec 13, 2020

raboof commented Dec 13, 2020

Mic92 commented Dec 13, 2020

Mic92 commented Dec 13, 2020

makefu commented Dec 13, 2020 •

edited

makefu commented Dec 13, 2020

FRidh commented Dec 13, 2020 •

edited

Mic92 commented Dec 13, 2020

makefu commented Dec 13, 2020

makefu commented Dec 13, 2020

Mic92 commented Dec 13, 2020

raboof commented Dec 13, 2020

makefu commented Dec 13, 2020

pyopenssl: 19.1.0 -> 20.0.0 #105454

pyopenssl: 19.1.0 -> 20.0.0 #105454

Conversation

raboof commented Nov 30, 2020

Motivation for this change

Things done

FRidh commented Dec 1, 2020

raboof commented Dec 1, 2020

FRidh commented Dec 2, 2020 • edited

FRidh commented Dec 2, 2020

jonringer commented Dec 2, 2020

FRidh commented Dec 8, 2020

makefu commented Dec 13, 2020

Mic92 commented Dec 13, 2020

raboof commented Dec 13, 2020

raboof commented Dec 13, 2020

Mic92 commented Dec 13, 2020

Mic92 commented Dec 13, 2020

makefu commented Dec 13, 2020 • edited

makefu commented Dec 13, 2020

FRidh commented Dec 13, 2020 • edited

Mic92 commented Dec 13, 2020

makefu commented Dec 13, 2020

makefu commented Dec 13, 2020

Mic92 commented Dec 13, 2020

raboof commented Dec 13, 2020

makefu commented Dec 13, 2020

FRidh commented Dec 2, 2020 •

edited

makefu commented Dec 13, 2020 •

edited

FRidh commented Dec 13, 2020 •

edited