Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pythonPackages.scrapy: 1.8.0 -> 2.0.0 #81621

Merged
merged 4 commits into from Mar 25, 2020
Merged

Conversation

marsam
Copy link
Contributor

@marsam marsam commented Mar 3, 2020

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@marsam
Copy link
Contributor Author

marsam commented Mar 4, 2020

@GrahamcOfBorg build python3Packages.scrapy

@evanjs
Copy link
Member

evanjs commented Mar 14, 2020

Thank you!

Poking around the repository, I found scrapy/scrapy#4356, which seems to (indirectly) refer to some of the tests that are failing.

e.g. https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/downloadermiddlewares/chunked.py#L7

Which refers to a statement found in one of the failing tests:
body = decode_chunked_transfer(response.body) on https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/downloadermiddlewares/chunked.py#L19

Though, interestingly.... it didn't mention the use of such deprecated methods in tests?
https://github.com/scrapy/scrapy/blob/f9bf4b8d4dd64a1d65e949927b8ea7ad34e756d3/tests/test_utils_http.py#L15, for example 🤷‍♂️

Anyway, would disabling the two offending tests be enough for the time being?

Tests disabled:

  • tests/test_utils_gz.py::GunzipTest::test_is_x_gzipped_right
  • tests/test_utils_http.py::ChunkedTest::test_decode_chunked_transfer
Relevant diff
diff --git a/pkgs/development/python-modules/scrapy/default.nix b/pkgs/development/python-modules/scrapy/default.nix
index cc22804d48a..6473cecca8b 100644
--- a/pkgs/development/python-modules/scrapy/default.nix
+++ b/pkgs/development/python-modules/scrapy/default.nix
@@ -32,7 +32,7 @@ buildPythonPackage rec {
   # Ignore xml encoding test on darwin because lxml can't find encodings https://bugs.launchpad.net/lxml/+bug/707396
   checkPhase = ''
     substituteInPlace pytest.ini --replace "addopts = --doctest-modules" "addopts ="
-    pytest --ignore=tests/test_linkextractors_deprecated.py --ignore=tests/test_proxy_connect.py --deselect tests/test_crawl.py::CrawlTestCase::test_retry_dns_error ${lib.optionalString stdenv.isDarwin "--deselect tests/test_utils_iterators.py::LxmlXmliterTestCase::test_xmliter_encoding"}
+    pytest --ignore=tests/test_linkextractors_deprecated.py --ignore=tests/test_proxy_connect.py --deselect tests/test_crawl.py::CrawlTestCase::test_retry_dns_error ${lib.optionalString stdenv.isDarwin "--deselect tests/test_utils_iterators.py::LxmlXmliterTestCase::test_xmliter_encoding --deselect tests/test_utils_gz.py::GunzipTest::test_is_x_gzipped_right --deselect tests/test_utils_http.py::ChunkedTest::test_decode_chunked_transfer"}
   '';
 
   src = fetchPypi {

Seems to compile on my end.

@marsam
Copy link
Contributor Author

marsam commented Mar 20, 2020

@GrahamcOfBorg build python3Packages.scrapy

@marsam marsam marked this pull request as ready for review March 20, 2020 10:11
@marsam marsam merged commit 5ce4be4 into NixOS:master Mar 25, 2020
@marsam marsam deleted the update-scrapy branch March 25, 2020 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants