Some python libraries, mostly scrapying-related #60775

mredaelli · 2019-05-02T16:23:48Z

Motivation for this change

Some libraries I needed and were missing

Things done

FRidh · 2019-05-02T16:25:58Z

pkgs/development/python-modules/langdetect/default.nix

+buildPythonPackage {
+  pname = "langdetect";
+  version = "1.0.7";
+  src = fetchzip {


fetchPyPi
Maybe you need to pass in a different pname here?

Maybe it's because there's only a zip?

extension = "zip";

FRidh · 2019-05-02T16:26:26Z

pkgs/development/python-modules/scrapinghub/default.nix

+
+  propagatedBuildInputs = [ requests retrying six ];
+
+  # Prevent unicode decoding error in setup.py


Should not be needed anymore...What's the exact issue?

To be honest, I don't see it anymore either :/

probably you wrote these expressions while using an older Nixpkgs (18.09)

I think I did

FRidh · 2019-05-02T16:26:39Z

pkgs/development/python-modules/shub/default.nix

+    # Tests are network-based
+    doCheck = false;
+
+    # Prevent unicode decoding error in setup.py


To be honest, I don't see it anymore either :/

FRidh · 2019-05-02T16:27:09Z

pkgs/development/python-modules/scrapinghub/default.nix

+  LC_ALL = "en_US.UTF-8";
+
+  # Tests are network-based
+  doCheck = false;


They don't have tests scraping from localhost?

Not sure, but this is not the scraping library. Rather, the library that manages the crawlers running in their cloud environment, like aws-cli, I'd say.
Anyway, I see lots of messages like this in the tests:

pool_timeout = None, release_conn = False, chunked = False, body_pos = None response_kw = {'decode_content': False, 'preload_content': False}, conn = None release_this_conn = True, err = None, clean_exit = False timeout_obj = <urllib3.util.timeout.Timeout object at 0x7ffff537ab00> is_new_proxy_conn = False

Don't know what that is, probably just test-related.

Well, I disabled the tests that give a problem. At least some are passing :)

FRidh · 2019-05-02T16:27:19Z

pkgs/development/python-modules/shub/default.nix

+}:
+
+buildPythonPackage rec {
+    pname = "shub";


indentation

mredaelli · 2019-05-03T16:20:25Z

I think I covered the changes
And added a few more :)

mredaelli · 2019-05-09T17:59:23Z

@FRidh do I have to do more changes?

c0bw3b · 2019-05-09T18:36:21Z

langdetect and pdftotext were also added in another PR (#44086) so you can drop those 2 commits to resolve the current conflicts.

mredaelli · 2019-05-09T18:54:08Z

@c0bw3b done :)

c0bw3b · 2019-05-09T19:57:59Z

Testing on NixOS with sandboxing :

Result of nix-review pr 60775 1

3 package failed to build:

python27Packages.backports-datetime-fromisoformat
python27Packages.betterreads
python27Packages.enolib

22 package were build:

python27Packages.cfscrape
python27Packages.docxtotxt
python27Packages.python-datemath
python27Packages.rauth
python27Packages.rules
python27Packages.scrapinghub
python27Packages.scrapy-crawlera
python27Packages.sqlitedict
python37Packages.backports-datetime-fromisoformat
python37Packages.betterreads
python37Packages.cfscrape
python37Packages.docxtotxt
python37Packages.enolib
python37Packages.monkeytype
python37Packages.python-datemath
python37Packages.rauth
python37Packages.retype
python37Packages.rules
python37Packages.scrapinghub
python37Packages.scrapy-crawlera
python37Packages.shub
python37Packages.sqlitedict

Log excerpt for python27Packages.backports-datetime-fromisoformat (Py3 only?)

creating build/lib.linux-x86_64-2.7/backports/datetime_fromisoformat
copying backports/datetime_fromisoformat/__init__.py -> build/lib.linux-x86_64-2.7/backports/datetime_fromisoformat
running build_ext
building 'backports._datetime_fromisoformat' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/backports
creating build/temp.linux-x86_64-2.7/backports/datetime_fromisoformat
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/nix/store/29xfvq39h5z7af90id0zaa4a09vgpsm7-python-2.7.16/include/python2.7 -c backports/datetime_fromisoformat/module.c -o build/temp.linux-x86_64-2.7/backports/datetime_fromisoformat/module.o
backports/datetime_fromisoformat/module.c: In function ‘init_datetime_fromisoformat’:
backports/datetime_fromisoformat/module.c:75:30: error: ‘module’ undeclared (first use in this function); did you mean ‘modfl’?
     initialize_timezone_code(module);
                              ^~~~~~
                              modfl
backports/datetime_fromisoformat/module.c:75:30: note: each undeclared identifier is reported only once for each function it appears in
error: command 'gcc' failed with exit status 1

Log excerpt for python27Packages.enolib (Py3 only?)

============================= test session starts ==============================
platform linux2 -- Python 2.7.16, pytest-4.2.1, py-1.7.0, pluggy-0.8.1
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/build/source/.hypothesis/examples')
rootdir: /build/source, inifile: setup.cfg
plugins: cov-2.6.1, hypothesis-4.7.3
collected 0 items / 117 errors

mredaelli · 2019-05-22T19:56:29Z

Sorry, was away.
Yes, they are py3 only. I forget, how do I handle that?

c0bw3b · 2019-05-22T21:19:37Z

No problem.
You can set disabled = !isPy3k; like alot does

stale · 2020-06-02T01:48:07Z

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the
related code and @ mention them in a comment.
Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
irc.freenode.net.

onny · 2022-09-04T19:43:53Z

Needs a rebase if still relevant

mredaelli · 2022-10-21T16:24:09Z

Needs a rebase if still relevant

I don't really use these anymore, so I'll close the PR

mredaelli requested a review from FRidh as a code owner May 2, 2019 16:23

FRidh requested changes May 2, 2019

View reviewed changes

ofborg bot added 6.topic: python 8.has: documentation 8.has: package (new) 11.by: package-maintainer 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels May 2, 2019

mredaelli force-pushed the scrapypy branch 2 times, most recently from bb067d9 to 51cc66a Compare May 2, 2019 21:53

mredaelli added 14 commits May 9, 2019 20:51

pythonPackages.docx2txt: init at 0.7

0dbdc19

pythonPackages.cfscrape: init at 1.9.7

83f76c6

pythonPackages.scrapinghub: init at 2.1.1

0605ba1

pythonPackages.shub: init at 2.8.2

b244357

pythonPackages.scrapy-crawlera: init at 1.5.0

de2df9f

pythonPackages.sqlitedict: init at 1.6.0

b11d8a3

pythonPackages.backports-datetime-fromisoformat: init at 1.0.0

71522d1

pythonPackages.rauth: init at 0.7.2

f3566cb

pythonPackages.betterreads: init at 0.4.2

3786c8d

pythonPackages.rules: init at 2.0.1

c2a20c7

pythonPackages.retype: init at 17.12.0

f5c411b

pythonPackages.monkeytype: init at 19.1.1

91bc3c8

pythonPackages.python-datemath: init at 1.4.7

4f64e25

pythonPackages.enolib: init at 0.5.1

3d8caaa

mredaelli force-pushed the scrapypy branch from 53bc4e7 to 3d8caaa Compare May 9, 2019 18:51

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 2, 2020

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 4, 2022

mredaelli closed this Oct 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some python libraries, mostly scrapying-related #60775

Some python libraries, mostly scrapying-related #60775

mredaelli commented May 2, 2019

FRidh May 2, 2019

mredaelli May 2, 2019

FRidh May 2, 2019

mredaelli May 6, 2019

FRidh May 2, 2019

mredaelli May 2, 2019

FRidh May 2, 2019

mredaelli May 6, 2019

FRidh May 2, 2019

mredaelli May 2, 2019

FRidh May 2, 2019

mredaelli May 2, 2019

FRidh May 2, 2019

mredaelli May 2, 2019

FRidh May 2, 2019

mredaelli May 6, 2019

mredaelli commented May 3, 2019 •

edited

mredaelli commented May 9, 2019

c0bw3b commented May 9, 2019

mredaelli commented May 9, 2019

c0bw3b commented May 9, 2019

mredaelli commented May 22, 2019

c0bw3b commented May 22, 2019

stale bot commented Jun 2, 2020

onny commented Sep 4, 2022

mredaelli commented Oct 21, 2022


		propagatedBuildInputs = [ requests retrying six ];

		# Prevent unicode decoding error in setup.py

Some python libraries, mostly scrapying-related #60775

Some python libraries, mostly scrapying-related #60775

Conversation

mredaelli commented May 2, 2019

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mredaelli commented May 3, 2019 • edited

mredaelli commented May 9, 2019

c0bw3b commented May 9, 2019

mredaelli commented May 9, 2019

c0bw3b commented May 9, 2019

mredaelli commented May 22, 2019

c0bw3b commented May 22, 2019

stale bot commented Jun 2, 2020

onny commented Sep 4, 2022

mredaelli commented Oct 21, 2022

mredaelli commented May 3, 2019 •

edited