Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pythonPackages.textacy: init at 0.4.1 #29334

Merged
merged 8 commits into from Sep 16, 2017
Merged

pythonPackages.textacy: init at 0.4.1 #29334

merged 8 commits into from Sep 16, 2017

Conversation

rvl
Copy link
Contributor

@rvl rvl commented Sep 13, 2017

Motivation for this change

Natural language processing with Python under Nix.

  • This change adds the textacy python package and its dependencies.
  • Versions have been tweaked on some dependencies where necessary.
  • Finally, this change also adds python package derivations for the language model datasets required for many NLP functions.
Datasets

The language data models can't be downloaded in the way described by the spaCy docs because their download script is a wrapper around pip install. The data files total a couple GB and need enough /tmp space to assemble. I'm assuming that because the derivations exist under an attrset, they won't be automatically built by Hydra (good).

Things done

  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • Linux
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip" -- didn't use nox but have tested build of mopidy-gmusic, but not the "oslo" packages because they are already marked broken.
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

/cc @sdll

@rvl rvl requested a review from FRidh as a code owner September 13, 2017 22:44
@mention-bot
Copy link

@rvl, thanks for your PR! By analyzing the history of the files in this pull request, we identified @jgillich, @carlsverre and @FRidh to be potential reviewers.

FRidh
FRidh requested changes Sep 14, 2017
View reviewed changes
pkgs/development/python-modules/pyemd/default.nix Outdated
@@ -0,0 +1,29 @@
{ stdenv
, pkgs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pkgs

pkgs/development/python-modules/spacy/models.nix Outdated
@@ -0,0 +1,31 @@
{ stdenv
, pkgs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pkgs

pkgs/development/python-modules/textacy/default.nix Outdated
@@ -0,0 +1,128 @@
{ stdenv
, pkgs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pkgs

pkgs/development/python-modules/textacy/default.nix Outdated
, nose
}:
let
ijson = buildPythonPackage rec {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these packages not called from python-packages.nix?

pkgs/top-level/python-packages.nix Outdated
@@ -14313,6 +14313,20 @@ in {
};
};

cachetools = buildPythonPackage rec {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

separate file in python-modules/

pkgs/top-level/python-packages.nix Outdated
@@ -24150,11 +24164,11 @@ EOF


unidecode = buildPythonPackage rec {
name = "Unidecode-0.04.18";
name = "Unidecode-0.04.21";

src = pkgs.fetchurl {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fetchPypi and preferable separate file

pkgs/top-level/python-packages.nix
@@ -27308,6 +27322,12 @@ EOF

spacy = callPackage ../development/python-modules/spacy { };

spacy_models = callPackage ../development/python-modules/spacy/models.nix { };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be nicer to add this to passthru.models of spacy. No need though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I will pass on this because the data model packages have dependencies on spacy.

@rvl
Copy link
Contributor Author

rvl commented Sep 15, 2017

Thanks for the review @FRidh. I have addressed your comments in new commits which can be autosquashed.

@FRidh
Copy link
Member

FRidh commented Sep 15, 2017

@rvl please do the squashing yourself; commit per package.

@rvl
pythonPackages.unidecode: 0.04.18 -> 0.04.21
151d0fd 
Higher version is a requirement of textacy.
@rvl
pythonPackages.cachetools: Add version 2.0.1
31037ba 
Textacy requires a later version.

v2.0.0 introduced breaking changes, so keep two versions for other
packages.
@rvl
pythonPackages.pyemd: init at 0.4.4
18161a5 
This module is a requirement of textacy.
@rvl
pythonPackages.textacy: init at 0.4.1
3b1877f
@rvl
pythonPackages.spacy: add models data attr set
fad89ed 
Model                   Version
es_core_web_md          1.0.0
fr_depvec_web_lg        1.0.0
en_core_web_md          1.2.1
en_depent_web_md        1.2.1
en_core_web_sm          1.2.0
de_core_news_md         1.0.0
en_vectors_glove_md     1.0.0
@rvl
pythonPackages.clld-cffi: init at 0.1.4
d851b95
@rvl
pythonPackages.ijson: init at 2.3
c5d1218
@rvl
pythonPackages.pyphen: init at 0.9.4
fdb90ed
@rvl
Copy link
Contributor Author

rvl commented Sep 16, 2017

@FRidh OK, have pushed the squashed commits.

@FRidh FRidh merged commit e8f8736 into NixOS:master Sep 16, 2017
@rvl
Copy link
Contributor Author

rvl commented Sep 16, 2017

Thanks @FRidh :-)

@rvl rvl deleted the python-textacy branch September 16, 2017 10:35
@disassembler disassembler mentioned this pull request Sep 16, 2017
Merged
8 tasks
rvl added a commit to rvl/nixpkgs that referenced this pull request Sep 18, 2017
@rvl
pythonPackages.spacy_models: fix evaluation
b7e602a 
PR NixOS#29334
@rvl
Copy link
Contributor Author

rvl commented Sep 18, 2017

Very sorry @FRidh, it appears I forgot to retest the spacy_models packages. Would you be able to apply this patch? rvl@b7e602a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh FRidh requested changes

Assignees
No one assigned
Labels
6.topic: python
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@rvl @mention-bot @FRidh