Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python2Packages.spacy_models: update all models to version 2.0.0 #46182

Merged
merged 1 commit into from Sep 7, 2018

Conversation

danieldk
Copy link
Contributor

@danieldk danieldk commented Sep 6, 2018

Motivation for this change

Update the model list to those models that are compatibly with the
latest spaCy version:

https://github.com/explosion/spacy-models/blob/master/compatibility.json

Tested loading and running one of the models to annotate some text.

The updated model file evaluates to the following packages:

python2Packages.spacy_models.de_core_news_sm
python2Packages.spacy_models.en_core_web_lg
python2Packages.spacy_models.en_core_web_md
python2Packages.spacy_models.en_core_web_sm
python2Packages.spacy_models.en_vectors_web_lg
python2Packages.spacy_models.es_core_news_md
python2Packages.spacy_models.es_core_news_sm
python2Packages.spacy_models.fr_core_news_md
python2Packages.spacy_models.fr_core_news_sm
python2Packages.spacy_models.it_core_news_sm
python2Packages.spacy_models.nl_core_news_sm
python2Packages.spacy_models.pt_core_news_sm
python2Packages.spacy_models.xx_ent_wiki_sm
python3Packages.spacy_models.de_core_news_sm
python3Packages.spacy_models.en_core_web_lg
python3Packages.spacy_models.en_core_web_md
python3Packages.spacy_models.en_core_web_sm
python3Packages.spacy_models.en_vectors_web_lg
python3Packages.spacy_models.es_core_news_md
python3Packages.spacy_models.es_core_news_sm
python3Packages.spacy_models.fr_core_news_md
python3Packages.spacy_models.fr_core_news_sm
python3Packages.spacy_models.it_core_news_sm
python3Packages.spacy_models.nl_core_news_sm
python3Packages.spacy_models.pt_core_news_sm
python3Packages.spacy_models.xx_ent_wiki_sm

ZHF #45960

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

Update the model list to those models that are compatibly with the
latest spaCy version:

https://github.com/explosion/spacy-models/blob/master/compatibility.json

The updated model file evaluates to the following packages:

python2Packages.spacy_models.de_core_news_sm
python2Packages.spacy_models.en_core_web_lg
python2Packages.spacy_models.en_core_web_md
python2Packages.spacy_models.en_core_web_sm
python2Packages.spacy_models.en_vectors_web_lg
python2Packages.spacy_models.es_core_news_md
python2Packages.spacy_models.es_core_news_sm
python2Packages.spacy_models.fr_core_news_md
python2Packages.spacy_models.fr_core_news_sm
python2Packages.spacy_models.it_core_news_sm
python2Packages.spacy_models.nl_core_news_sm
python2Packages.spacy_models.pt_core_news_sm
python2Packages.spacy_models.xx_ent_wiki_sm
python3Packages.spacy_models.de_core_news_sm
python3Packages.spacy_models.en_core_web_lg
python3Packages.spacy_models.en_core_web_md
python3Packages.spacy_models.en_core_web_sm
python3Packages.spacy_models.en_vectors_web_lg
python3Packages.spacy_models.es_core_news_md
python3Packages.spacy_models.es_core_news_sm
python3Packages.spacy_models.fr_core_news_md
python3Packages.spacy_models.fr_core_news_sm
python3Packages.spacy_models.it_core_news_sm
python3Packages.spacy_models.nl_core_news_sm
python3Packages.spacy_models.pt_core_news_sm
python3Packages.spacy_models.xx_ent_wiki_sm
@xeji
Copy link
Contributor

xeji commented Sep 7, 2018

Zero rebuilds - does that look correct? Aren't these models used by any package?

@danieldk
Copy link
Contributor Author

danieldk commented Sep 7, 2018

I am not the maintainer/author, but I think the idea was that these models are too large to include as a dependency to e.g. spaCy. So, you install spaCy and then install the specific models that you need.

@xeji
Copy link
Contributor

xeji commented Sep 7, 2018

@GrahamcOfBorg build python2Packages.spacy_models.en_core_web_lg python3Packages.spacy_models.en_core_web_lg

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python2Packages.spacy_models.en_core_web_lg, python3Packages.spacy_models.en_core_web_lg

Partial log (click to expand)

reading manifest template 'MANIFEST.in'
writing manifest file 'en_core_web_lg.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
/nix/store/hji7b6h4kb19rzjjinxgfz6i5vp4839n-python2.7-en_core_web_lg-2.0.0
/nix/store/qn0hcnsfm5yy1bvw8yy2m3604iw7qxg7-python3.6-en_core_web_lg-2.0.0

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: python2Packages.spacy_models.en_core_web_lg, python3Packages.spacy_models.en_core_web_lg

Partial log (click to expand)

KeyError: 'model'
builder for '/nix/store/3xj8sci324bcbrkqdicfkk0pz4vkvhj9-python3.6-thinc-6.11.2.drv' failed with exit code 1
cannot build derivation '/nix/store/8p2iwrkhsip703j9gx5ah75q8wjzdsma-python3.6-spacy-2.0.12.drv': 1 dependencies couldn't be built
 66  812M   66  541M    0     0  1743k      0  0:07:57  0:05:18  0:02:39     0
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
error: cannot download en_core_web_lg-2.0.0.tar.gz from any mirror
builder for '/nix/store/kwf8sa0nil4namigsy5vbi96w0a1mkrn-en_core_web_lg-2.0.0.tar.gz.drv' failed with exit code 1
cannot build derivation '/nix/store/03kpyy7jwm4c2mvyyv5fmdb5p0fh7g1k-python2.7-en_core_web_lg-2.0.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/ffn77zzxyawlhpfrr6lsdrnd7vd0xl7k-python3.6-en_core_web_lg-2.0.0.drv': 2 dependencies couldn't be built
error: build of '/nix/store/03kpyy7jwm4c2mvyyv5fmdb5p0fh7g1k-python2.7-en_core_web_lg-2.0.0.drv', '/nix/store/ffn77zzxyawlhpfrr6lsdrnd7vd0xl7k-python3.6-en_core_web_lg-2.0.0.drv' failed

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: python2Packages.spacy_models.en_core_web_lg, python3Packages.spacy_models.en_core_web_lg

Partial log (click to expand)

/nix/store/nml2n9vgmx1mldzxf5zq0ksafwy82vzb-python2.7-en_core_web_lg-2.0.0
/nix/store/d76gcxcqiz1n8nrg2mcn4s2lai0y2ipa-python3.6-en_core_web_lg-2.0.0

@xeji xeji merged commit d261df5 into NixOS:master Sep 7, 2018
xeji pushed a commit that referenced this pull request Sep 7, 2018
)

Update the model list to those models that are compatibly with the
latest spaCy version:

https://github.com/explosion/spacy-models/blob/master/compatibility.json

The updated model file evaluates to the following packages:

python2Packages.spacy_models.de_core_news_sm
python2Packages.spacy_models.en_core_web_lg
python2Packages.spacy_models.en_core_web_md
python2Packages.spacy_models.en_core_web_sm
python2Packages.spacy_models.en_vectors_web_lg
python2Packages.spacy_models.es_core_news_md
python2Packages.spacy_models.es_core_news_sm
python2Packages.spacy_models.fr_core_news_md
python2Packages.spacy_models.fr_core_news_sm
python2Packages.spacy_models.it_core_news_sm
python2Packages.spacy_models.nl_core_news_sm
python2Packages.spacy_models.pt_core_news_sm
python2Packages.spacy_models.xx_ent_wiki_sm
python3Packages.spacy_models.de_core_news_sm
python3Packages.spacy_models.en_core_web_lg
python3Packages.spacy_models.en_core_web_md
python3Packages.spacy_models.en_core_web_sm
python3Packages.spacy_models.en_vectors_web_lg
python3Packages.spacy_models.es_core_news_md
python3Packages.spacy_models.es_core_news_sm
python3Packages.spacy_models.fr_core_news_md
python3Packages.spacy_models.fr_core_news_sm
python3Packages.spacy_models.it_core_news_sm
python3Packages.spacy_models.nl_core_news_sm
python3Packages.spacy_models.pt_core_news_sm
python3Packages.spacy_models.xx_ent_wiki_sm

(cherry picked from commit d261df5)
@xeji
Copy link
Contributor

xeji commented Sep 7, 2018

Backported: 4f475fe

@danieldk danieldk deleted the spacy-models branch June 4, 2019 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants