pythonPackages.wordfreq and dependencies #37784

ixxie · 2018-03-25T18:58:46Z

Initialized the python wordfreq module at 2.0, with dependencies.

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

ixxie · 2018-03-27T07:46:49Z

@dotlambda - got time to review this?

dotlambda

Sorry for the late reply! I was on vacation.

dotlambda · 2018-04-03T08:13:44Z

pkgs/development/python-modules/langcodes/default.nix

+  disabled = pythonOlder "3.3";
+
+  meta = with lib; {
+    description = "langcodes is a toolkit for working with and comparing the standardized codes for languages, such as ‘en’ for English, ‘es’ for Spanish, and ‘zh-Hant’ for Traditional Chinese.";


https://nixos.org/nixpkgs/manual/#sec-standard-meta-attributes

dotlambda · 2018-04-03T08:14:09Z

pkgs/development/python-modules/mecab-python3/default.nix

+   src = fetchPypi {
+    inherit pname version;
+    sha256 = "007dg4f5fby2yl7cc44x6xwvcrf2w2ifmn0rmk56ss33mhs8l6qy";
+   };


indentation

dotlambda · 2018-04-03T08:14:14Z

pkgs/development/python-modules/mecab-python3/default.nix

+  pname = "mecab-python3";
+  version = "0.7";
+
+


dotlambda · 2018-04-03T08:15:44Z

pkgs/development/python-modules/wordfreq/default.nix

+
+  # patch to relax version requirements for regex
+  # dependency to prevent break in upgrade
+  patches = [ ./relax-regex-dep.patch ];


I'd prefer using substituteInPlace in postPatch. Something like

postPatch = '' substituteInPlace setup.py \ --replace "regex ==" "regex >=" '';

dotlambda · 2018-04-03T08:20:43Z

@GrahamcOfBorg build python2.pkgs.wordfreq python3.pkgs.wordfreq python3.pkgs.langcodes python2.pkgs.mecab-python3 python3.pkgs.mecab-python3 python2.pkgs.marisa-trie python3.pkgs.marisa-trie

GrahamcOfBorg · 2018-04-03T08:21:40Z

Failure on aarch64-linux (full log)

Attempted: python3.pkgs.wordfreq, python3.pkgs.langcodes, python2.pkgs.mecab-python3, python3.pkgs.mecab-python3, python2.pkgs.marisa-trie, python3.pkgs.marisa-trie

The following builds were skipped because they don't evaluate on aarch64-linux: python2.pkgs.wordfreq

Partial log (click to expand)

writing manifest file 'mecab_python3.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-aarch64-3.6/_MeCab.cpython-36m-aarch64-linux-gnu.so ->

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
cannot build derivation '/nix/store/vnjrbmk6zkxgck2d2p85q9l4vnqjvpwz-python3.6-wordfreq-2.0.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/0hdq6pv5gihb02yq49hm0v64ri2xkh5y-python3.6-marisa-trie-0.7.4.drv', '/nix/store/bxgif1ihrmk15mfianiw26yc290sv0qz-python3.6-langcodes-1.4.1.drv', '/nix/store/vnjrbmk6zkxgck2d2p85q9l4vnqjvpwz-python3.6-wordfreq-2.0.drv', '/nix/store/wwilj8w8r96sfzrfpdkmvxxjya5m0h80-python2.7-marisa-trie-0.7.4.drv' failed

GrahamcOfBorg · 2018-04-03T08:21:49Z

Failure on x86_64-linux (full log)

Attempted: python3.pkgs.wordfreq, python3.pkgs.langcodes, python2.pkgs.mecab-python3, python3.pkgs.mecab-python3, python2.pkgs.marisa-trie, python3.pkgs.marisa-trie

The following builds were skipped because they don't evaluate on x86_64-linux: python2.pkgs.wordfreq

Partial log (click to expand)

-

1
0
.
2
M

cannot build derivation ‘/nix/store/wyjqq931z4yfd6bx5bajc147l0hfc68b-python3.6-wordfreq-2.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/l7rm4d50cxs0a7xmm2zly1wz41dhp9fg-python2.7-marisa-trie-0.7.4.drv’, ‘/nix/store/rqfsbiqlv0l0rw5hy9a97vz3nswy3q44-python3.6-marisa-trie-0.7.4.drv’, ‘/nix/store/sxfmzhb0l10dxmlys5mz6p5kfmqsmlaf-python3.6-langcodes-1.4.1.drv’, ‘/nix/store/wyjqq931z4yfd6bx5bajc147l0hfc68b-python3.6-wordfreq-2.0.drv’ failed

dotlambda · 2018-04-03T08:23:39Z

pkgs/development/python-modules/marisa-trie/default.nix

+  };
+
+  meta = with lib; {
+    description = "Static memory-efficient Trie-like structures for Python (2.x and 3.x) based on marisa-trie C++ library.";


Please explain the difference between this and pythonPackages.marisa in longDescription:

There are official SWIG-based Python bindings included in C++ library distribution; this package provides alternative Cython-based pip-installable Python bindings.

dotlambda · 2018-04-03T08:25:10Z

Ran 0 tests in 0.000s

Please make sure every package has an appropriate checkPhase if it provides tests. Otherwise, please set doCheck = false and add an inline comment with the reason.

ixxie · 2018-04-04T21:02:55Z

@dotlambda I think I managed to address all your concerns but one: I tried to get wordfreq's tests going by switching to fetchFromGitHub and triggering nose tests like you instructed me to do for hdbscan. However the tests fail:

OSError: Couldn't find the MeCab dictionary named 'mecab-ko-dic'.
You should download or use your system's package manager to install
the 'mecab-ko-dic' package.

We looked in the following locations:
    /homeless-shelter/.local/lib/mecab/dic/mecab-ko-dic
    /homeless-shelter/.local/lib/mecab/dic/ko-dic
    /var/lib/mecab/dic/mecab-ko-dic
    /var/lib/mecab/dic/ko-dic
    /var/local/lib/mecab/dic/mecab-ko-dic
    /var/local/lib/mecab/dic/ko-dic
    /usr/lib/mecab/dic/mecab-ko-dic
    /usr/lib/mecab/dic/ko-dic
    /usr/local/lib/mecab/dic/mecab-ko-dic
    /usr/local/lib/mecab/dic/ko-dic

Name                        Stmts   Miss  Cover
-----------------------------------------------
wordfreq/__init__.py          116     11    91%
wordfreq/language_info.py      32      0   100%
wordfreq/mecab.py              33      9    73%
wordfreq/preprocess.py         35      0   100%
wordfreq/tokens.py             50      7    86%
wordfreq/transliterate.py       9      1    89%
-----------------------------------------------
TOTAL                         275     28    90%
----------------------------------------------------------------------
Ran 31 tests in 5.301s

The first hit on Google is all in Korean, which makes sense for a Korean dictionary package. I guess since wordfreq looks for this inside mecab's install directory, I guess it may be some sort of plugin or extension to the main package, but I am uncertain.

I am not sure how to proceed at the moment, perhaps you have an idea?

dotlambda · 2018-04-04T22:11:17Z

I updated all expressions so that the tests pass and so that all packages are tested. Please have a look.

The Korean and Japanese dictionaries are indeed optional dependencies of wordfreq (cf. https://github.com/LuminosoInsight/wordfreq#additional-cjk-installation), so I disabled the respective tests.

ixxie · 2018-04-05T08:31:50Z

Awesome, so I guess we are done :)

ixxie requested a review from FRidh as a code owner March 25, 2018 18:58

dotlambda requested changes Apr 3, 2018

View reviewed changes

dotlambda reviewed Apr 3, 2018

View reviewed changes

GrahamcOfBorg added the 6.topic: python label Apr 4, 2018

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Apr 4, 2018

Matan Shenhav and others added 5 commits April 5, 2018 00:02

pythonPackages.mecab-python3: init at 0.7

3d93a9b

pythonPackages.marisa-trie: init at 0.7.4

8cd264f

pythonPackages.langcodes: init at 1.4.1

d59fafd

pythonPackages.jieba: init at 0.39

ad5ff7b

pythonPackages.wordfreq: init at 2.0

1f82712

dotlambda merged commit addb985 into NixOS:master Apr 5, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.wordfreq and dependencies #37784

pythonPackages.wordfreq and dependencies #37784

ixxie commented Mar 25, 2018

ixxie commented Mar 27, 2018

dotlambda left a comment

dotlambda Apr 3, 2018

dotlambda Apr 3, 2018

dotlambda Apr 3, 2018

dotlambda Apr 3, 2018

dotlambda commented Apr 3, 2018

GrahamcOfBorg commented Apr 3, 2018

GrahamcOfBorg commented Apr 3, 2018

dotlambda Apr 3, 2018

dotlambda commented Apr 3, 2018

ixxie commented Apr 4, 2018 •

edited

dotlambda commented Apr 4, 2018 •

edited

ixxie commented Apr 5, 2018

pythonPackages.wordfreq and dependencies #37784

pythonPackages.wordfreq and dependencies #37784

Conversation

ixxie commented Mar 25, 2018

ixxie commented Mar 27, 2018

dotlambda left a comment

Choose a reason for hiding this comment

dotlambda Apr 3, 2018

Choose a reason for hiding this comment

dotlambda Apr 3, 2018

Choose a reason for hiding this comment

dotlambda Apr 3, 2018

Choose a reason for hiding this comment

dotlambda Apr 3, 2018

Choose a reason for hiding this comment

dotlambda commented Apr 3, 2018

GrahamcOfBorg commented Apr 3, 2018

GrahamcOfBorg commented Apr 3, 2018

dotlambda Apr 3, 2018

Choose a reason for hiding this comment

dotlambda commented Apr 3, 2018

ixxie commented Apr 4, 2018 • edited

dotlambda commented Apr 4, 2018 • edited

ixxie commented Apr 5, 2018

ixxie commented Apr 4, 2018 •

edited

dotlambda commented Apr 4, 2018 •

edited