Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python3Packages.tokenizers: 0.9.4 -> 0.10.0 #109532

Merged
merged 1 commit into from Jan 23, 2021
Merged

python3Packages.tokenizers: 0.9.4 -> 0.10.0 #109532

merged 1 commit into from Jan 23, 2021

Conversation

danieldk
Copy link
Contributor

Motivation for this change

Changelog:
https://github.com/huggingface/tokenizers/releases/tag/python-v0.10.0

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@jonringer
Copy link
Contributor

seems to fail for me locally:

https://github.com/NixOS/nixpkgs/pull/109532

4 packages failed to build:
python37Packages.tokenizers python37Packages.transformers python38Packages.tokenizers python38Packages.transformers

/nix/store/yi9a1fafijn0nf5m8qkmnl955vi2mncm-python3-3.8.6/lib/python3.8/os.py:213: in makedirs
    makedirs(head, exist_ok=exist_ok)
/nix/store/yi9a1fafijn0nf5m8qkmnl955vi2mncm-python3-3.8.6/lib/python3.8/os.py:223: in makedirs
    mkdir(name, mode)
E   PermissionError: [Errno 13] Permission denied: '/homeless-shelter'

jonringer
jonringer reviewed Jan 16, 2021
View reviewed changes
pkgs/development/python-modules/tokenizers/default.nix
@@ -105,6 +107,11 @@ in rustPlatform.buildRustPackage rec {
pipInstallPhase
'';

disabledTests = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
disabledTests = [
preCheck = ''
HOME=$TMPDIR
'';
disabledTests = [ 
/nix/store/yi9a1fafijn0nf5m8qkmnl955vi2mncm-python3-3.8.6/lib/python3.8/os.py:213: in makedirs
    makedirs(head, exist_ok=exist_ok)
/nix/store/yi9a1fafijn0nf5m8qkmnl955vi2mncm-python3-3.8.6/lib/python3.8/os.py:223: in makedirs
    mkdir(name, mode)
E   PermissionError: [Errno 13] Permission denied: '/homeless-shelter'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, and sorry for the delay (busy week). Should be fixed now.

@SuperSandro2000
Copy link
Member

Result of nixpkgs-review pr 109532 run on x86_64-linux 1

4 packages failed to build:
  • python37Packages.tokenizers
  • python37Packages.transformers
  • python38Packages.tokenizers
  • python38Packages.transformers

@SuperSandro2000
Copy link
Member

Result of nixpkgs-review pr 109532 run on x86_64-darwin 1

4 packages failed to build:
  • python37Packages.tokenizers
  • python37Packages.transformers
  • python38Packages.tokenizers
  • python38Packages.transformers
rootdir: /private/tmp/nix-build-tokenizers-0.10.0.drv-0/source/bindings/python
collected 117 items / 1 error / 116 selected

==================================== ERRORS ====================================
__ ERROR collecting tests/documentation/test_tutorial_train_from_iterators.py __
tests/documentation/test_tutorial_train_from_iterators.py:4: in <module>
    import datasets
/nix/store/janmd6006jr9kmxipkc22g0g09b9j0n7-python3.8-datasets-1.1.2/lib/python3.8/site-packages/datasets/__init__.py:27: in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
/nix/store/janmd6006jr9kmxipkc22g0g09b9j0n7-python3.8-datasets-1.1.2/lib/python3.8/site-packages/datasets/arrow_dataset.py:40: in <module>
    from .arrow_reader import ArrowReader
/nix/store/janmd6006jr9kmxipkc22g0g09b9j0n7-python3.8-datasets-1.1.2/lib/python3.8/site-packages/datasets/arrow_reader.py:31: in <module>
    from .utils import cached_path, logging
/nix/store/janmd6006jr9kmxipkc22g0g09b9j0n7-python3.8-datasets-1.1.2/lib/python3.8/site-packages/datasets/utils/__init__.py:20: in <module>
    from .download_manager import DownloadManager, GenerateMode
/nix/store/janmd6006jr9kmxipkc22g0g09b9j0n7-python3.8-datasets-1.1.2/lib/python3.8/site-packages/datasets/utils/download_manager.py:25: in <module>
    from .file_utils import HF_DATASETS_CACHE, cached_path, get_from_cache, hash_url_to_filename
/nix/store/janmd6006jr9kmxipkc22g0g09b9j0n7-python3.8-datasets-1.1.2/lib/python3.8/site-packages/datasets/utils/file_utils.py:118: in <module>
    os.makedirs(HF_MODULES_CACHE, exist_ok=True)
/nix/store/wsv9iz7p87a72blxkj24828vm4ibb9in-python3-3.8.6/lib/python3.8/os.py:213: in makedirs
    makedirs(head, exist_ok=exist_ok)
/nix/store/wsv9iz7p87a72blxkj24828vm4ibb9in-python3-3.8.6/lib/python3.8/os.py:213: in makedirs
    makedirs(head, exist_ok=exist_ok)
/nix/store/wsv9iz7p87a72blxkj24828vm4ibb9in-python3-3.8.6/lib/python3.8/os.py:213: in makedirs
    makedirs(head, exist_ok=exist_ok)
/nix/store/wsv9iz7p87a72blxkj24828vm4ibb9in-python3-3.8.6/lib/python3.8/os.py:223: in makedirs
    mkdir(name, mode)
E   PermissionError: [Errno 1] Operation not permitted: '/homeless-shelter'
=============================== warnings summary ===============================
tests/bindings/test_processors.py:150
  /private/tmp/nix-build-tokenizers-0.10.0.drv-0/source/bindings/python/tests/bindings/test_processors.py:150: DeprecationWarning: invalid escape sequence \(
    with pytest.raises(Exception, match="Missing SpecialToken\(s\) with id\(s\)"):

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
ERROR tests/documentation/test_tutorial_train_from_iterators.py - PermissionE...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
========================= 1 warning, 1 error in 2.78s ==========================
builder for '/nix/store/hkbpqdw9f0p77rwy4g1l2yg8in82jp5v-tokenizers-0.10.0.drv' failed with exit code 2
cannot build derivation '/nix/store/zix4mw1ar77xjrjdr66jsx4d8s37vaw0-python3.8-transformers-4.1.1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/zix4mw1ar77xjrjdr66jsx4d8s37vaw0-python3.8-transformers-4.1.1.drv' failed

@symphorien
Copy link
Member

this breaks python3Packages.transformers. Presumably because of overly restrictive pinning, but as maintainer you probably know better.

@danieldk
Copy link
Contributor Author

this breaks python3Packages.transformers. Presumably because of overly restrictive pinning, but as maintainer you probably know better.

Indeed, transformers pins tokenizers to a specific version. We do relax the bounds in transformers, but it was too specific. Now also fixed as part of #109505 . Thanks!

@symphorien symphorien merged commit 4f33765 into NixOS:master Jan 23, 2021
@danieldk danieldk deleted the tokenizers-0.10.0 branch January 23, 2021 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jonringer jonringer jonringer left review comments

@FRidh FRidh Awaiting requested review from FRidh

Assignees
No one assigned
Labels
6.topic: python 8.has: clean-up 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 11.by: package-maintainer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@danieldk @jonringer @SuperSandro2000 @symphorien