Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python3Packages.transformers: 2.2.1 -> 3.0.1 #91915

Merged
merged 2 commits into from Jul 3, 2020
Merged

python3Packages.transformers: 2.2.1 -> 3.0.1 #91915

merged 2 commits into from Jul 3, 2020

Conversation

danieldk
Copy link
Contributor

@danieldk danieldk commented Jul 1, 2020 •
edited

Motivation for this change

Changelog:
https://github.com/huggingface/transformers/releases/tag/v3.0.0

Also add myself as a maintainer.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

cc: @pashashocky

@danieldk
Copy link
Contributor Author

danieldk commented Jul 1, 2020

@ofborg build python37Packages.transformers python38Packages.transformers

@danieldk danieldk marked this pull request as ready for review July 1, 2020 11:15
drewrisinger
drewrisinger suggested changes Jul 1, 2020
View reviewed changes
Copy link
Contributor

@drewrisinger drewrisinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest using pytestCheckHook for disabling these tests.

pkgs/development/python-modules/transformers/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/transformers/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/transformers/default.nix Outdated Show resolved Hide resolved
drewrisinger
drewrisinger suggested changes Jul 2, 2020
View reviewed changes
Copy link
Contributor

@drewrisinger drewrisinger left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python38.transformers: Willing to accept, maybe include pytorch/tensorflow in inputs?? No strong opinion. (also, the spacing in this error message makes me want to file a PR... grr :D )

>>> import transformers
Neither PyTorch nor TensorFlow >= 2.0 have been found.Models won't be available and only tokenizers, configurationand file/data utilities can be used.

python37Packages.transformers: getting the same issue when launched from command-line or via transformers-cli. Not sure what's causing this, seems like it's trying to use py38.numpy for some reason??

[nix-shell:~/.cache/nixpkgs-review/pr-91915/results/python37Packages.transformers]$ ./bin/transformers-cli
Traceback (most recent call last):
  File "/nix/store/0ryfy7j7azg5sgnh75sxlxbkbnd9vvbj-python3.8-numpy-1.18.5/lib/python3.8/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/nix/store/0ryfy7j7azg5sgnh75sxlxbkbnd9vvbj-python3.8-numpy-1.18.5/lib/python3.8/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/nix/store/0ryfy7j7azg5sgnh75sxlxbkbnd9vvbj-python3.8-numpy-1.18.5/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/rsxk262fy8xv1dsmpqb6klbw4sf1q4hf-python3.7-transformers-3.0.0/bin/.transformers-cli-wrapped", line 6, in <module>
    from transformers.commands.transformers_cli import main
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/__init__.py", line 48, in <module>
    from .data import (
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/data/__init__.py", line 6, in <module>
    from .processors import (
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/data/processors/__init__.py", line 5, in <module>
    from .glue import glue_convert_examples_to_features, glue_output_modes, glue_processors, glue_tasks_num_labels
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/data/processors/glue.py", line 24, in <module>
    from ...tokenization_utils import PreTrainedTokenizer
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 26, in <module>
    from .tokenization_utils_base import (
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 30, in <module>
    import numpy as np
  File "/nix/store/0ryfy7j7azg5sgnh75sxlxbkbnd9vvbj-python3.8-numpy-1.18.5/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/nix/store/0ryfy7j7azg5sgnh75sxlxbkbnd9vvbj-python3.8-numpy-1.18.5/lib/python3.8/site-packages/numpy/core/__init__.py", line 50, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "/nix/store/73iskwap4wblkwbq2jy9cndmrn49n6ss-python3-3.7.7/bin/python3.7"
  * The NumPy version is: "1.18.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

@danieldk
Copy link
Contributor Author

danieldk commented Jul 2, 2020 

* [ ]  Commits: Squash add maintainer. Remove tokenizers when #91342 is approved

I think it's nice to have that in a commit separate from the version bump.

python38.transformers: Willing to accept, maybe include pytorch/tensorflow in inputs?? No strong opinion. (also, the

@pashashocky probably excluded these dependencies intentionally. transformers works with either, so I think it's up to the user to pick one. We use transformers with PyTorch and it would be nasty if we had to pull in Tensorflow as an additional dependency every time. Also Tensorflow has had quite a history of being broken frequently in nixpkgs.

spacing in this error message makes me want to file a PR... grr :D )

😆

python37Packages.transformers: getting the same issue when launched from command-line or via transformers-cli. Not sure what's causing this, seems like it's trying to use py38.numpy for some reason??

Let me check.

@danieldk
Copy link
Contributor Author

danieldk commented Jul 2, 2020 •
edited

python37Packages.transformers: getting the same issue when launched from command-line or via transformers-cli. Not sure what's causing this, seems like it's trying to use py38.numpy for some reason??

Let me check.

The command seems to work fine?

> $(nix-build --no-out-link -A python3Packages.transformers)/bin/transformers-cli
Neither PyTorch nor TensorFlow >= 2.0 have been found.Models won't be available and only tokenizers, configurationand file/data utilities can be used.
usage: transformers-cli <command> [<args>]

positional arguments:
  {convert,download,env,run,serve,login,whoami,logout,s3,upload}
[...]

drewrisinger
drewrisinger approved these changes Jul 2, 2020
View reviewed changes
Copy link
Contributor

@drewrisinger drewrisinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirm that everything looks fine & builds. Approved pending removal of python3Packages.tokenizers commit from this PR.

@danieldk
Copy link
Contributor Author

danieldk commented Jul 2, 2020

Rebased now that python3Packages.tokenizers is merged.

drewrisinger
drewrisinger approved these changes Jul 2, 2020
View reviewed changes
Copy link
Contributor

@drewrisinger drewrisinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Diff LGTM
  • Commits LGTM, minus subjective style of adding maintainer in separate commit
  • Builds clean via nix-review
  • tested via pytest & trivially w/ transformer-cli (same issue with python37Packages.transformers-cli as before, maybe re-test with that command & python37Packges.transformers in your test, vs python3Packages.transformers).

@danieldk
Copy link
Contributor Author

danieldk commented Jul 3, 2020 •
edited

  • tested via pytest & trivially w/ transformer-cli (same issue with python37Packages.transformers-cli as before, maybe re-test with that command & python37Packges.transformers in your test, vs python3Packages.transformers).

Ah, I see the problem you are running into. If you run nix-review, the Python 3.8 modules end up in PYTHONPATH. Try to run nix-review as follows: nix-review pr 91915 -p python37Packages.transformers

Then transformers-cli should run as expected.

@danieldk danieldk changed the title python3Packages.transformers: 2.2.1 -> 3.0.0 python3Packages.transformers: 2.2.1 -> 3.0.1 Jul 3, 2020
@danieldk
Copy link
Contributor Author

danieldk commented Jul 3, 2020

Updated to transformers 3.0.1.

@FRidh FRidh merged commit 4855aa6 into NixOS:master Jul 3, 2020
@danieldk danieldk deleted the transformers-3.0.0 branch July 3, 2020 16:57
@drewrisinger
Copy link
Contributor

@danieldk yeah, I figured something like that was happening, but I didn't have time to debug it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@drewrisinger drewrisinger drewrisinger approved these changes

@FRidh FRidh Awaiting requested review from FRidh

@jonringer jonringer Awaiting requested review from jonringer

Assignees
No one assigned
Labels
6.topic: python 8.has: clean-up 8.has: package (new) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 11.by: package-maintainer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@danieldk @drewrisinger @FRidh