Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python3Packages.pytorch: remove oneDNN dependency #96350

Merged
merged 1 commit into from Aug 29, 2020
Merged

python3Packages.pytorch: remove oneDNN dependency #96350

merged 1 commit into from Aug 29, 2020

Conversation

danieldk
Copy link
Contributor

Motivation for this change

oneDNN was added as a dependency, but it is not actually used by
PyTorch. PyTorch uses oneDNN from the vendored iDeep dependency.

Using a system-provided oneDNN is currently not a supported build
option.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@danieldk danieldk changed the title python3Packages: remove oneDNN dependency python3Packages.pytorch: remove oneDNN dependency Aug 26, 2020
@danieldk
python3Packages.pytorch: remove oneDNN dependency
a4ed797 
oneDNN was added as a dependency, but it is not actually used by
PyTorch. PyTorch uses oneDNN from the vendored iDeep dependency.

Using a system-provided oneDNN is currently not a supported build
option.
@danieldk danieldk force-pushed the pytorch-remove-onednn-dependency branch from 361e1e5 to a4ed797 Compare August 26, 2020 08:56
@danieldk
Copy link
Contributor Author

@ofborg build python37Packages.pytorch python38Packages.pytorch

@ofborg ofborg bot requested review from teh and thoughtpolice August 26, 2020 09:34
@danieldk
Copy link
Contributor Author

cc: @alexarice @bhipple

bhipple
bhipple approved these changes Aug 29, 2020
View reviewed changes
pkgs/development/python-modules/pytorch/default.nix
# pytorch still calls it by the old name mkldnn.
# Unlike MKL, oneDNN (née MKLDNN) is FOSS, so we enable support for
# it by default. PyTorch currently uses its own vendored version
# of oneDNN through Intel iDeep.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sigh vendored dependencies . . . I guess it is what it is though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's a bit annoying that they do not provide some option to provide your own oneDNN. Doesn't help that CMake is hidden behind their own script that gates build flags :/.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be really nice if you could contribute a patch upstream to support non-vendored distribution deps, but in the meantime this PR seems OK as it at least faithfully represents the status quo.

Copy link
Contributor Author

@danieldk danieldk Aug 29, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will do that if I find some time. First some more BLAS implementation fights, erm, I mean benchmarks. Intel has removed MKL_DEBUG_CPU_TYPE since MKL 2020 release 1, so Ryzen performance is abysmal. There is a workaround, but not too happy with it ;).

@danieldk danieldk merged commit 3e2dfb9 into NixOS:master Aug 29, 2020
@danieldk danieldk deleted the pytorch-remove-onednn-dependency branch August 29, 2020 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bhipple bhipple bhipple approved these changes

@FRidh FRidh Awaiting requested review from FRidh

@jonringer jonringer Awaiting requested review from jonringer

@teh teh Awaiting requested review from teh

@thoughtpolice thoughtpolice Awaiting requested review from thoughtpolice

Assignees
No one assigned
Labels
6.topic: python 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@danieldk @bhipple