Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python2Packages.google_resumable_media: fix tests #97768

Merged
merged 3 commits into from Sep 23, 2020
Merged

python2Packages.google_resumable_media: fix tests #97768

merged 3 commits into from Sep 23, 2020

Conversation

freezeboy
Copy link
Contributor

@freezeboy freezeboy commented Sep 11, 2020 •
edited

Motivation for this change

Adding crcmod for tests
ZHF: #97479

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@ofborg ofborg bot requested a review from costrouc September 11, 2020 13:33
@doronbehar
Copy link
Contributor

@GrahamcOfBorg build python2Packages.google_resumable_media

@risicle
Copy link
Contributor

risicle commented Sep 13, 2020

@ofborg build python37Packages.google_resumable_media

@risicle
Copy link
Contributor

risicle commented Sep 13, 2020

The python3Packages version of this now fails to build on linux or macos for me (and ofborg)

...
  Executing pipInstallPhase
  /build/google-resumable-media-0.7.1/dist /build/google-resumable-media-0.7.1
  Processing ./google_resumable_media-0.7.1-py2.py3-none-any.whl
  ERROR: Could not find a version that satisfies the requirement google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5" (from google-resumable-media==0.7.1) (from versions: none)
  ERROR: No matching distribution found for google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5" (from google-resumable-media==0.7.1)

@freezeboy
Copy link
Contributor Author

Ok I'll add google-crc32c

@freezeboy
Copy link
Contributor Author

Had to add new package and update google_resumable_media to have the tests working

@ofborg ofborg bot requested a review from andir September 13, 2020 12:07
risicle
risicle reviewed Sep 13, 2020
View reviewed changes
pkgs/development/python-modules/google-crc32c/default.nix
export LDFLAGS="-L${pkgs.crc32c}/lib"
export CFLAGS="-I${pkgs.crc32c}/include"
'';

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, we're probably overwhelming you here but if you do

  src = fetchFromGitHub {
    owner = "googleapis";
    repo = "python-crc32c";
    rev = "v${version}";
    sha256 = "0n3ggsxmk1fhq0kz6p5rcj4gypfb05i26fcn7lsawakgl7fzxqyl";
  };

  ...

  checkInputs = [ pytestCheckHook ];
  pythonImportsCheck = [ "google_crc32c" ];

we should get the tests working. We should at least have the pythonImportsCheck otherwise we have zero warning that this package is broken.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no probem always love to add more tests

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hum it broke the tests, apparently the c library is no more found

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like it is the LDFLAGS/CFLAGS changes you proposed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok it is the pytestCheckHook that fails, it cannot load the module when this hook is enabled

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh .. I tried to force c++11 and it told me it is C, not C++, that must be the problem

Copy link
Contributor

@risicle risicle Sep 13, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup we're finding this in parallel re the shared lib - it works if we set cmakeFlags = [ "-DBUILD_SHARED_LIBS=1" ]; in pkgs.crc32c. The annoying thing is it won't build both the static and shared libs at the same time, and I'm not sure we want to take away the static lib from packages that might be happily using it. Just checking how many packages it actually affects...

Copy link
Contributor

@risicle risicle Sep 13, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what we do is add a shared argument to pkgs/development/libraries/crc32c/default.nix, defaulting to false, which just does a cmakeFlags = [ "-DBUILD_SHARED_LIBS=${if shared then "1" else "0"}" ];. Then we just pass crc32c.override { shared = true; } to this package in python-packages.nix.

Thoughts @jonringer ?

Copy link
Contributor

@jonringer jonringer Sep 20, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

usually you opt into static.

I would add a staticOnly ? false input.

Then in pkgs/top-level/static.nix you would add:

  crc = super.crc.override {
    staticOnly = true;
  };

and adjust the build flags accordingly

that way, when you do pkgsStatic.crc32 you get the .a, but normally you get the .so

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applied these fixes and built locally successfully 🎉 :

  • pkgs.crc32c
  • pkgsStatic.crc32c
  • pkgs.python2Packages.google_resumable_media
  • pkgs.python37Packages.google_resumable_media
  • pkgs.python38Packages.google_resumable_media

@freezeboy freezeboy force-pushed the fix-google_resumable_media branch 2 times, most recently from 4cff45f to c0bb59c Compare September 20, 2020 16:01
@jonringer
Copy link
Contributor

jonringer commented Sep 20, 2020 •
edited

the crc32 changes should be their own commit

crc32: enable static

otherwise LGTM

jonringer
jonringer approved these changes Sep 23, 2020
View reviewed changes
Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

failures are broken on target branch

https://github.com/NixOS/nixpkgs/pull/97768
3 packages failed to build:
google-cloud-cpp python37Packages.google_cloud_bigquery python38Packages.google_cloud_bigquery

11 packages built:
crc32c dvc-with-remotes python27Packages.google_resumable_media python37Packages.google-crc32c python37Packages.google_cloud_automl python37Packages.google_cloud_storage python37Packages.google_resumable_media python38Packages.google-crc32c python38Packages.google_cloud_automl python38Packages.google_cloud_storage python38Packages.google_resumable_media

@jonringer jonringer merged commit 908d8c3 into NixOS:master Sep 23, 2020
@freezeboy freezeboy mentioned this pull request Sep 23, 2020
Merged
10 tasks
@freezeboy freezeboy deleted the fix-google_resumable_media branch September 23, 2020 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@risicle risicle risicle left review comments

@jonringer jonringer jonringer approved these changes

@FRidh FRidh Awaiting requested review from FRidh

@costrouc costrouc Awaiting requested review from costrouc

@andir andir Awaiting requested review from andir

Assignees
No one assigned
Labels
6.topic: python 8.has: package (new) 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@freezeboy @doronbehar @risicle @jonringer