Skip to content

pythonPackages.fs-s3fs: init at 1.0.0 #50471

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 29, 2018
Merged

pythonPackages.fs-s3fs: init at 1.0.0 #50471

merged 2 commits into from
Nov 29, 2018
+28 −3

Conversation

Ma27
Copy link
Member

@Ma27 Ma27 commented Nov 16, 2018 •
edited
Loading

Motivation for this change

fs is a python-based file system abstraction layer. The new package
fs-s3fs is an implementation of it which stores files inside an S3
bucket.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

Sorry, something went wrong.

@Ma27 Ma27 requested a review from FRidh as a code owner November 16, 2018 20:29
@Ma27
Copy link
Member Author

Ma27 commented Nov 16, 2018

cc @ironpinguin

@Ma27
Copy link
Member Author

Ma27 commented Nov 21, 2018

@GrahamcOfBorg build pythonPackages.fs-s3fs python3Packages.fs-s3fs

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

reading manifest file 'fs_s3fs.egg-info/SOURCES.txt'
writing manifest file 'fs_s3fs.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.341s

OK
/nix/store/zm0z25gsn7n4z9navvcpigs1dh8xrgf8-python2.7-fs-s3fs-1.0.0
/nix/store/spjm31qfhcz5syrkbj8acixc4l7q28d6-python3.7-fs-s3fs-1.0.0

@GrahamcOfBorg
Copy link

Timed out, unknown build status on aarch64-linux (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

AssertionError: [psutil.Process(pid=2272, name='python2.7', started='12:17:54'), psutil.Process(pid=2274, name='python2.7', started='12:17:54'), psutil.Process(pid=2275, name='python2.7', started='12:17:54'), psutil.Process(pid=2212, name='python2.7', started='12:17:51'), psutil.Process(pid=2277, name='python2.7', started='12:17:54'), psutil.Process(pid=2278, name='python2.7', started='12:17:54'), psutil.Process(pid=2273, name='python2.7', started='12:17:54'), psutil.Process(pid=2280, name='python2.7', started='12:17:54'), psutil.Process(pid=2281, name='python2.7', started='12:17:54'), psutil.Process(pid=2282, name='python2.7', started='12:17:54'), psutil.Process(pid=2279, name='python2.7', started='12:17:54'), psutil.Process(pid=2276, name='python2.7', started='12:17:54'), psutil.Process(pid=2267, name='python2.7', started='12:17:53'), psutil.Process(pid=2268, name='python2.7', started='12:17:53'), psutil.Process(pid=2269, name='python2.7', started='12:17:53'), psutil.Process(pid=2270, name='python2.7', started='12:17:53'), psutil.Process(pid=2271, name='python2.7', started='12:17:53')]

----------------------------------------------------------------------
Ran 764 tests in 31.563s

FAILED (failures=6, errors=344, skipped=55)
building of '/nix/store/1jjs8zwd48qlfm7z87rlwzljvdfy2ggs-python2.7-pyftpdlib-1.5.4.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/2hfabsmc2gqnsazj78fagsxzhkbl4rhs-python2.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/9lanjncb4d0npbvlmj7s4ai6ckqlxs4r-python2.7-fs-s3fs-1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/9lanjncb4d0npbvlmj7s4ai6ckqlxs4r-python2.7-fs-s3fs-1.0.0.drv' failed

@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

cannot build derivation '/nix/store/v269j0cs7x3lvf6c36j22kdiyjjpfzhp-python3.7-botocore-1.10.84.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/xmg03ck8h1aw0bx3mlm0n58dgqsv4qki-python3.7-s3transfer-0.1.13.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/j8anqgk22rs0r5m4dx65ms3s82fswng3-python3.7-boto3-1.7.84.drv': 2 dependencies couldn't be built
building of '/nix/store/pp8j2di24va5z0rfs5crbhg8a1ix4h7m-python2.7-pyftpdlib-1.5.4.drv' timed out after 3600 seconds
building of '/nix/store/p6dd7haagsvbwvg3ng5zq07f3h8r5g2y-python3.7-pyftpdlib-1.5.4.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/4500nxmpafjf0pdfsqhbpm82wxnvhzmn-python2.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8mmwyrx7zx2bqg2ck32lck7dj0rmh2jz-python3.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/bf8wjsi239kh904rr9avhnb80dmhh9mm-python2.7-fs-s3fs-1.0.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vpid6za0g5i10z29r03gj62hnn5vmh09-python3.7-fs-s3fs-1.0.0.drv': 2 dependencies couldn't be built
error: build of '/nix/store/bf8wjsi239kh904rr9avhnb80dmhh9mm-python2.7-fs-s3fs-1.0.0.drv', '/nix/store/vpid6za0g5i10z29r03gj62hnn5vmh09-python3.7-fs-s3fs-1.0.0.drv' failed

@Ma27
Copy link
Member Author

Ma27 commented Nov 25, 2018

rebuilding due to timeouts (I guess that's been an issue with borg as the package is actually trivial)

@GrahamcOfBorg build pythonPackages.fs-s3fs python3Packages.fs-s3fs

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

reading manifest file 'fs_s3fs.egg-info/SOURCES.txt'
writing manifest file 'fs_s3fs.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.340s

OK
/nix/store/zm0z25gsn7n4z9navvcpigs1dh8xrgf8-python2.7-fs-s3fs-1.0.0
/nix/store/spjm31qfhcz5syrkbj8acixc4l7q28d6-python3.7-fs-s3fs-1.0.0

@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

cannot build derivation '/nix/store/h1gwncrradj44jig8m4k90dnzr8lp5xw-python3.7-botocore-1.10.84.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/5snh28m89jkxqcwzb5r2n95a4isssqqz-python3.7-s3transfer-0.1.13.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/lr28gdp9qnja0pxd421jxgvn1gizd2dz-python3.7-boto3-1.7.84.drv': 2 dependencies couldn't be built
building of '/nix/store/qhwh1fnv5b38b7pb6b180w9rpvxy73xd-python2.7-pyftpdlib-1.5.4.drv' timed out after 1800 seconds
building of '/nix/store/s790nqhii0bpzilvhaylj4sqzn4qglr1-python3.7-pyftpdlib-1.5.4.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/7as6c1yy2hrx7axmzmw1si3mrr624hmn-python2.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8pyy447wb5j1a7apdxllpr9pykfidrxg-python3.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/i2i3fwk2sbm0ynx7072x0vga37p37ri2-python2.7-fs-s3fs-1.0.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/d4ii59f8gc2n052qzjd665m6xszdn776-python3.7-fs-s3fs-1.0.0.drv': 2 dependencies couldn't be built
error: build of '/nix/store/d4ii59f8gc2n052qzjd665m6xszdn776-python3.7-fs-s3fs-1.0.0.drv', '/nix/store/i2i3fwk2sbm0ynx7072x0vga37p37ri2-python2.7-fs-s3fs-1.0.0.drv' failed

@GrahamcOfBorg
Copy link

Timed out, unknown build status on aarch64-linux (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

AssertionError: [psutil.Process(pid=2272, name='python2.7', started='03:04:55'), psutil.Process(pid=2274, name='python2.7', started='03:04:56'), psutil.Process(pid=2211, name='python2.7', started='03:04:53'), psutil.Process(pid=2276, name='python2.7', started='03:04:56'), psutil.Process(pid=2277, name='python2.7', started='03:04:56'), psutil.Process(pid=2278, name='python2.7', started='03:04:56'), psutil.Process(pid=2273, name='python2.7', started='03:04:56'), psutil.Process(pid=2280, name='python2.7', started='03:04:56'), psutil.Process(pid=2281, name='python2.7', started='03:04:56'), psutil.Process(pid=2279, name='python2.7', started='03:04:56'), psutil.Process(pid=2275, name='python2.7', started='03:04:56'), psutil.Process(pid=2266, name='python2.7', started='03:04:55'), psutil.Process(pid=2267, name='python2.7', started='03:04:55'), psutil.Process(pid=2268, name='python2.7', started='03:04:55'), psutil.Process(pid=2269, name='python2.7', started='03:04:55'), psutil.Process(pid=2270, name='python2.7', started='03:04:55'), psutil.Process(pid=2271, name='python2.7', started='03:04:55')]

----------------------------------------------------------------------
Ran 764 tests in 49.522s

FAILED (failures=6, errors=345, skipped=55)
building of '/nix/store/mvgdnsikmk6iqpwkgfc9mqj8vvfs8kj4-python2.7-pyftpdlib-1.5.4.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/fa0s38038gjfc6cp6kng75cxslvgkm17-python2.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ni2hfcwq1hxgnry1dfz2wdb1m84r2ml0-python2.7-fs-s3fs-1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/ni2hfcwq1hxgnry1dfz2wdb1m84r2ml0-python2.7-fs-s3fs-1.0.0.drv' failed

@Ma27
Copy link
Member Author

Ma27 commented Nov 26, 2018

The original problem seems to be pyftpdlib which has been packaged some months ago and seems to break on Hydra. Currently investigating....

@Ma27
Copy link
Member Author

Ma27 commented Nov 26, 2018

I decided to skip the tests in pyftpdlib for now. I'm not that surprised that an FTP library has a certain set of impure tests (that behave differently in Hydra builds and Borg tests, locally the simply worked).

Ma27 added 2 commits November 29, 2018 02:01
@Ma27
pythonPackages.fs-s3fs: init at 1.0.0
fd08b98 
`fs` is a python-based file system abstraction layer. The new package
`fs-s3fs` is an implementation of it which stores files inside an S3
bucket.
@Ma27
pythonPackages.pyftpdlib: skip tests

Verified

This commit was signed with the committer’s verified signature.
ruyadorno Ruy Adorno
GPG key ID: 97B01419BD92F80A
Verified
Learn about vigilant mode
9d2160e 
Although the tests are passing locally, it seems as the excessive
filesystem usage causes several build failures and timeouts in the Hydra
and OfBorg infrastructure:

* https://hydra.nixos.org/build/84374861 (python3 on linux.x86_64)
* https://hydra.nixos.org/build/84368459 (python2 on linux.x86_64)

Some of these tests are failing after several seconds though, but I
couldn't identify a pattern and I'm not overly surprised that a FTP
library has impure tests. However the API seems to be usable in a Python
{2,3} environment, so it should be safe to use even with disabled tests.
@Ma27 Ma27 force-pushed the package-python-fs-s3-support branch from b4e73b0 to 9d2160e Compare November 29, 2018 01:01
@Ma27
Copy link
Member Author

Ma27 commented Nov 29, 2018

Thanks for the reminder! I decided to skip the tests as they heavily rely on s3 buckets and such tests arehighly impure.

@worldofpeace
Copy link
Contributor

Thanks for the reminder! I decided to skip the tests as they heavily rely on s3 buckets and such tests arehighly impure.

All these impurities and we'll need a software filter 😄

I'm testing locally. Should be good though.

worldofpeace
worldofpeace approved these changes Nov 29, 2018
View reviewed changes
Copy link
Contributor

@worldofpeace worldofpeace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, merging at will.

Also 👍 for your detailed commit msg for pythonPackages.pyftpdlib.

@worldofpeace worldofpeace merged commit f9714fd into NixOS:master Nov 29, 2018
@Ma27 Ma27 deleted the package-python-fs-s3-support branch November 29, 2018 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@worldofpeace worldofpeace

@FRidh FRidh

Assignees
No one assigned
Labels
6.topic: python 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@Ma27 @GrahamcOfBorg @worldofpeace