pythonPackages.fs-s3fs: init at 1.0.0 #50471

Ma27 · 2018-11-16T20:29:39Z

Motivation for this change

fs is a python-based file system abstraction layer. The new package
fs-s3fs is an implementation of it which stores files inside an S3
bucket.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Ma27 · 2018-11-16T20:30:04Z

cc @ironpinguin

Ma27 · 2018-11-21T12:17:13Z

@GrahamcOfBorg build pythonPackages.fs-s3fs python3Packages.fs-s3fs

GrahamcOfBorg · 2018-11-21T12:20:22Z

Success on x86_64-linux (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

reading manifest file 'fs_s3fs.egg-info/SOURCES.txt'
writing manifest file 'fs_s3fs.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.341s

OK
/nix/store/zm0z25gsn7n4z9navvcpigs1dh8xrgf8-python2.7-fs-s3fs-1.0.0
/nix/store/spjm31qfhcz5syrkbj8acixc4l7q28d6-python3.7-fs-s3fs-1.0.0

GrahamcOfBorg · 2018-11-21T13:17:37Z

Timed out, unknown build status on aarch64-linux (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

AssertionError: [psutil.Process(pid=2272, name='python2.7', started='12:17:54'), psutil.Process(pid=2274, name='python2.7', started='12:17:54'), psutil.Process(pid=2275, name='python2.7', started='12:17:54'), psutil.Process(pid=2212, name='python2.7', started='12:17:51'), psutil.Process(pid=2277, name='python2.7', started='12:17:54'), psutil.Process(pid=2278, name='python2.7', started='12:17:54'), psutil.Process(pid=2273, name='python2.7', started='12:17:54'), psutil.Process(pid=2280, name='python2.7', started='12:17:54'), psutil.Process(pid=2281, name='python2.7', started='12:17:54'), psutil.Process(pid=2282, name='python2.7', started='12:17:54'), psutil.Process(pid=2279, name='python2.7', started='12:17:54'), psutil.Process(pid=2276, name='python2.7', started='12:17:54'), psutil.Process(pid=2267, name='python2.7', started='12:17:53'), psutil.Process(pid=2268, name='python2.7', started='12:17:53'), psutil.Process(pid=2269, name='python2.7', started='12:17:53'), psutil.Process(pid=2270, name='python2.7', started='12:17:53'), psutil.Process(pid=2271, name='python2.7', started='12:17:53')]

----------------------------------------------------------------------
Ran 764 tests in 31.563s

FAILED (failures=6, errors=344, skipped=55)
building of '/nix/store/1jjs8zwd48qlfm7z87rlwzljvdfy2ggs-python2.7-pyftpdlib-1.5.4.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/2hfabsmc2gqnsazj78fagsxzhkbl4rhs-python2.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/9lanjncb4d0npbvlmj7s4ai6ckqlxs4r-python2.7-fs-s3fs-1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/9lanjncb4d0npbvlmj7s4ai6ckqlxs4r-python2.7-fs-s3fs-1.0.0.drv' failed

GrahamcOfBorg · 2018-11-21T13:17:42Z

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

cannot build derivation '/nix/store/v269j0cs7x3lvf6c36j22kdiyjjpfzhp-python3.7-botocore-1.10.84.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/xmg03ck8h1aw0bx3mlm0n58dgqsv4qki-python3.7-s3transfer-0.1.13.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/j8anqgk22rs0r5m4dx65ms3s82fswng3-python3.7-boto3-1.7.84.drv': 2 dependencies couldn't be built
building of '/nix/store/pp8j2di24va5z0rfs5crbhg8a1ix4h7m-python2.7-pyftpdlib-1.5.4.drv' timed out after 3600 seconds
building of '/nix/store/p6dd7haagsvbwvg3ng5zq07f3h8r5g2y-python3.7-pyftpdlib-1.5.4.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/4500nxmpafjf0pdfsqhbpm82wxnvhzmn-python2.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8mmwyrx7zx2bqg2ck32lck7dj0rmh2jz-python3.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/bf8wjsi239kh904rr9avhnb80dmhh9mm-python2.7-fs-s3fs-1.0.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vpid6za0g5i10z29r03gj62hnn5vmh09-python3.7-fs-s3fs-1.0.0.drv': 2 dependencies couldn't be built
error: build of '/nix/store/bf8wjsi239kh904rr9avhnb80dmhh9mm-python2.7-fs-s3fs-1.0.0.drv', '/nix/store/vpid6za0g5i10z29r03gj62hnn5vmh09-python3.7-fs-s3fs-1.0.0.drv' failed

Ma27 · 2018-11-25T03:04:07Z

rebuilding due to timeouts (I guess that's been an issue with borg as the package is actually trivial)

@GrahamcOfBorg build pythonPackages.fs-s3fs python3Packages.fs-s3fs

GrahamcOfBorg · 2018-11-25T03:07:08Z

Success on x86_64-linux (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

reading manifest file 'fs_s3fs.egg-info/SOURCES.txt'
writing manifest file 'fs_s3fs.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.340s

OK
/nix/store/zm0z25gsn7n4z9navvcpigs1dh8xrgf8-python2.7-fs-s3fs-1.0.0
/nix/store/spjm31qfhcz5syrkbj8acixc4l7q28d6-python3.7-fs-s3fs-1.0.0

GrahamcOfBorg · 2018-11-25T03:34:24Z

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

cannot build derivation '/nix/store/h1gwncrradj44jig8m4k90dnzr8lp5xw-python3.7-botocore-1.10.84.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/5snh28m89jkxqcwzb5r2n95a4isssqqz-python3.7-s3transfer-0.1.13.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/lr28gdp9qnja0pxd421jxgvn1gizd2dz-python3.7-boto3-1.7.84.drv': 2 dependencies couldn't be built
building of '/nix/store/qhwh1fnv5b38b7pb6b180w9rpvxy73xd-python2.7-pyftpdlib-1.5.4.drv' timed out after 1800 seconds
building of '/nix/store/s790nqhii0bpzilvhaylj4sqzn4qglr1-python3.7-pyftpdlib-1.5.4.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/7as6c1yy2hrx7axmzmw1si3mrr624hmn-python2.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8pyy447wb5j1a7apdxllpr9pykfidrxg-python3.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/i2i3fwk2sbm0ynx7072x0vga37p37ri2-python2.7-fs-s3fs-1.0.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/d4ii59f8gc2n052qzjd665m6xszdn776-python3.7-fs-s3fs-1.0.0.drv': 2 dependencies couldn't be built
error: build of '/nix/store/d4ii59f8gc2n052qzjd665m6xszdn776-python3.7-fs-s3fs-1.0.0.drv', '/nix/store/i2i3fwk2sbm0ynx7072x0vga37p37ri2-python2.7-fs-s3fs-1.0.0.drv' failed

GrahamcOfBorg · 2018-11-25T04:04:31Z

Timed out, unknown build status on aarch64-linux (full log)

Attempted: pythonPackages.fs-s3fs, python3Packages.fs-s3fs

Partial log (click to expand)

AssertionError: [psutil.Process(pid=2272, name='python2.7', started='03:04:55'), psutil.Process(pid=2274, name='python2.7', started='03:04:56'), psutil.Process(pid=2211, name='python2.7', started='03:04:53'), psutil.Process(pid=2276, name='python2.7', started='03:04:56'), psutil.Process(pid=2277, name='python2.7', started='03:04:56'), psutil.Process(pid=2278, name='python2.7', started='03:04:56'), psutil.Process(pid=2273, name='python2.7', started='03:04:56'), psutil.Process(pid=2280, name='python2.7', started='03:04:56'), psutil.Process(pid=2281, name='python2.7', started='03:04:56'), psutil.Process(pid=2279, name='python2.7', started='03:04:56'), psutil.Process(pid=2275, name='python2.7', started='03:04:56'), psutil.Process(pid=2266, name='python2.7', started='03:04:55'), psutil.Process(pid=2267, name='python2.7', started='03:04:55'), psutil.Process(pid=2268, name='python2.7', started='03:04:55'), psutil.Process(pid=2269, name='python2.7', started='03:04:55'), psutil.Process(pid=2270, name='python2.7', started='03:04:55'), psutil.Process(pid=2271, name='python2.7', started='03:04:55')]

----------------------------------------------------------------------
Ran 764 tests in 49.522s

FAILED (failures=6, errors=345, skipped=55)
building of '/nix/store/mvgdnsikmk6iqpwkgfc9mqj8vvfs8kj4-python2.7-pyftpdlib-1.5.4.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/fa0s38038gjfc6cp6kng75cxslvgkm17-python2.7-fs-2.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ni2hfcwq1hxgnry1dfz2wdb1m84r2ml0-python2.7-fs-s3fs-1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/ni2hfcwq1hxgnry1dfz2wdb1m84r2ml0-python2.7-fs-s3fs-1.0.0.drv' failed

Ma27 · 2018-11-26T00:53:39Z

The original problem seems to be pyftpdlib which has been packaged some months ago and seems to break on Hydra. Currently investigating....

Ma27 · 2018-11-26T02:02:14Z

I decided to skip the tests in pyftpdlib for now. I'm not that surprised that an FTP library has a certain set of impure tests (that behave differently in Hydra builds and Borg tests, locally the simply worked).

pkgs/development/python-modules/fs-s3fs/default.nix

`fs` is a python-based file system abstraction layer. The new package `fs-s3fs` is an implementation of it which stores files inside an S3 bucket.

Although the tests are passing locally, it seems as the excessive filesystem usage causes several build failures and timeouts in the Hydra and OfBorg infrastructure: * https://hydra.nixos.org/build/84374861 (python3 on linux.x86_64) * https://hydra.nixos.org/build/84368459 (python2 on linux.x86_64) Some of these tests are failing after several seconds though, but I couldn't identify a pattern and I'm not overly surprised that a FTP library has impure tests. However the API seems to be usable in a Python {2,3} environment, so it should be safe to use even with disabled tests.

Ma27 · 2018-11-29T01:02:51Z

Thanks for the reminder! I decided to skip the tests as they heavily rely on s3 buckets and such tests arehighly impure.

worldofpeace · 2018-11-29T01:10:09Z

Thanks for the reminder! I decided to skip the tests as they heavily rely on s3 buckets and such tests arehighly impure.

All these impurities and we'll need a software filter 😄

I'm testing locally. Should be good though.

worldofpeace

Great, merging at will.

Also 👍 for your detailed commit msg for pythonPackages.pyftpdlib.

Ma27 requested a review from FRidh as a code owner November 16, 2018 20:29

GrahamcOfBorg added 6.topic: python 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Nov 16, 2018

worldofpeace reviewed Nov 29, 2018

View reviewed changes

pkgs/development/python-modules/fs-s3fs/default.nix Outdated Show resolved Hide resolved

Ma27 added 2 commits November 29, 2018 02:01

pythonPackages.fs-s3fs: init at 1.0.0

fd08b98

`fs` is a python-based file system abstraction layer. The new package `fs-s3fs` is an implementation of it which stores files inside an S3 bucket.

Ma27 force-pushed the package-python-fs-s3-support branch from b4e73b0 to 9d2160e Compare November 29, 2018 01:01

worldofpeace approved these changes Nov 29, 2018

View reviewed changes

worldofpeace merged commit f9714fd into NixOS:master Nov 29, 2018

Ma27 deleted the package-python-fs-s3-support branch November 29, 2018 01:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.fs-s3fs: init at 1.0.0 #50471

pythonPackages.fs-s3fs: init at 1.0.0 #50471

Ma27 commented Nov 16, 2018 •

edited

Ma27 commented Nov 16, 2018

Ma27 commented Nov 21, 2018

GrahamcOfBorg commented Nov 21, 2018

GrahamcOfBorg commented Nov 21, 2018

GrahamcOfBorg commented Nov 21, 2018

Ma27 commented Nov 25, 2018

GrahamcOfBorg commented Nov 25, 2018

GrahamcOfBorg commented Nov 25, 2018

GrahamcOfBorg commented Nov 25, 2018

Ma27 commented Nov 26, 2018

Ma27 commented Nov 26, 2018

Ma27 commented Nov 29, 2018

worldofpeace commented Nov 29, 2018

worldofpeace left a comment

pythonPackages.fs-s3fs: init at 1.0.0 #50471

pythonPackages.fs-s3fs: init at 1.0.0 #50471

Conversation

Ma27 commented Nov 16, 2018 • edited

Motivation for this change

Things done

Ma27 commented Nov 16, 2018

Ma27 commented Nov 21, 2018

GrahamcOfBorg commented Nov 21, 2018

GrahamcOfBorg commented Nov 21, 2018

GrahamcOfBorg commented Nov 21, 2018

Ma27 commented Nov 25, 2018

GrahamcOfBorg commented Nov 25, 2018

GrahamcOfBorg commented Nov 25, 2018

GrahamcOfBorg commented Nov 25, 2018

Ma27 commented Nov 26, 2018

Ma27 commented Nov 26, 2018

Ma27 commented Nov 29, 2018

worldofpeace commented Nov 29, 2018

worldofpeace left a comment

Choose a reason for hiding this comment

Ma27 commented Nov 16, 2018 •

edited