arion: init at 0.1.0.0 #71092

roberth · 2019-10-13T20:36:04Z

Motivation for this change

Package Arion, a tool for working with Docker through Nix!

Things done

cdepillabout · 2019-10-14T03:36:59Z

@roberth Thanks for sending this PR.

I don't quite understand why arion-compose can't just be used from haskellPackages.arion-compose. Why is this additional arion derivation needed? It also seems somewhat fishy to rely on a "magic" file ./cabal2nix-arion-compose.nix that makes the derivation do something slightly different depending on whether or not it exists.

roberth · 2019-10-14T10:11:09Z

@cdepillabout

Why is this additional arion derivation needed?

To make it more lightweight by building a static executable as far as haskell libraries are concerned
To provide the arion.build and arion.eval functions (although this is technically an answer to why an addition expression is needed)
- These functions require builtins.fetchTarball rather than the usual fetchers because some people do not want to use IFD that doesn't work well on Hydra
To rename the product to its actual name. It's called Arion, but that name turned out to be already taken on hackage
To include a known-compatible version of docker compose. Arion should package its own version of docker compose. Docker compose itself can not be used with arion projects, so if the user wants to use both programs they should be able to install a different version in the profile (or shell or whatever)

It also seems somewhat fishy to rely on a "magic" file ./cabal2nix-arion-compose.nix that makes the derivation do something slightly different depending on whether or not it exists.

I do agree in general, but I try to keep the number of variations of the expressions low. This way I can backport the expression without having to maintain another variation of this file. So it does serve a purpose.
Do you think this is an acceptable reason to keep it?

cdepillabout · 2019-10-14T12:40:55Z

pkgs/applications/virtualization/arion/default.nix

+  optionalCabal2nixFile = ./cabal2nix-arion-compose.nix;
+
+  arion-compose =
+    # Test suite needs nixpkgs as dependency which would make it quite heavy.
+    pkgs.haskell.lib.dontCheck (
+      if ! builtins.pathExists optionalCabal2nixFile
+      then haskellPackages.arion-compose
+      else haskellPackages.callPackage optionalCabal2nixFile {});


Instead of hard-coding the path ./cabal2nix-arion-compose.nix, why not take optionalCabal2nixFile as an argument to this file, so you'd have something like:

{ pkgs , haskellPackages ? pkgs.haskellPackages , optionalCabal2nixFile ? null }:

Then, the arion-compose expression would become:

arion-compose = # Test suite needs nixpkgs as dependency which would make it quite heavy. if isNull optionalCabal2nixFile then haskellPackages.arion-compose else pkgs.haskell.lib.dontCheck (haskellPackages.callPackage optionalCabal2nixFile {});

This should make it more natural to use for most users who are used to the callPackage style of packages.

cdepillabout · 2019-10-14T12:44:38Z

pkgs/applications/virtualization/arion/default.nix

+{ pkgs
+, haskellPackages ? pkgs.haskellPackages
+}:


Instead of taking pkgs as an argument here, could you only declare the things you need from pkgs? That would make it more natural to use as a callPackage-style package.

I'm specifically thinking of things like haskellPackages (which you do declare), haskell (for haskell.lib, or even just the specific things you use from haskell.lib).

cdepillabout · 2019-10-14T12:46:54Z

pkgs/applications/virtualization/arion/arion.nix

+{ pkgs ? import ./. {}
+, arion-compose
+, arion-compose-eval-src
+}:


The same thing here about turning this into a callPackage-style package.

The current way it is written, it is not obvious that arion depends on things like justStaticExecutables, overrideCabal, docker-compose, makeWrapper, etc. It would also be more difficult for someone to override those arguments, since they are taken directly from pkgs.

cdepillabout · 2019-10-14T12:50:13Z

pkgs/applications/virtualization/arion/arion.nix

+, arion-compose
+, arion-compose-eval-src


Could you add comments explaining what these are supposed to be?

cdepillabout · 2019-10-14T12:50:42Z

pkgs/applications/virtualization/arion/arion.nix

+  eval = args@{...}:
+    import
+      (arion-compose-eval-src + "/nix/eval-composition.nix")
+      ({ inherit pkgs; } // args);
+
+  build = args@{...}:
+    let composition = eval args;
+    in composition.config.out.dockerComposeYaml;


Could you add comments on what eval and build are supposed to be? Also, an example of how they are intended to be used would be helpful.

cdepillabout · 2019-10-14T12:53:55Z

pkgs/applications/virtualization/arion/update

+# Usage ./update ~/src/arion
+#
+# Make sure that ~/src/arion matches the latest version on hackage.


I don't think this necessarily needs to be changed, but it might be easier for other people to run if this update script just pulled arion from the latest version on Hackage and used that. Then you wouldn't need arion checked out locally (and have to match versions by hand).

cdepillabout · 2019-10-14T12:56:34Z

pkgs/development/haskell-modules/hackage-packages.nix

@@ -30842,7 +30842,6 @@ self: {
       description = "Run docker-compose with help from Nix/NixOS";
       license = stdenv.lib.licenses.asl20;
       hydraPlatforms = stdenv.lib.platforms.none;
-       broken = true;


This file is generated automatically, so if you edit it, there is a chance it will cause a build conflict.

Can you remove this change? It will be marked as unbroken next time the file is automatically generated (which happens quite frequently).

You might be interested in this video which explains the process:

https://discourse.nixos.org/t/video-tutorial-how-to-fix-broken-haskell-packages-in-nix/3968

How can you tell I didn't run it?

Tbh I got a linker error in cabal2nix. I wrote an issue here to make that process reproducible if you're interested: NixOS/cabal2nix#431

How can you tell I didn't run it?

As far as I know, in general hackage2nix is run automatically by some sort of CI setup by peti. So when people send PRs to nixpkgs, they are often asked to not change pkgs/development/haskell-modules/hackage-packages.nix by hand, because it will sometimes cause merge conflicts.

Also, hackage2nix will always(?) remove both the broken = true; and hydraPlatforms = stdenv.lib.platforms.none; lines.

cdepillabout · 2019-10-14T13:01:42Z

@roberth Thanks for the additional explanation.

I think one of the things I'm having a hard time following is why this is split into a default.nix file and an arion.nix file? Would it be possible to just put everything in one file?

If not, could you add some documentation explaining the reason for the split? I think I'm having a tough time following what is going on in this derivation.

roberth · 2019-10-14T13:58:10Z

@cdepillabout, thanks for the extensive review!

The reason for the split is that I can use the same arion.nix both here and in the arion source project. I'll try to make it more like a callPackage package, but since most things are Haskell related, it's not going to be a big difference. For instance I can only inject pkgs.haskell, not pkgs.haskell.lib directly, let alone pkgs.haskell.lib.justStaticExecutables.

cdepillabout · 2019-10-14T14:12:32Z

One more thing I forgot: since you've changed the pkgs/development/haskell-modules/configuration-hackage2nix.yaml, could you target the haskell-updates branch instead of master?

This adds me as a maintainer of arion-compose, which provides the executable for the arion package (NixOS#71092). Note that it has a different name because it was already taken on Hackage before arion switched to Haskell.

This adds me as a maintainer of arion-compose, which provides the executable for the arion package (#71092). Note that it has a different name because it was already taken on Hackage before arion switched to Haskell.

roberth · 2019-10-24T11:11:21Z

@cdepillabout I've dropped the ability to move files between here and the arion project. It probably wasn't worth optimizing for that.

cdepillabout

@roberth Thanks for updating this. This looks good.

I think this should be able to be merged as-is. However, I'm still not 100% sure how the eval and build functions are supposed to be used. Maybe you're using them on the upstream arion repo? Are they meant to be used by normal nix users (people other than you and Domen)?

roberth · 2019-10-24T14:11:27Z

@cdepillabout yes, they are intended for arion users. We will publish documentation very soon and add a link in the inline docs here.

I'll merge it so it's there just in time for NixCon.

jbmusso · 2019-10-25T10:14:56Z

Experimenting building on Mac OS X with a simple derivation - nixpkgs master as of this morning. Roughly:

{
  arionSrc = pkgs.fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs";
    rev = "ce14f092eab6efcb2915c1db189558c1d5a469e8"; # master as of 20191025 ~Friday morning
    sha256 = "1w901v41gfnxd9xaanxns5qxrjnhlpfgsn6g7bhf76jg11lggxf8";
  };

  arion = (pkgs.callPackage arionSrc {}).arion;
}

Install tests are failing with an OSError: AF_UNIX path too long error.
Log details are a bit verbose, so they're collapsed below (click to expand):

OSError: AF_UNIX path too long

cannot build derivation '/nix/store/cqamkpf3gw05bk7sw163ysm53yqyfh2z-docker-compose-1.24.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ml4dz80glkkxgv1nv1ndi8rzp08hyl7r-arion-0.1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/ml4dz80glkkxgv1nv1ndi8rzp08hyl7r-arion-0.1.0.0.drv' failed

running install tests
============================= test session starts ==============================
platform darwin -- Python 3.7.5, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
rootdir: /private/var/folders/8c/cjp8r0355mbdsygc0_mg00l80000gn/T/nix-build-python3.7-docker-4.1.0.drv-0/docker-4.1.0
collected 547 items

tests/unit/api_build_test.py .........s...                               [  2%]
tests/unit/api_container_test.py ....................................... [  9%]
.........................................................                [ 19%]
tests/unit/api_exec_test.py .....                                        [ 20%]
tests/unit/api_image_test.py .........................                   [ 25%]
tests/unit/api_network_test.py ......                                    [ 26%]
tests/unit/api_test.py .......................F.............             [ 33%]
tests/unit/api_volume_test.py ..........                                 [ 35%]
tests/unit/auth_test.py ................................................ [ 43%]
........                                                                 [ 45%]
tests/unit/client_test.py ..........                                     [ 47%]
tests/unit/dockertypes_test.py ......................................... [ 54%]
.....x............                                                       [ 57%]
tests/unit/errors_test.py ..................                             [ 61%]
tests/unit/models_containers_test.py ................................... [ 67%]
...                                                                      [ 68%]
tests/unit/models_images_test.py ................                        [ 71%]
tests/unit/models_networks_test.py ......                                [ 72%]
tests/unit/models_resources_test.py ..                                   [ 72%]
tests/unit/models_services_test.py .                                     [ 72%]
tests/unit/ssladapter_test.py ......                                     [ 73%]
tests/unit/swarm_test.py ...                                             [ 74%]
tests/unit/types_containers_test.py .                                    [ 74%]
tests/unit/utils_build_test.py ...s.......................s.........F... [ 82%]
...                                                                      [ 82%]
tests/unit/utils_config_test.py .....s....                               [ 84%]
tests/unit/utils_json_stream_test.py .......                             [ 85%]
tests/unit/utils_proxy_test.py .....                                     [ 86%]
tests/unit/utils_test.py ...........................................s... [ 95%]
..........................                                               [100%]

=================================== FAILURES ===================================
_______________ UnixSocketStreamTest.test_early_stream_response ________________

self = <tests.unit.api_test.UnixSocketStreamTest testMethod=test_early_stream_response>

    def setUp(self):
        socket_dir = tempfile.mkdtemp()
        self.build_context = tempfile.mkdtemp()
        self.addCleanup(shutil.rmtree, socket_dir)
        self.addCleanup(shutil.rmtree, self.build_context)
        self.socket_file = os.path.join(socket_dir, 'test_sock.sock')
>       self.server_socket = self._setup_socket()

tests/unit/api_test.py:372:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.unit.api_test.UnixSocketStreamTest testMethod=test_early_stream_response>

    def _setup_socket(self):
        server_sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>       server_sock.bind(self.socket_file)
E       OSError: AF_UNIX path too long

tests/unit/api_test.py:387: OSError
_________________________ TarTest.test_tar_socket_file _________________________

self = <tests.unit.utils_build_test.TarTest testMethod=test_tar_socket_file>

    @pytest.mark.skipif(IS_WINDOWS_PLATFORM, reason='No UNIX sockets on Win32')
    def test_tar_socket_file(self):
        base = tempfile.mkdtemp()
        self.addCleanup(shutil.rmtree, base)
        for d in ['foo', 'bar']:
            os.makedirs(os.path.join(base, d))
        sock = socket.socket(socket.AF_UNIX)
        self.addCleanup(sock.close)
>       sock.bind(os.path.join(base, 'test.sock'))
E       OSError: AF_UNIX path too long

tests/unit/utils_build_test.py:463: OSError
============= 2 failed, 539 passed, 5 skipped, 1 xfailed in 4.68s ==============
builder for '/nix/store/0j7mc6zhfq7x70n3cbl4wzplbn424y71-python3.7-docker-4.1.0.drv' failed with exit code 1
cannot build derivation '/nix/store/cqamkpf3gw05bk7sw163ysm53yqyfh2z-docker-compose-1.24.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ml4dz80glkkxgv1nv1ndi8rzp08hyl7r-arion-0.1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/ml4dz80glkkxgv1nv1ndi8rzp08hyl7r-arion-0.1.0.0.drv' failed

It appears that the path length limit for a Unix domain socket is around 104 on Mac OS. This could be related.

Please let me know if you would rather want me to open a distinct issue so this merged PR doesn't get hijacked.

roberth · 2019-10-25T15:21:47Z

@jbmusso This looks like a build failure in the python docker package in docker-compose's closure.

It's probably best to open a Nixpkgs issue and @ mention jonringer there.

This adds me as a maintainer of arion-compose, which provides the executable for the arion package (NixOS#71092). Note that it has a different name because it was already taken on Hackage before arion switched to Haskell. (cherry picked from commit 3ca15e5)

roberth added 8.has: package (new) 11.by: nixpkgs-member 11.by: package-maintainer 11.by: upstream-developer labels Oct 13, 2019

roberth requested a review from basvandijk as a code owner October 13, 2019 20:36

ofborg bot added 6.topic: haskell 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Oct 13, 2019

roberth changed the title ~~Arion init~~ arion: init at 0.1.0.0 Oct 13, 2019

cdepillabout reviewed Oct 14, 2019

View reviewed changes

roberth mentioned this pull request Oct 14, 2019

arion-compose: Enable without tests #71132

Closed

10 tasks

roberth mentioned this pull request Oct 14, 2019

arion-compose: Enable without tests #71133

Merged

10 tasks

arion: init at 0.1.0.0

d3913c8

roberth force-pushed the arion-init branch from b3c90e6 to d3913c8 Compare October 24, 2019 11:08

roberth requested a review from cdepillabout October 24, 2019 11:09

ofborg bot removed the 6.topic: haskell label Oct 24, 2019

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 and removed 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Oct 24, 2019

cdepillabout approved these changes Oct 24, 2019

View reviewed changes

roberth mentioned this pull request Oct 24, 2019

Improve documentation hercules-ci/arion#10

Closed

roberth merged commit becfba9 into NixOS:master Oct 24, 2019

jbmusso mentioned this pull request Oct 24, 2019

Arion.Nix evaluateComposition "matches an example" test failure (on unsupported Mac OS) hercules-ci/arion#83

Closed

jbmusso mentioned this pull request Oct 25, 2019

arion package build failure on Mac Os X ("OSError: AF_UNIX path too long" error when building docker) #71993

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

arion: init at 0.1.0.0 #71092

arion: init at 0.1.0.0 #71092

roberth commented Oct 13, 2019

cdepillabout commented Oct 14, 2019

roberth commented Oct 14, 2019

cdepillabout Oct 14, 2019

cdepillabout Oct 14, 2019

cdepillabout Oct 14, 2019

cdepillabout Oct 14, 2019

cdepillabout Oct 14, 2019 •

edited

cdepillabout Oct 14, 2019

cdepillabout Oct 14, 2019

roberth Oct 14, 2019

cdepillabout Oct 14, 2019

cdepillabout commented Oct 14, 2019 •

edited

roberth commented Oct 14, 2019

cdepillabout commented Oct 14, 2019

roberth commented Oct 24, 2019

cdepillabout left a comment

roberth commented Oct 24, 2019

jbmusso commented Oct 25, 2019

roberth commented Oct 25, 2019

arion: init at 0.1.0.0 #71092

arion: init at 0.1.0.0 #71092

Conversation

roberth commented Oct 13, 2019

Motivation for this change

Things done

cdepillabout commented Oct 14, 2019

roberth commented Oct 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdepillabout Oct 14, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdepillabout commented Oct 14, 2019 • edited

roberth commented Oct 14, 2019

cdepillabout commented Oct 14, 2019

roberth commented Oct 24, 2019

cdepillabout left a comment

Choose a reason for hiding this comment

roberth commented Oct 24, 2019

jbmusso commented Oct 25, 2019

roberth commented Oct 25, 2019

cdepillabout Oct 14, 2019 •

edited

cdepillabout commented Oct 14, 2019 •

edited