Third time's a charm? :)

In think my comments in the previous proposal still apply;

1. Since we're defining a "stack" not a "group", perhaps the file should be called Dockerstack.yml? (Proposal: stack composition #9175 (comment))
2. For naming and possibly multiple instances of a stack (Proposal: stack composition #9175 (comment))

Thanks (again), will give these builds a try soon.

As has been discussed in fig issue #159 are there plans to allow for sharing (in this example) redis between YAML files? So I can have 2 applications that both get the same redis when I docker up?

@andrewmichaelsmith I agree that we should support links to existing containers. This should actually work in the test build (though I haven't stress-tested it much) - try prepending a link with a slash, e.g.

links:
  - /external-redis:redis

Zooming out, ideally there'd be a way to do "dependency injection" of containers, so I could e.g. specify a stock redis container in group.yml and then override it in production so it points to my already-running, separately-managed Redis instance.

Including the app name in the file.
I’m unsure about making this the default - lots of Fig users want to be able to do it, but I’m worried that it’ll hurt portability (we don’t do it with Dockerfile, and in my opinion it’s better off for it). Alternate approaches include using the basename of the current directory (like Fig does), or generating a name and storing it in a separate, unversioned file.

tl;dr

Are there use cases for referencing the app name besides building test or dev containers? If not, I think you should keep the current fig behavior, but also support referencing other containers in a FROM :web style. If you don't add support for FROM :web, then I do support adding an app_name key to the yaml file, since that removes broken behavior when working in differently-named directories.

I'd love to do FROM :web to build a tests-only container, rather than have to fig build web && FROM myappdir_web.

Using fig for building testing containers

I'm currently using fig to build a container for running tests, which requires knowing the basename of the current directory. This is problematic if devs check out the code into differently-named directories. Let me show you an example, using the sample python app you provided in the ticket description.

In addition to the Dockerfile and fig.yml you provide above, I also have a dockerfiles/test directory which contains the following Dockerfile:

FROM myapp_web
ADD requirements-dev.txt /srv/myapp
RUN pip install -r requirements-dev.txt

requirements-dev.txt:

nosetests

And I add these lines to fig.yml:

test:
    build: dockerfiles/test
    environment:
        DEBUG: True

This lets me do: fig run test nosetests, and lets me avoid polluting the production container with my development dependencies. However, it also means that before running the tests I have to do fig build web so the myapp_web tag will get created.

The detail to note is FROM myapp_web, which is of the format <myapp_basedir>_<fig_base_container_name>. I'd love to do something like FROM :web. Or, if support for that will not be added, then I'd at least like to be independent on the directory name. A nice bonus would be allowing me to run fig run test without first building base (I'd like fig docker composition to just figure that out).

I haven't found a good pattern for building containers for testing, so please let me know if you know of a better pattern that sidesteps these issues without adding new features.

I haven't found a good pattern for building containers for testing, so please let me know if you know of a better pattern that sidesteps these issues without adding new features.

@sbuss Fig previously supported --projectname (or --project-name) Can't remember right now off the top of my head. I codified this in the project Makefile with something like FIG:=fig --project-name foo, then use $(FIG) everywhere. If you're invoking fig through the shell itself, there is an environment variable you can use as well. All in all I've found the directory scoping things very annoying in general. I'd like to opt out of it completely since I run everything inside a VM anyways, so no collisions can happen regardless.

@aanand I previously used fig for development environments for my team. I switched away because it did not expose docker's full functionality or made it awkward in places. The previous paragraph also speaks to this. Does your implementation also postfix everything with _N e.g. some_container_1? Also does your implementation also remove underscores from containers/group names? This may be a small gripe but I found it be very annoying if you ever wanted to do something outside fig's context. There were times when I wanted to start a one off docker container via docker run but had to link to something created with fig. This requires you to know two things: the namespace may change (if not using --project-name as mentioned above), that container name is not what's listed in the config files. Of course you learn these things but it's a strange hurdle to jump and introduces odd dependencies between different things. Does your proposal include referencing containers for use with --link? e.g. --link :posgres:db. If not I think you should certainly consider it. That abstraction should stretch out to all docker aspects of the docker CLI. Also, does this mean docker groups will be built into docker officially and use the same internal APIs as the CLI? I also ran into problems with fig where the API it communicated with did use the same configuration as the daemon (--insecure-registry). Will such concerns go away in your implementation?

This is awesome!

I would say keep the name: optional and use directory as default, like fig does. But it's definitely a needed feature. One thing I don't like about fig is that I am not able to choose what my images are named aside from renaming directories.

And I don't want to be a killjoy here but, this really is just fig. Re-implemented in Go and looking to find its way into the core. So my preference, as a heavy fig & Docker user, would be to keep this stuff separate.

• It's mostly syntactic sugar.
• It already exists as a healthy and active open source project
• Fig is already governed by Docker, so it's direction is not an issue. Any desired improvements can find their way into fig instead.
• The direction we've been hearing about is towards a tight and modular core.

In the argument as to whether to have compose included in the docker binary or separate, I vote included. I've been using fig and found myself wishing I could do docker up instead of fig up.

Other thoughts:

• It would be great to support the --env-file option in compose from the beginning. Having a file for environment variables is a nice way to tweak the container config instead of having to tweak the fig.yml/group.yml files directly, especially for teams where some settings are individual (e.g. a catchall email address for the dev container). (fig issue: Support for --env-file docker/compose#479)
• I find fig adding _1 to the container name really annoying, especially as I don't need the scale option. It would be great if compose didn't do the same. and when a scale option is implemented, subsequent copies can start from _2 onwards.
• As well as having a file (group.yml) which docker up magically knows to use, it would be great to also be able to provide the file directly, e.g. docker up myconfig.yml . I have a folder with several fig.yml files and they each have to be in their own sub-folder to be usable, which is a little annoying. (Not valid - thanks for the tip @itorres )

@aanand 👍

@tomfotherby: about the fig.yml per directory I think you're missing this fig option:

$ fig -h
[...]
-f, --file FILE           Specify an alternate fig file (default: fig.yml)

I'm not sure that I really like the idea of including composition in the main docker binary. I see composition/fig as a (very) convenient rather than core functionality (very as in "I don't really use docker without fig nowadays").

I vote to keep Compose separate. It will be much easier to add the hooks necessary to do cluster management between Compose and Swarm without risking regressions to Docker. Commands like 'docker up' could be added when Swarm is installed onto a system where Docker resides - ie: shell aliases.

I vote to keep Compose separate, since its functions are largely independent from the core and bear quite different responsibilities. Modularity would make maintenance (of both the core & Compose) and third-party integration easier. So far I'm not aware of strong technical reasons against separation.

Please keep it separate.
It's best having different tools, each with a clear and simple goal.
This will make life of users better by making each tool purpose simple to understand and letting them choose which tools to use for each purpose (several projects are working on composition already). This will make life of coders easier by reducing test/scope surface and dependencies.

I agree to separate it. Just like git-review, if you installed it, you can use either git review or git-review.

As composing seems to be becoming the norm, I'd like to see it included in the Docker binary.

it's preferable to avoid making changes or additions to the Docker API.

This is unfortunate. I thought at least adding support to query for a list of images or containers by prefix would make the client a LOT better. Is it possible this could include some minor API additions, even if it doesn't include all of the docker groups stuff? This is what I was looking forward to the most.

As far as incremental releases, by adding the API support first, current users of fig would start to gain some of the benefits right away. Otherwise we are probably waiting for this to be feature complete with, and as polished as, fig.

docker up
2. ... either build or pull it as needed.

I think this is one of the parts of fig that I'm less than thrilled with. I've tried to outline my concerns here: docker/compose#693. I'd like to see this behaviour change from what fig does.

1. don't build/pull by default, error if things are missing. If a flag is specified (let say something like --build-and-pull) then all containers are force rebuilt and images pulled, even if an container/image exists. Otherwise it seems like you have to build/pull anyway, and there is no way to skip those steps
2. don't accept any command line args for build/pull, otherwise the list of arguments will just be ridiculous (all of docker build , docker pull, plus docker up specific ones)

A new syntax is introduced as a shorthand for referring to containers

I like this

Including the app name in the file

I would like to see this as an option at the very least, it doesn't have to be required. I can say that every single fig setup I have forces an explicit project name because it's necessary for jenkins hosts. Being able to override it locally is still nice for hosts with multiple users, or being able to run multiple environments on the same host (one for interactive testing, one for someone else to preview, one for automated testing, etc). Override from environment variable I think would be my preference.

Scaling

I've never used it personally, and it feels like you could get away with just adding multiple entries to the groups.yml in some cases.

Overall, I am still a fan of this being separate repo. I agree with @chenzhiwei and the comparison to git commands, this would be nice.

Docker can still distribute packages that contain multiple clients, which makes it feel like a single binary. But for users that are interested in custom features, or experimenting with new features before they get accepted upstream, being able to build and run a separate client is much easier than trying to rebuild the entire docker client.

Separate repositories also makes it possible to have separate release schedules, which is always nice.

It would be nice to have python bindings to programatically make the files/containers

Any chance TOML and/or JSON can be supported in addition to (or instead of) YAML? I've had some issues in the past with ambiguous YAML structures.

Don't get deterred by the "monolithic blob fiasco" and too defensive: this belongs to core docker.

Once you do "docker up", it might be cool if there's a way to launch multiple instances of a "role" (like "heroku ps:scale web=3 bg=4"), for testing.

One other thought:

• With Dockerfile, I can give my repo to Docker Hub, Quay.io, etc., and they know exactly how to build my image.
• With Dockerstack.yml/group.yml, I can (hopefully) give my repo to Heroku and they know what tiers to launch and how to link them.
• For running tests, I still have to write a proprietary config (e.g. circle.yml for CircleCI), although a very basic one.

This is probably out of scope, but just mentioning it, since for tests we also need a sort of a stack/group.

@jokeyrhyme +1 for JSON support in addition to (or instead of) YAML.

Maybe this is not well thought through but the simplest for me would be to just integrate it into fig.

Setting the app name would also be nice in fig as well. Maybe we could just deprecate the old file format while supporting both for a while in parallel until "everyone" migrated?

At least I would support having a separate tool for doing the orchestration and people familiar with fig can also then use the same tool to run their containers in production.
People already use fig for production btw...

@jokeyrhyme +1 for JSON support in addition to (or instead of) YAML

@jokeyrhyme +1 for JSON support

-1 for bundling with docker releases

I think there are few implementations out there now that do this task. Fig happens to be the one that pop up more often on the radar as it's the (only?) one promoted by Docker, Inc. Since there are a few implementations out there, it means that the other 20% of the 80% solution needed accounting for. Bundling the 80% with docker seems for force additional complexity, repressibility, opinions, an features into the docker cli.

I think this kind of idea was the motivation for one of the heaviest docker users, coreos, to go their own route a try to envision how a container platform could be structured with a unix philosophy in mind

I get the argument for release syncing but there's a flip side to that. You don't always get everything right in a software release. The more complexity and responsibility you take on the likely hook that something breaks goes up. Decoupled software allows for incremental releases, in other words, the component that has the bug can be released independently with faster turn around than re-releasing every component even if every other component has not changed.

+1 for providing a separate library

I'm not at all against docker being in this game. I believe its definitely a use case. I think that's why docker bought the company that makes fig in the first place. Since Docker, inc already owns that company, why not just make fig do what you're proposing here. If it's go you you're getting at, why not just make a go version of fig? If that's difficult, another area to focus on would be improved (remote) apis. In that case every tool wins!

I agree with @softprops so +1 for providing a separate library

Also agree with previous replies. +1 for separate library. Stays modular and contained and leaves people open to other approaches to orchestration.

Sounds like I'm in the minority ...

+1 for having it in the main binary.

I understand the reasoning in keeping docker binary smaller and modularizing it all, but but simple container dependency management I would've expected to come out of the box with docker.

• Ease of development
• Ease of distributing open source projects. (Docker up, instead of, run this container, then this, then this, etc ...)
• Really, how much larger is the binary with essentially fig tacked on to it?
• When docker updates with new functionality, I don't have to worry about do I have the right version of compose to match with the right version of docker.
• When docker updates, compose updates with all of the new functionality(piggybacking off my last point)

I suppose I see quite a few upside to having it contained in the main binary.

All this hand-wringing over whether or not to bundle the functionality into the docker binary seems like a good reason look at moving towards git's model of allowing 3rd-party commands to be added via extension.

@gabrielgrant allowing for an extension model like git would be great. I still see fig-like templating as part of the core binary though.

I actually like the "git" approach as it might give the best of both worlds; a separate binary, but a "single" endpoint from a user perspective. If only to stop the negative hype (founded or not) that Docker is over-reaching.

What I'm worried about when using this approach, is the vast amount of duplicated code between the compose binary and docker-client. If I understand correctly, compose (in its current state) basically is the docker-client with only a limited number of additions (parsing the group.yml and convert that to API calls, filter containers based on "stack prefix", extract image/container names from the information in group.yml). If I'm correct in this, creating a separate binary would almost be possible via a build-flag to enable/disable those parts in the client.

On the other hand, creating a separate repo does make it easier to add more functionality in the future, without affecting the standard client, so I'm not sure what's best here.

To @thaJeztah's point, it's worth noting that in git everything is an external command. Even the "internals" are implemented as git-* commands. That's how git avoids the code duplication @thaJeztah is worried about here.

I use to have two Dockerfile for my project :

• one for development mode with source code mounted in container and hot reloaded by my dev framework, for efficiency (i.e play run)
• one to compile and package my app ready for production (i.e play dist)

(so I miss #7284, but that's unrelated here)

In both case I need the container to be configured with third party middleware and like the fig/compose approach, but I'm missing some "profile" option. So I propose to allow some switch-like statement to define the target environment and avoid duplicating configuration just because I don't run the development container the same way I deploy the production app.

name: counter

containers:
  web:
    profiles:
        - production
                image: my_cool_app:latest
                command: /target/universal/bin/run
        - development
                build: .
                command: play run
    ports:
      - "5000:5000"
    volumes:
      - .:/code
    links:
      - redis
    environment:
      - PYTHONUNBUFFERED=1
  redis:
    image: redis:latest
    command: redis-server --appendonly yes

+1 for having profiles in Dockerstack.yml file.

So I propose to allow some switch-like statement to define the target environment and avoid duplicating configuration

Personally, I'm more in favor of separate files for that. Easier to compare and less clutter in the groups.yml. I share your concern wrt duplication, so perhaps an #include group.base.yml option could solve this?

However, I wonder if this should be part of the initial implementation. Yes, I want this (functionally), but there are a lot of related issues that will need to be taken care of as well. For example, multiple Dockerfiles, sharing the same build context (#9198, #7284, #2112), .dockerignore files depending on environment, and perhaps templating? (#8446).

Because those haven't really materialised yet, I think this should be put on the roadmap, but not for the initial implementation, otherwise this may take a long time before it gets implemented... baby steps.

Again, just my thoughts, always open to other opinions.

What I'm worried about when using [the separate repo] approach, is the vast amount of duplicated code between the compose binary and docker-client.

@thaJeztah Ideally this would be solved by having a docker-go client library like there is with docker-py, docker-java, etc. That way clients don't have to re-implement anything. Docker is (primarily?) a developer tool, and developers are always going to prefer an API. I think these client libraries are an important part of the docker ecosystem. This is related to an argument that I made in an earlier proposal. Moving fig into docker doesn't really solve some of the perceived problems with having a separate client, it just shifts them onto other developers.

If I understand correctly, compose (in its current state) basically is the docker-client with only a limited number of additions

That is my understanding as well. Unfortunately the client is in the same codebase as the daemon, so this is actually a lot of code already. I think there are still a lot of interesting (and useful) features missing from fig. I would hate to see these features rejected because of the existing complexity in the code repo.

On the other hand, creating a separate repo does make it easier to add more functionality in the future

Yes it does!

I think there is another issue in this debate that has been overlooked so far as well. Right now https://github.com/docker/docker/issues has 850+ open issues. It's very difficult to track down existing issues related to the area you care about (requires a lot of work, and often luck, to hit the right search query and filter through dozens of issues). If compose is yet another feature in this repo it's going to just make the situation worse. With a lot of management , labels could be used to improve this situation, but as it is now, most things are unlabeled, and I don't see any reason to expect that to change. This shifts more of the maintenance/bug triage burden onto the user.

@dnephin I'll give a response, but I have a feeling that we're going into too many related (but important) issues here to keep this discussion readable. Perhaps there's some way to split up "topics"?

Ideally this would be solved by having a docker-go client library

Sounds good, similar to libcontainer is used right now. This will take time before that is realized, so if that's the desired (and possible) approach, either compose must be put "on hold" until that is realized or a roadmap /migration path layed out. Apart from being very interested in the project, I'm not familiar enough with the code to give a reasonable comment on that if it's possible.

Unfortunately the client is in the same codebase as the daemon, so this is actually a lot of code already.

I also wonder how that will work out (in the current situation) wrt maintainers; both compose and the "regular" client commands will cover the same parts of the code, but with different priorities / interests that may conflict. Something that may need to be solved.

If compose is yet another feature in this repo it's going to just make the situation worse.

Fully agree on that. I keep track of the docker issues on a daily base and it's a lot of issues. Unfortunately, a lot of issues regarding (for example) boot2docker also end up in the docker issue tracker, so I fear that the same will happen with compose, swarm and machine. Docker will have to find a way to handle that; close issues asap and redirect the people to the right repo. Maybe this is part of the "operators" responsibility (#9137), but I'm not sure.

First of all, I very much welcome a Go implementation of Compose. After all, a Go binary is much easier to distribute than a Python tool with several external dependencies (fig).

I don't mind Compose eventually becoming part of the main docker client; batteries included, as mentioned by @twirkman, is important for increased ease of use and adoption of Docker itself.

Initially it might feel safer for users if we could download and use this tool without replacing the system docker daemon (and client). If there was a patch available for the latest stable docker client(s), as well as patched binaries, that might do the trick right?

Otherwise, having the docker client and docker daemon part ways, and having a docker-go client library, would be very nice long-term goals. As @thaJeztah mentions though, this may take a long time, and I would hate to see "Compose" being put on hold until that is fixed. Ripping out part of the docker tool's code, and putting it into a very crude, incomplete and unstable first version of a docker-go client library, might be a faster aproach?

In the end, what ever gets compose out sooner, is a good approach:-)

Don't let Docker get monolithic. The core binary should contain the bare minimum to build, start, stop, view containers etc.

Orchestration is a distinct layer above the basic functionality of managing a container, and while the current proposal seems like a Fig rewrite, it could potentially grow beyond anything we imagine for now. A separate orchestration component would have far more freedom to grow and evolve without worrying about bloating the core.

On a political level, I would say this: Provide a separate application (like Fig) and those who thought it ought to be bundled will likely continue to use Docker anyway, but bundle another functionality layer into the core and you might lose the people who believe in a separate tool for each job.

👍 this sounds awesome. I prefer Dockerstack for the name of the compose file as well, even dropping the .yml.

+1 for separate application to orchestrate docker containers, or at least a docker-go client library

I also agree it would be preferable to have this tool in a separate application from the core docker container engine: docker-compose , docker-stack , stacker ...

+1 for separate application

Agree with @softprops (#9459 (comment))
Docker itself should only provide the interface to control containers, not implement an orchestration layer on top of this.

Also agree with @dnephin (#9459 (comment)) there should be a Docker Go binding.
This would support splitting the Docker daemon from the Docker CLI as well, which is a good idea in general. That way the Docker daemon provides the interface to control containers and tools like the Docker CLI and Compose talk to this interface using the Docker Go bindings.
Maybe we can start with Compose simply importing from the main Docker repo?

An orchestration tool should not control the containers itself but should only communicate with the Docker interface the Daemon exposes.

Also +1 for the git-* approach, this makes it possible to develop separate parts but still present them to the enduser in a coherent way.

It's a very good news! I'm really looking forward to see this in daily usage.
For the debate, I'm really a big fan of the unix philosophy and the KISS principle, so I would prefer a set of little binaries composing themselves beautifully.

+1 for integrating into core. If for no other reason so that the test suites are integrated. Docker 1.4 has broken Fig's ability to properly mount volumes and this really should never happen.

https://github.com/michaelsauter/crane don't look to bad...
It's Go, it supports both JSON and YAML, it exposes most docker run options (without renaming them!), and it still seems to work with volumes after the Docker 1.4 upgrade.

Thank you @smyrman - Crane is actually even better than Fig for my needs, I'm now back up and running better than ever.

indeed crane does look nice at first glance

+1 for json support.
why 'docker up' doesn't receive a specific file as parameter? doesn't it limit the way users can arrange configuration files?

I don't get what's happening here.
The "docker" company has already acquired fig.
Then why do we need the same functionality as fig in the core?

Is this proposal an evolving of fig or a totally different thing?

Deprecated in favour of #9694. Onwards!

Oh no, not again? 😄

Compose has now been released and is available here: http://docs.docker.com/compose/