+1

Much prefer this to #7576. Strikes a nice balance between prescriptive and extensible, imho. 👍

👍

Very cool to see fig-like behaviour coming to Docker. 😄

I am sad to see links go away, though. I like how links provide an explicit dependency graph on a distributed application. This is, I think, what fig users think they are describing when they write a fig file - therefore IMO thinking through how a group will be able to span multiple hosts is very important.

The problem with the proposal for composability of groups (hypothetical clustering tool exposing the groups API externally and internally telling Docker about the existence of "remote" containers just by specifying other IPs), is that each cluster node might only have one (internal/external) IP address, and so "refer to service by IP address" may not permit the level of multi-tenancy we'd like to be able to in a cluster of containers (each host can be running many containers, balanced according to some sensible scheduling policy).

I would love it if we could have links and all the lovely semantic dependency information they provide first-class in the Docker API, and provide hooks for a clustering tool to provide an (IP, port) pair which it might dynamically allocate for such a link, perhaps using the classic Docker links environment variable syntax. This would allow us to extend nicely to multi-host links with an interface which remains backward compatible for folks who are already developing Docker{files,images} against these expected env vars.

@lukemarsden isn't specifying the container's in a group an explicit dependency graph on a distributed application. If you don't want them communicating with each other then why are you starting them in a group and interacting with them as a single unit? Atleast that is my take on it. If I wanted them to be separate then I wouldn't have put them together ;)

+1

How will cgroups be setup for containers in a group? Ideally they should
share a common resource pool.

On Fri, Oct 17, 2014 at 9:10 PM, Shreyas Karnik notifications@github.com
wrote:

+1

—
Reply to this email directly or view it on GitHub
#8637 (comment).

@crosbymichael Aanand has not explicitly stated so, but one can infer that a group is intended to represent the entirety of a multi-tiered microservice-based application:

For example, a group named myapp may have the containers myapp/web and myapp/db.

I too would lament the loss of links as the Docker standard for composing applications using multiple containers. @aanand what is the motivation for "Inter-container communication using these hostnames is preferred over explicit links"? I think this merits some discussion (or perhaps this has already taken place and I have missed it!)

Really exited to see this, initial impression is good!

I'm wondering if the "group" namespace is just for organisation, or if it also acts as a sandbox? I.e., would I be able to create tennant1/redis and tennant2/redis and restrict access to each namespace, even if tennant1 tries to communicate with tennant2/redis by IP-address?

@crosbymichael re: links;
I agree that using groups defines which containers belong together, but I agree with @lukemarsden that using links may have its place. Personally, I see links as the "patch cables" in a container stack. Although it's currently not possible to dynamically "patch" those cables, I would really like it if that was possible. A possible use case for this;

some-project/web
some-project/db-dev
some-project/db-production

Where db-production is a copy of the production database. Being able to "swap" the database during development (just for testing) and alias it as db, so that the web container doesn't have to be changed would be awesome.

Unless I'm mistaken, env vars are pretty much going away in links v2.
Other than that, the only other way to do linking currently is from /etc/hosts.

This container grouping is really akin to Kubernetes pods (which share a net namespace).
I think for this particular use case is just right.
You have an app, a database, a web frontend, etc.
If you need them to be split up, then don't use grouping.

@cpuguy83 can you reference the Docker links v2 discussion please? Also "If you need them to be split up" - please can you define "split up"? Do you mean "spread over multiple hosts" ?

@robhaswell #7468 #7467

I mean "split up" as in separate entities, whatever that may mean (same host or multi-host).

Thanks for the "links" (hur) 😄

In light of links v2 I can see how this is functionality identical, so +1 from me. Losing the implied dependency graph (that Fig makes use of today) would be sad though.

I find the proposed networking model confusing. In Kubernetes, the containers within a pod actually share a network namespace and therefore share the same IP address and port space.

We've proposed that they share resources (@vishh's question) and other namespaces, also (kubernetes/kubernetes#1615).

Copying text from https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/pods.md:

A pod (as in a pod of whales or pea pod) models an application-specific "logical host" in a containerized environment. It may contain one or more containers which are relatively tightly coupled -- in a pre-container world, they would have executed on the same physical or virtual host.

Pods facilitate data sharing and communication among their constituents.

The containers in the pod all use the same network namespace/IP and port space, and can find and communicate with each other using localhost. Each pod has an IP address in a flat shared networking namespace that has full communication with other physical computers and containers across the network.

In addition to defining the containers that run in the pod, the pod specifies a set of shared storage volumes. Volumes enable data to survive container restarts and to be shared among the containers within the pod.

In the future, pods will share IPC namespaces, CPU, and memory (http://www.linuxplumbersconf.org/2013/ocw//system/presentations/1239/original/lmctfy%20(1).pdf).

Pods also simplify application deployment and management by providing a higher-level abstraction than the raw, low-level container interface. Pods serve as units of deployment and horizontal scaling/replication. Co-location, fate sharing, coordinated replication, resource sharing, and dependency management are handled automatically.

Pods can be used to host vertically integrated application stacks, but their primary motivation is to support co-located, co-managed helper programs, such as:

• content management systems, file and data loaders, local cache managers, etc.
• log and checkpoint backup, compression, rotation, snapshotting, etc.
• data change watchers, log tailers, logging and monitoring adapters, event publishers, etc.
• proxies, bridges, and adapters
• controllers, managers, configurators, and updaters

Individual pods are not intended to run multiple instances of the same application, in general.

Something else we need in order to be able to use this is hooks: #6982. Otherwise if we want to do anything during container lifecycle (say, count the number of container restarts), we either need to add the support directly to Docker, which is sometimes desirable but sometimes not, or abandon the mechanism. For this reason, we can't currently use Docker's container restarts, for example.

@bgrant0607 Are events not good enough for counting container restarts?

While hooks (or plugins) are indeed important, I don't see how they are relevant to this particular discussion.

@cpuguy83 Ok, counting restarts was not a good example. Hooks are relevant because they are a prerequisite to make this feature usable by anyone for whom the provided behavior does not fully cover and/or match what they need to do.

@bgrant0607

Let's talk about what missing functionality you are referring to.
If something has been overlooked then we need to discuss.

@cpuguy83 Would be happy to go into more detail, here or in a hangout or something.

In Kubernetes, containers within pods are never spread across hosts. That's the whole point of that abstraction.

For managing sets of pods across multiple hosts, we use labels.

My OSCON presentation explains the rationale:
https://speakerdeck.com/bgrant0607/managing-containerized-applications

I liked the proposal until the last section, where I feel it jumped the
rails.

Providing a simple grouping mechanism, akin to what Kubernetes calls pods,
is part of a layered design. That's what I thought was being proposed (in
fact, the proposed API is pretty much identical to what we've been doing
with Kubernetes since the beginning.

Trying to make a cluster look like one big docker node is cute, but not
really that useful. Don't hide the cluster-ness. Embrace it. A cluster
is not a node. It fails to meet the node abstraction in all sorts of ways.

Also, I +1 Vish's comment strongly. Once you create grouping you have to
answer questions like resources and namespaces (other than net).

On Fri, Oct 17, 2014 at 1:21 PM, Rob Haswell notifications@github.com
wrote:

@crosbymichael https://github.com/crosbymichael Aanand has not
explicitly stated so, but one can infer that a group is intended to
represent the entirety of a multi-tiered microservice-based application:

For example, a group named myapp may have the containers myapp/web and
myapp/db.

I too would lament the loss of links as the Docker standard for composing
applications using multiple containers. @aanand
https://github.com/aanand what is the motivation for "Inter-container
communication using these hostnames is preferred over explicit links"? I
think this merits some discussion (or perhaps this has already taken place
and I have missed it!)

Reply to this email directly or view it on GitHub
#8637 (comment).

I have a much longer comment I'd like to make, but first I'll get some random small things out of the way.

@crosbymichael Links form a directed graph between containers. Groups in this PR form a set of containers, or essentially a bidirectional full mesh. The key things missing here is that groups do not describe directional explicit relationships between containers. Imagine a simple web, app, db application. They would be in the same group but you no longer know that web talks to app, app to db. So there are definitely some capabilities of describing an application lost here. Especially when you consider security. In the links case you could essentially prevent (assuming some other tool you built) outbound connections from DB because due to the links you know no outbound traffic will be initiated.

@vishh @bgrant0607 Groups seem to be an entirely different concept than pods so it doesn't seem like were comparing apples to apples. This does point out that maybe "Groups" is a wrong term for this API. All groups means at the moment is a shared DNS namespace and common start/stop lifecycle.

More comments to come.... :)

I am surprised to hear that groups are supposed to be different than pods -
you use the exact same words and API to describe them, right up until the
last piece of the OP description.
On Oct 17, 2014 11:23 PM, "Darren" notifications@github.com wrote:

I have a much longer comment I'd like to make, but first I'll get some
random small things out of the way.

@crosbymichael https://github.com/crosbymichael Links form a directed
graph between containers. Groups in this PR form a set of containers, or
essentially a bidirectional full mesh. The key things missing here is that
groups do not describe directional explicit relationships between
containers. Imagine a simple web, app, db application. They would be in the
same group but you no longer know that web talks to app, app to db. So
there are definitely some capabilities of describing an application lost
here. Especially when you consider security. In the links case you could
essentially prevent (assuming some other tool you built) outbound
connections from DB because due to the links you know no outbound traffic
will be initiated.

@vishh https://github.com/vishh @bgrant0607
https://github.com/bgrant0607 Groups seem to be an entirely different
concept than pods so it doesn't seem like were comparing apples to apples.
This does point out that maybe "Groups" is a wrong term for this API. All
groups means at the moment is a shared DNS namespace.

More comments to come.... :)

Reply to this email directly or view it on GitHub
#8637 (comment).

I agree with @thockin. I don't understand the point of the proposal, then. This model won't scale to non-trivial deployment and scaling scenarios.

@aanand what would be the relationship between groups and volumes?

@aanand would this deprecate container-to-container links, and the naming/hierarchy system that comes with it? If so, how would we deal with reverse compatibility?

Where do we draw the line? As an architect I like lines (circles and squares too). Last DockerCon Docker was given the moniker Docker Engine in an attempt to distinguish the core Docker from all the other services and functionality that we will begin to layer on top of it. This proposal is one of the first pieces of functionality that I see that is logically layered above the core functionality of the Docker Engine.

Previously with fig and Docker the separation was clear and obvious. But, as we want to increase the convenience, utility, and usability of Docker it makes sense to pull fig “into the fold” and make it a first class citizen. This way users can get fig functionality but still with a single binary download, and single CLI interface. This is a tricky maneuver.

With fig, lets say I had some magical Docker orchestration tool (let’s just completely artribtrarily pick Stampede.io as an example 😄). If I was to perfectly implement the Docker Remote API with Stampede.io, then fig would just magically work with Stampede.io. Fig, or any tool that uses the remote API, would work because the remote API is the interface and contract to Docker.

With this proposal, the Docker Remote API is expanded to add this functionality. While some of the functionality is indeed client side, the remote API is extended to add the Groups API. Now lets say the creator of Stampede.io (who I happen to know well), doesn’t really want to implement Groups functionality because he’s lazy, or wants to do different things. (Let’s not fool ourselves, group start/stop/update will become very complex if one wants to do real production worthy deployment methodologies.) The problem is that if he doesn’t implement the API, then it just won’t work. Now looking back at the fig approach, it will work because it’s physically a layer outside of Docker.

So where do we draw the line. Where does the Docker Engine end and where does higher level functionality start. What I would propose is that the goal is to not expand the Docker Remote API. Keep it as small as possible with only the bare minimum primitives to allow higher level functionality to be implemented. Then higher level functionality, such as Groups should be developed as a logically separate component that only relies on the Remote API to perform its functionality. If we still want a single binary download, that is fine. That is a packaging issue, I’m purely talking about architecture and design. Logically something like Groups should be implemented as a bolt on that requires only the Remote API. This way people building magical Docker platforms are only required to support the Docker Remote API and then all other higher level functionality can be layered on top.

You can imagine we are going to see more and more higher level functionality: scheduling, application templating/management, policy management, service discovery, etc. If all of these things creep into the core Remote API we lose composability. We want to encourage a pluggable ecosystem and not massive vertical implementations.

I still have some real technical comments on the implementation, but I want to hold off until we can talk about this higher level approach stuff first. I feel this proposal is going to essentially set the precedent on how we approach bringing higher level functionality into Docker and I’d like to get it right (or as close to it) now.

@shykes thoughts?

@aanand

The first thing I think we should try to get agreement on here is: "what are groups for?"

If groups are meant to be "fig in Docker" then I think we should observe and respect the practice of fig users (much like philosophers of maths observe and reason about the practice of mathematicians).

As far as I see it, fig is all about describing a distributed application as a set of connected containers.

Even looking at the simplest possible example of fig on its homepage, it seems to be about describing an app (the counter app) in terms of its flask server and redis database. That's the whole app, not just one service within it.

I've heard from fig users when talking to them that they love being able to package their entire distributed application within a single fig file.

As the author of fig I think you probably have a lot more visibility into fig users' current practice. Is what I've described so far consistent with your analysis?

If this is accurate, then, it seems really very different to the kubernetes pod concept, which seems primarily motivated to allow encapsulation of a component container along with its adjunct services, e.g. a web server with logging (as per @bgrant0607's quote).

My view is that the multi-layered approach in Kubernetes is certainly valid (and is a sophisticated way of describing apps), but we should either:

1. Go all in on that approach, and drag in the concepts of services as loosely-coupled (multi-host) collections of (tightly-coupled) pods, or,
2. Agree that groups are, for now, supposed to be simpler, higher level entities than this multi-layered approach. This view is sympathetic to current usage of fig which represents collections of containers which make up an application. This view is from the application developer's point-of-view, paying little heed at this point to the specifics of how it gets deployed - which is more of an ops concern.

Given that Docker is currently single-host, the second option would seem like a simpler, more incremental step.

In which case, I would support this proposal with the removal of the statement that "groups are supposed to be single host entities", and I suggest that we figure the multi-host story out later.

If we can agree on this, the next step would be a discussion of links (and perhaps volumes).

Thanks @aanand.

So for multi-host apps to work, we either need to make cross-host links exist first

This sounds like the remit of the "orchestration system"? It definitely feels outside the scope of the Docker binary. It's good that one can achieve this result without needing an orchestration system (the ambassadors pattern). Might make a nice side project for github.com/docker though.

I don't think that a shared network is the ideal solution - it seems that, just because nobody outside of Docker has provided this functionality yet, there's no reason to implement a sub-optimal solution here.

there’s also software which relies on the presence of somewhat conventional networking to do inter-node discovery

You should always be able to tack on Weave or something for this.

On the subject of environment variables - 12factor has these as a core tenet (http://12factor.net/config), and 12factor is a philosophy I thought Docker subscribed to. It's a shame when you have to resort to the hosts file but it's becoming less necessary as time goes on. Perhaps this is the wrong issue for this discussion though!

OK, I’ve slept on it and here are my conclusions:

• Nothing about grouping should be expressly single-host, or make distributing it across multiple hosts difficult enough that people will abandon it as soon as they scale.
• A shared hosts file is bad in a clustered scenario, for reasons outlined previously (essentially, it gives the clustering system less knowledge about what needs to talk to what, which restricts the decisions it can make). Links are better.
• Software which needs networking capabilities beyond what links provide should be able to run, but doesn’t need to be treated as a first-class use case. It’s fine to say “use something like Weave”, as long as that’s not a total hack.

As such, I propose to advance the current grouping implementation by dropping the shared hosts file and adopting a Fig-like links configuration option.

While thinking about multi-host, I also realised that creating/updating a group atomically (i.e. with a single JSON POST) is great in a multi-host scenario, because it lets the cluster make decisions on how exactly to do that. Docker’s single-host implementation can work differently than a multi-host one in terms of the actual steps taken (e.g. a fancy clustering system might choose to start new containers, update a load balancer, then let the old ones die etc). So we should definitely stick with that.

+1 😄

Great. So I guess the networking and possibly general multi-host semantics part of this discussion is closed until you complete a new proposal?

Seperately.. what is the reasoning behind:

A group has a unique name which occupies the same namespace as container names.

Feels like a hack.

Having groups as "folders" within the container namespace seems natural enough to me.

@robhaswell Multi-host semantics is likely going to need a lot more thinking about; I don't want to give the impression that discussion of that is "closed" in any way. If you have thoughts or concerns, please voice them!

Groups and containers sharing the same namespace is the opposite of a hack in my opinion: the way Fig separates things right now (by prefixing everything with a name) is much more hacky and less performant (it really breaks down when a docker server is being used for lots of things at once).

(This makes me realise I omitted that it'll be possible to filter docker ps and GET /containers/json to a single group.)

@shykes:

what would be the relationship between groups and volumes?

The current implementation does nothing special with volumes where the host path is explicitly specified. For unspecified host paths, we mount them at a predictable path:

/var/lib/docker/groups/<group-name>/<hash-of-container-path>

This is nice because it means that when we destroy and recreate containers, we don't need to bother with VolumesFrom and all the intermediate container shenanigans, which caused a lot of bugs in Fig.

would this deprecate container-to-container links, and the naming/hierarchy system that comes with it? If so, how would we deal with reverse compatibility?

I've been convinced that (semi-)deprecating links is a bad idea; this leaves the question of the naming/hierarchy system.

It's true that foo/bar can now mean either "a container inside group foo, referred to in the group JSON as bar" or "a container linked to from a container named foo under the alias bar". I think it's fine for these to coexist, though - there's still no way for them to actually clash, as you can't have both a container named foo and a group named foo. So in short: there's no need to deprecate anything.

I think:

Nothing about grouping should be expressly single-host

is incompatible with:

Containers within a group share a network namespace

It feels like this proposal is trying to blend two notion of grouping:

• one used for collocation where volume and network namespace are shared and discovery is implicit.
• one used for orchestration where network and volumes are isolated, and discovery is explicit through link and/or service definition.

I think it would be useful to highlight this more explicitly in the proposal description, and either explain how the implementation will allow to organize your groups seamlessly between these two notions or recognize this should be exposed as two different primitives.

It feels wrong having them share the same namespace, because you cannot perform the same operations on them. Also as the container namespace is externally controlled (by your registry; usually Docker) it opens up the possibility for unexpected namespace collisions - say there is a new public contributor with the same name as your group. Now you have to rename your group if you want to use their images. I like the proposal, just not shared namespaces.

I think everyone is saying "namespace" and meaning something different.

On Wed, Oct 22, 2014 at 3:18 PM, Rob Haswell notifications@github.com
wrote:

It feels wrong having them share the same namespace, because you cannot
perform the same operations on them. Also as the container namespace is
externally controlled (by your registry; usually Docker) it opens up the
possibility for unexpected namespace collisions - say there is a new public
contributor with the same name as your group. Now you have to rename your
group if you want to use their images. I like the proposal, just not shared
namespaces.

Reply to this email directly or view it on GitHub
#8637 (comment).

I feel going back to links configuration in a way simplifies this proposal in that less new concepts are introduced. I'd like to take it a bit further and reiterate that maybe we should not even add a Groups API. We need to tread lightly here. What a group is and how it will be used has been a large portion of this discussion. There's a very good reason for that. As we move towards orchestration and cross server composition of applications a group and the definition of that becomes a powerful construct. Groups form the bases of things like auto scaling, anti-affinity scheduling, and many other things.

The basics of how Docker orchestration across many servers will work going down the line is really yet to be determine. It's evident in this discussion that while Kubernetes may be the predominate Docker orchestration platform at the moment, the approach of wrapping Docker may not be the way we want to go ultimately. Because of the apparent uncertainty of orchestration in Docker, I would defer introducing any orchestration related features until we have a clearer direction.

If Docker was to allow arbitrary metadata to be attached to a container (which I believe has been asked for many times), fig like functionality could be implemented completely in the Docker client and not require much change to the Docker Remote API.

👍

I find this proposal very interesting and have a few points of feedback. At Discourse we are using runit for which I know you are actively trying to provide a workable solution for. So here are some ideas in no particular order:

sockets

If a group is on a single machine it is key to have access to sockets, a prime example is hosting of a Rails app. The application itself runs in a process called unicorn and NGINX fronts it, for best performance/security its ideal to use sockets for this communication.

Thinking of a more general solution here groups should be able to define a shared portions of the filesystem either volume like on the host or internal between members of the groups.

Examples:

members A, B may elect to share /shared/bla ... which is only visible between the 2
member B may elect to grant read/write to member A on /shared/bla
members A,B may elect to have a volume on the host

yaml

In the year I have worked on Discourse Docker I have grown to detest yaml, even though it is great in theory end-users just keep mucking it up and it becomes impossible to debug.

I know this is "optional" but my vote would be to ship with nothing yaml in core docker (including CLI) and just shipping with:

• A feature limited Dockerfile like format
• A feature full json like format

If people want yaml they can always write the extra tooling to generate the feature full json.

dependencies

Start/stop process often has various dependencies, for example you must first start postgres and only after it started start the web, same goes for group shut down, this logic needs to be written

logging

we would need something like docker group logs to get the list of logs, how would would logs for a group integrate to other tooling etc needs to be thought out, what about stuff like log rotation and so on.

upgrades

Groups open up the window to allowing for seamless upgrades without needing to front stuff with haproxy.

For example say I have a web and data container and elect to upgrade the web container it would be super nice if it could start the new web unamed, ensure it is all started up and then switch it in to where the old web was.

distribution

Is the registry going to be group aware? Can we use it to distribute groups?

PS. I really wish we could have this kind of discussion on https://forums.docker.com/ it is far better suited at dealing with big discussions like this :)

As for YAML, it's really helpful that it's the standard for configuration input in cloud-init. That means users may supply Docker orchestration information as user-data to VMs launched on EC2, GCE, or OpenStack clouds. The user-data document could indicate that a VM should have (or install) Docker and should spawn and configure a group of containers as specified. That's powerful for anyone using Docker on a VM-based cloud infrastructure (which I imagine is a pretty large number of users, I hear EC2 is still popular...)

I wouldn't say to make the format YAML for just this singular reason, but it's certainly a very nice property in the short-term, for as long as cloud-init is a standard.

I don't see it referenced here so I thought I'd drop a pointer:

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/design/namespaces.md

In Kubernetes we've separated out some things related to this proposal:

• Object Namespaces -- the idea that different "objects" in the container/cluster management world can belong to different users/applications and operate independently. The doc above talks about that. Note that this is different from Linux kernel namespaces. We struggled with finding a better name here (I'm not sure that group is any better).
• Pod -- A set of co-managed containers that share resources on a single machines. Proposal for Docker here: Proposal: Make Pods (collections of containers) a first order container object. #8781
• Application Config -- Being able to capture all of the configuration that makes up an application is a higher level thing that we haven't nailed down in Kubernetes at this point. I see fig and systems like Panamax filling this need. These systems can get highly opinionated (think puppet, chef, etc.) and there will be variety. An application will typically consist of a set of "micro-services" or tiers in the form of sets of replicated pods. There are then connections between those. Describing, deploying and upgrading is the job of the app config system/layer.

Typically, the app config system would work within a namespace, but it doesn't define the namespace.

Update: see here for the current state of composition (which builds on top of grouping): https://gist.github.com/aanand/9e7ac7185ffd64c1a91a

I like this a lot, but I still have one big concern. It sounds like the client portion of this is going to be added to docker (core).

Right now if I want to run pre-release features of fig (or even features that maybe aren't considered general enough to be added to fig upstream, ex: docker/compose/issues/318), it's very easy for me to run a fork. I don't have to run a custom dockerd or rebuild the docker package at all, I can just run a custom client (fig).

If this support is added directly to the core docker client, I now have to rebuild everything to get client-only customizations.

I really wish/hope the client portions could be released as a separate package (docker-groups?) so that running a custom client is much less involved. An official docker release (debian/rpm packaging) could still bring together these repos and include all the clients, but it would make it easier for people to run their own fork of individual clients.

@aanand re: group.yml config format and name (from another issue)

My use-case for setting this explicitly has always been to keep it consistent between environments. Having to set an environment variable is not ideal for this case, so I really like being able to set it in the group.yml.

Could it be made optional, and still support an override from environment variable? I believe that should handle all use cases (consistent between environments, and changing for each environment).

I've been wondering about having a separate override file that could be unversioned.

group.yml:

name: foo
containers:
  ...

.override.yml:

name: bar

This could also be used to give users the default option of not specifying the name in a versioned file: if group.yml contains no name, docker up can automatically create an override file with a generated name in it.

However, this is more of a composition concern than a grouping concern. (Proposal for that should land later this week.)

-1 and it seems to make docker client more and more complicated. Fig or Kubernetes do this well and keep working on them.

Oops, forgot to close this as it's been deprecated in favour of #9694