cc @tianon @proppy @nathanleclaire @aanand @hopetobelievein @jeffmendoza @cpuguy83 @polvi

Ben, can you outline how the client's public key will get placed in the server's "authorized_keys" upon new host creation? I understand that we'll need to merge with the libtrust work before getting this fully implemented.

@bfirsh can you detail the integration points for host drivers? Are those similar to boot2docker-cli drivers?

Also can you hint at the implication on the installation experience for new user? Does that mean the boot2docker installer morph into a docker client install with support drivers?

Driver's responsibility to select OS and if possible (which should be in most cases) user specify-able, as well as a bootstrapping script. If the driver's provider provides boot2docker as an image, it should be used as a default.

@proppy Right now drivers implement this interface:

// Driver defines how a host is created and controlled. Different types of
// driver represent different ways hosts can be created (e.g. different
// hypervisors, different cloud providers)
type Driver interface {
    DriverName() string
    GetURL() (string, error)
    GetIP() (string, error)
    GetState() (state.State, error)
    Create() error
    SetConfigFromFlags(flags interface{}) error
    Remove() error
    Start() error
    Stop() error
    Restart() error
    Kill() error
    GetSSHCommand(args ...string) *exec.Cmd
    // Pause() error
}

and register themselves with a _ import in api/client/commands.go.

Feedback

First off, awesome work @bfirsh , @cpuguy83 and countless others who have been involved in this idea and moving this implementation forward. It's very exciting stuff.

I got interested in the movement around hosts and went off and implemented an Amazon EC2 driver. It's a lot of fun to be able to bootstrap a host pretty quickly from the command line.

Ben and the boot2docker-cli maintainers (which is where the original driver code came from) did a great job implementing an interface that's pretty easy to write to. Like anything new, there's plenty of rough surfaces that need to get ironed out before we even consider merging this into Docker.

My feedback on the proposal is as follows:

Implement docker hosts inspect foo

I've made a PR for this (bfirsh#4), I think it is a vital part of making an implementation of this succeed. Users shouldn't have to know of ~/.docker/hosts/myhost/config.json as an implementation detail.

Provide a way to track (and probably to plan) infrastructure changes

What is Docker about? Docker is about repeatability and about speed. If I can't take what I've created with Docker and repeat it elsewhere, fast, it is not much use to me. Therefore if Docker is going to get into the business of creating hosts to run Docker, it must provide a way to do so which is tracked and repeatable. Something like git is not just a tool to share code with others, it is a way to share code and associated metadata that is very useful if Something Goes Wrong (and it always does). Likewise docker hosts should be.

There is very real money in very real wallets on the line when hosts are created, so this is not something to be taken lightly.

I propose a docker hosts history command (or equivalent), as well as a mechanism for dry-runs with real-deal infrastructure (think Terraform's plan). Then you do your standard docker hosts create --driver ec2 kaiju, get back a graph of what will happen when you pull the trigger, and do the real thing with docker hosts apply (or whatever).

You could have a ~/.docker/hosts/foo/plan.json and/or ~/docker/hosts/foo/history.json to track these.

The way I envision this working is making drivers register their API requests (anything that mutates infra at least) and responses and considering it a bug if they don't. (all of my requests to the AWS API go through a common "gateway" function, so I know it would be pretty easy for me to implement but I don't know about other drivers) Alternatively we could try to implement some kind of base http.Client equivalent that logs stuff automatically.

Sync mechanism to keep hosts consistent across... hosts

This might entail the use of Docker Hub or a self-hosted mechanism, I'm not really sure. But most assuredly there needs to be a way to sync hosts across machines. The current implementation couples docker hosts extremely tightly to one machine (because it's all client side), if you use a different physical device, different VM, or server, you suddenly lose all of the stuff that went into creating your carefully curated flock of Docker hosts. And god forbid your hard drive catches on fire or something.

Move most of the code to a server-side API and implement a --bootstrap flag to optionally run it client-side

This takes care of the "install boot2docker using just the docker binary" use-case as well as the "I want to run docker hosts commands against a host that I created with docker hosts" fun stuff.

Figure out running commands against multiple hosts

I'm not sure exactly what form this will take (ability to select multiple active hosts?) - I'd prefer the community to throw in their 2c - but I do know that having access to only one host at a time will be a very big issue. Consider a basic infrastructure setup of three servers: two that run application code and a load balancer in front of the other two. To set this up (using Docker of course :)) with docker hosts in its current form I would have to switch the active host, pull the app image, switch the active host again, pull the app image, switch the active host again, and then pull the load balancer image. As much of this should be concurrent as possible or people just won't use it.

I'm super excited by this proposal and I encourage everyone to give the bfirsh/host-management branch a play (the Digital Ocean driver is probably pretty easy to get started with).

I have a few more things that I will add later as well 😄

Implement docker hosts inspect foo

+1, good to see this merged already!

Provide a way to track (and probably to plan) infrastructure changes

This would be rad as hell. I'd argue that this isn't necessarily a day 1 feature - but on the other hand, it'll be harder to add later (drivers might have to be significantly rewritten).

Sync mechanism to keep hosts consistent across... hosts
Figure out running commands against multiple hosts

I think as soon as we're talking about a set of hosts that share an application, we're out of the scope of host management.

If whatever system is making the decisions about what host to run what on, and what order to do things in, exposes the Docker API, then that could be a selectable host in docker hosts; otherwise I'm inclined to say it's none of our business.

Move most of the code to a server-side API and implement a --bootstrap flag to optionally run it client-side
This takes care of the "install boot2docker using just the docker binary" use-case as well as the "I want to run docker hosts commands against a host that I created with docker hosts" fun stuff.

Not sure I understand the second use case here.

The rest of my feedback:

Don't switch active host by default on docker hosts create

Think git checkout vs. git checkout -b. I would have an optional flag to switch the host upon creation, since if anything went wrong with the creation of the host etc. it's well possible that the user's docker is now completely hosed and they probably won't be able to fix it themselves (and they shouldn't have to). Several times developing a driver I would get into awkward situations where my code messed up but the default host would change, so not only would I have to blow away ~/.docker/hosts and then delete the created EC2 instance manually, but then I would have to do things like docker -H=unix:///var/run/docker.sock hosts active default or else I would get panics. Fine for someone with deep docker knowledge, but not so fine for the average end user.

Don't assume the path is happy

It's situations like this that we should be prepared for:

• What if the user's connection drops in the middle of host creation?
• What if the API responds with something we didn't expect, like respond with a 503 due to them hammering it with requests? (this actually did happen to me while trying to spin up like 20 hosts at once to test my driver, haha)
• What if the service messes up and mis-reports that an instance was created when it wasn't? etc.

Even if it's not our fault, we are the gateway and we will get blamed.

docker hosts -q doesn't work

This is probably not a big deal, but a small bug: -q works with docker hosts list and not with plain old docker hosts (and I almost always default to docker hosts without list since I'm lazy). This breaks the good ol' docker hosts rm $(docker hosts -q) idiom that people are used to from containers.

Consider making --driver blah an argument instead of a flag

So that basically, instead of :

docker hosts create --driver ec2 giantrobot

You have:

docker hosts create ec2 giantrobot

(this might actually help with the next point as well, since you could now try and figure out a way to chuck the driver-specific flags underneath the arg they belong to in docker hosts create ec2 -h or whatever)

Traditionally, UNIX flags have been for optional things, and arguments have been for required things; drivers are pretty close to required in our case (although --url for arbitrary hosts throws a weird wrench into things), so let's support them first class. Speaking of driverless hosts, what the heck happens when you run hosts commands on a host that was added without a driver? If we're going to support that, we must account for it.

Discuss how to clean up driver option flags

Right now it is a huge jungle of various flags for the various providers, and if/when we add more it will only get worse:

Usage: docker hosts create [OPTIONS] NAME

Create hosts

  --aws-access-key=""                                                                                    AWS Access Key
  --aws-image-id="ami-27939962"                                                                          AMI to use for the selected region
  --aws-instance-name=""                                                                                 Name of created instance
  --aws-instance-type="m1.small"                                                                         Type of instance to create
  --aws-instance-username="ubuntu"                                                                       Username for SSH on the instance (depends on AMI)
  --aws-region="us-west-1"                                                                               AWS Region
  --aws-secret-key=""                                                                                    AWS Secret Key
  --aws-security-group="docker-hosts"                                                                    Security group to use for the created instance
  --azure-docker-cert-dir=".docker"                                                                      Azure docker cert directory
  --azure-docker-port="4243"                                                                             Azure docker port
  --azure-image="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140724-en-us-30GB"    Azure image name
  --azure-location="West US"                                                                             Azure location
  --azure-name=""                                                                                        Azure name
  --azure-password=""                                                                                    Azure user password
  --azure-publish-settings-file=""                                                                       Azure publish settings file
  --azure-size="Small"                                                                                   Azure size
  --azure-ssh="22"                                                                                       Azure ssh port
  --azure-subscription-cert=""                                                                           Azure subscription cert
  --azure-subscription-id=""                                                                             Azure subscription ID
  --azure-username="tcuser"                                                                              Azure username
  -d, --driver="none"                                                                                    Driver to create host with. Available drivers: azure, digitalocean, ec2, none, virtualbox
  --digitalocean-access-token=""                                                                         Digital Ocean access token
  --digitalocean-image="docker"                                                                          Digital Ocean image
  --digitalocean-region="nyc3"                                                                           Digital Ocean region
  --digitalocean-size="512mb"                                                                            Digital Ocean size
  --no-install=false                                                                                     Do not install Docker during provisioning, assume it already exists
  --no-provision=false                                                                                   Do not provision the instance automatically
  --url=""                                                                                               URL of host when no driver is selected
  --virtualbox-boot2docker-url=""                                                                        The URL of the boot2docker image. Defaults to the latest available version
  --virtualbox-disk-size=20000                                                                           Size of disk for host in MB
  --virtualbox-memory=1024                                                                               Size of memory for host in MB

This is tough to parse quickly and e.g. having giant amounts of whitespace for everyone else because one of the azure defaults is really long is annoying. This should be cleaned up in the final implementation.

run and use instead of create and active commands, respectively

This one is kind of a minor UI nitpick, and I don't feel particularly strongly about it compared to some of the other points, but I would prefer docker hosts run --driver digitalocean foobar (since create in context of a container references both creating and starting one, just like we're doing here) with separate subcommands for create and start (and we have start already) just like docker. Additionally docker hosts use foobar would be nice since it's just easier to type and it is a verb.

docker hosts should report if the daemon is reachable or not, instead of just the instance status

This is one of those subtle things that you don't really think about until you get bitten by it, but I firmly believe that hosts should also have some kind of IsDaemonAvailable method to implement as well (it shouldn't be too bad - it will be something like a net.DialTimeout check on :2376 in most cases), and the results of that should be shown in docker hosts list. Otherwise things will race and if as an end user you assume that "host running" == "daemon available" it could be easy to screw yourself by not checking for the right thing.

Small footnote as well: there seems to not really be any enforced discipline around signing of the commits that are going into this branch, and that's going to really hurt a lot if we do end up re-using them.

This would be rad as hell. I'd argue that this isn't necessarily a day 1 feature - but on the other hand, it'll be harder to add later (drivers might have to be significantly rewritten).

Yes, I think this is absolute essential to implement and get correct now, or else people will not use this feature - we should log each mutating request and the received response, as well as have a staging / dry-run feature.

I think as soon as we're talking about a set of hosts that share an application, we're out of the scope of host management.

Fair enough, but what about a set of clients that share hosts? There needs to be some way to account for this. To continue the git comparison: you can easily set up remotes and track branches across different clients / machines, and I think users will want to run commands against the same hosts from different docker clients (I know I will).

Additionally, let's suppose that I have some SSH keys which were generated by docker hosts and are the only way to get into the created hosts on, say, my laptop: then I accidentally drop it into a puddle of water (or my hard drive gets corrupted somehow). With this implementation if that happens I am completely screwed.

If whatever system is making the decisions about what host to run what on, and what order to do things in, exposes the Docker API, then that could be a selectable host in docker hosts; otherwise I'm inclined to say it's none of our business.

Yes, let's not get into the business of aggregate host / resource scheduling etc. with this feature. But what's the answer to e.g. "how do I pull the same image concurrently on two or more hosts"? docker -H=myHost pull image & docker -H=myHost2 pull image?

Not sure I understand the second use case here.

I suppose what I was envisioning is exposing host management as an API, so that you could remotely run docker hosts commands. That way you could :

• Support concurrent operations (mostly you can't run hosts create in the background right now because it's all client-side)
• Run docker hosts commands on remote hosts themselves, creating a virtuous cycle and opening up the doors to take advantage of something like private IP addresses for additional speed etc.
• Support docker hosts operations elegantly on 3rd-party clients like fig (valuable part of the ecosystem). No daemon API, no docker-py bindings (unless I'm mistaken and docker-py assumes the presence of a docker binary on the local system- and I very well could be).

Discuss how to clean up driver option flags

How about docker hosts create --help takes the --driver option? Not sure if this is easily doable with the current help infrastructure.

I like the idea of the driver being an option to the hosts create command from a user mindset pov. Having each driver create be a separate "command" encourages discontinuity.

For now, we'll remove the default option for the azure image, and behave like the b2d iso:

--virtualbox-boot2docker-url=""     The URL of the boot2docker image. Defaults to the latest available version

@proppy Here are some examples of drivers: https://github.com/bfirsh/docker/tree/host-management/hosts/drivers

We plan to make this the primary installation experience for new users on OS X and we'll be deprecating boot2docker-cli. For Windows, there is ongoing work to add Windows support to the Docker client, so it depends if that lands or not.

@nathanleclaire Thanks for your pull requests Nathan, really appreciate it.

I agree we need a sync mechanism. Version 2 of host management will be a hosted service, probably as part of the Hub, but this is a first step to get us towards that. The migration path makes sense, I think. In the future version of Docker it'll ask you, "do you want to push your hosts up to the Hub?" (think iCloud migration).

With regards to selecting the host on create: I think this makes sense from a behaviour point of view. I think your objection is that sometimes it throws panics, and it shouldn't do that. It's a known issue that Docker won't when a stopped host is active, and I'm currently refactoring the client to make that work.

We should be using create, because that maps with docker create. I'm not 100% sold on "active", but I haven't come up with anything better yet. I like how docker hosts active prints the current active host, which is why I don't think use will work, but maybe that's not important. Easy to change, so I'll let this discussion continue!

In terms of cleaning up CLI commands, I went back and forth on this design with @tianon, so he might have opinions.

I don't think there's anything wrong with long help commands. In fact, I like them, because it makes it easy to get a quick overview of how the whole thing works. I made the Docker help much longer for this reason.

I don't think it makes sense for there to be separate commands for each driver. It's optional, for a start.

I think a good solution could be to group the options based on what driver they're for. It'll make the help text easier to scan and won't complicate the design.

/cc @cpuguy83

I have added some builds to the description if you'd like to try it out.

Er, just a quick sanity check to make sure I'm not missing something obvious. In the example you have

$ docker -H dev echo hello I am dev
hello I am dev

Is that supposed to be an image named echo on the host dev with whatever entrypoint (presumably echo) and the args I am dev? Because I don't see where you pulled or built an image named echo – I think you must've meant to run it on busybox.

@kojiromike Oops. Yes. Thank you!

+1 Adding this as a separate tool would be great.

It feels irresponsible to add remote host functionality to core before the necessary security, permissions and logging are supported and stable.

Discussion of this functionality and similar comments by @nathanleclaire at 50+ minutes into: http://new.livestream.com/primeimagemedia/events/3532966

I've updated the binaries in the description with a more recent build.

@inthecloud247 while we agree this is terrific functionality that Docker users will love to have, I disagree sharply with the idea that we should provide it as a separate tool. This feature set is so incredibly compelling that every docker user should have access to it built in. Yes, the default provider list will need to be carefully curated and managed, and this will cause docker to be larger. These drawbacks are well worth the advantage of making a simple and effective union between a local docker environment, a group of local docker servers, and even a collection of clouds where workloads can be run. Let's find a way to get at least support providers who can assign maintainers to provide for the long term care and feeding of their contributions.

Some of the multi-host functionality seems to be available now using fig... just need to merge the changes proposed by the recent Docker Global Hack Day 2 winner. The pull request has been open for 11 days:

docker/compose#607
https://blog.docker.com/2014/11/announcing-docker-global-hack-day-2-winners/

Kinda sad if he wins the hack day and the pull request gets closed :-(

Does the driver expect to be able control the daemon? or is that cool if the daemon start with the VM?

Also if the VM generates the certs at boot, it there a standard way for the hosts code to pull the client certs over ssh? or should it be done per driver?

@proppy The driver just needs to boot up the vm with the daemon listening on an available port. Weather that means installing the daemon or having docker baked into the image, it doesn't matter.

None of the drivers have security at the moment, we're moving away from the CA based certs to ssh style: #8265

I'd like to get an interface into that PR for the driver to be able to pull the clients 'public key,' so that the driver can properly configure the deamons 'authorized_clients' file, but haven't had a chance. We'll need this when these features merge.

Love this proposal 👍

I just filed an issue over at bfirsh#18 as we have several VMware drivers ready to be PR'd (vCloud Air, Fusion and vSphere), can you guys shed some light on what are the plans for this proposal?

@jeffmendoza does the driver assume it has lifecycle control over the daemon?

In the current spec does:

    Start() error
    Stop() error
    Restart() error

refers to the host or the daemon? or it doesn't matter?

@proppy Those refer to the host. Example: you would want to use docker hosts stop to save money on your public cloud based host.

For those working on host management, just a heads up: I have started to squash my branch and base it on top of #8265. The new branch is here: https://github.com/bfirsh/docker/tree/host-management-with-client-daemon-auth

I will transfer any changes from the host-management branch over to that branch, but if you start building on top of that branch now, the merge conflicts might be less painful. :)

Man docker is moving so fast.

I have now rebased the host management branch on top of #8265 and squashed it:

https://github.com/bfirsh/docker/compare/host-management

Any pull requests should now be based on top of that. The driver interface hasn't changed, so it should be a trivial matter to rebase any existing pull requests. The main thing which has changed is that drivers are expected to set up identity auth for communication with the host. See this commit for an example of how to do so.

The old branch is here for reference.

Full update and preview builds coming soon.

Has there been any progress on splitting the actual driver implementations
out of the core binary?

@tianon Not really official sanctioned, but I've been playing around with implementing plugins, specifically implenenting a graphdriver for plugins, which could be used for this, in theory: https://github.com/cpuguy83/docker/tree/poc_storage_plugin

Thanks for your feedback, everyone. Host management is now Docker Machine!

https://github.com/docker/machine

Silly me, I assumed we were having an open discussion here... Secrets within puzzles inside a mystery.

It's been real.