Docker-machine Google (GCE) driver not working, unable to create instance - docker

Question

Okay, this is very easy to reproduce and incredibly frustrating. Would be super grateful for any help or advice! I'm using Docker for Mac, running on OS X El Capitan (10.11.6). The gist is that Docker seems to not work with Google Compute Engine (GCE) via the Docker GCE driver (Docker official docs reference here).
1: Sign up for a new free GCP (Google Cloud) account at http://console.cloud.google.com/. Also download and install the Google Cloud SDK from here: https://cloud.google.com/sdk/.
2: Create a new Google Cloud project.
3: Go to "API Manager" in the Google Cloud console and click on "Credentials"
4: Click on "Create Credentials" and select "Service Account Key". Select "Compute Engine default service account", make sure JSON is selected as the output type, and click "Create". Move the outputted JSON file to your user root directory (/Users/MYUSERNAME).
5: Add the following line to your .bash_profile config:
export GOOGLE_APPLICATION_CREDENTIALS=/Users/MYUSERNAME/NAME_OF_CREDENTIALS_FILE.json. Save the file.
6: Exit the terminal and open up a new one so that the env variable is now set.
7: Run gcloud config set project PROJECT_ID (where PROJECT_ID is the name of the project just created in the Google Cloud Console).
8: Run gcloud auth login which will open a browser tab to log you into Google and grant permissions. Click 'Allow'.
9: Now the fun part, run the following command, per the Docker documentation (I've added a --debug flag):
docker-machine --debug create --driver google --google-project PROJECT_ID vm01
('vm01' is the name of the virtual machine here, this could be anything you want.)
At the end of the very lengthy output I get the following, concluded by the error message at the very bottom:
(LOTS OF OTHER OUTPUT BEFORE THIS, NOT ABLE TO COPY-PASTE EVERYTHING DUE TO STACKOVERFLOW 30000 CHAR LIMIT)
(vm01) Calling .GetURL
(vm01) Calling .DriverName
Setting Docker configuration on the remote daemon...
(vm01) Calling .GetSSHHostname
(vm01) Calling .GetSSHPort
(vm01) Calling .GetSSHKeyPath
(vm01) Calling .GetSSHKeyPath
(vm01) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/nathan/.docker/machine/machines/vm01/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker-user#104.198.166.134 -o IdentitiesOnly=yes -i /Users/nathan/.docker/machine/machines/vm01/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
printf %s "[Service]
ExecStart=/usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=google
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
Environment=
[Install]
WantedBy=multi-user.target
" | sudo tee /etc/systemd/system/docker.service
SSH cmd err, output: <nil>: [Service]
ExecStart=/usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=google
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
Environment=
[Install]
WantedBy=multi-user.target
(vm01) Calling .GetSSHHostname
(vm01) Calling .GetSSHPort
(vm01) Calling .GetSSHKeyPath
(vm01) Calling .GetSSHKeyPath
(vm01) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/nathan/.docker/machine/machines/vm01/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker-user#104.198.166.134 -o IdentitiesOnly=yes -i /Users/nathan/.docker/machine/machines/vm01/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo systemctl daemon-reload
SSH cmd err, output: <nil>:
(vm01) Calling .GetSSHHostname
(vm01) Calling .GetSSHPort
(vm01) Calling .GetSSHKeyPath
(vm01) Calling .GetSSHKeyPath
(vm01) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/nathan/.docker/machine/machines/vm01/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker-user#104.198.166.134 -o IdentitiesOnly=yes -i /Users/nathan/.docker/machine/machines/vm01/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo systemctl -f start docker
SSH cmd err, output: <nil>:
(vm01) Calling .GetSSHHostname
(vm01) Calling .GetSSHPort
(vm01) Calling .GetSSHKeyPath
(vm01) Calling .GetSSHKeyPath
(vm01) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/nathan/.docker/machine/machines/vm01/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker-user#104.198.166.134 -o IdentitiesOnly=yes -i /Users/nathan/.docker/machine/machines/vm01/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
netstat -tln
SSH cmd err, output: <nil>: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 10.0.3.1:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp6 0 0 :::22 :::* LISTEN
Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded
notifying bugsnag: [Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded]

Answer 1

Solved this just now. I used an updated image from the Google registry (Ubuntu 16.04 LTS, versus the default Ubuntu 15 that gets used by the docker-machine --driver google command) and it seems to have worked properly. Not sure why. The full command was:
docker-machine --debug create --driver google --google-project PROJECT_ID --google-machine-image https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1604-xenial-v20161205 vm02