Safely setup Ubuntu vm with Terraform and Cloud-init - docker

For personal use (and fun) I'm trying to setup a VM on which I want to host my website (Nginx, Django and Postgres running in docker containers). I'm trying to learn how to setup the server using Terraform and Cloud init in a safe manner.
My current cloud-init code:
#cloud-config
groups:
- docker
users:
- default
# the docker service account
- name: test
shell: /bin/bash
home: /home/test
groups: docker
sudo: ALL=(ALL) NOPASSWD:ALL
ssh_import_id: None
lock_passwd: true
ssh-authorized-keys:
- ssh-rsa my_public_ssh_key
package_update: true
package_upgrade: true
packages:
- git
- sudo
- apt-transport-https
- ca-certificates
- curl
- gnupg-agent
- software-properties-common
runcmd:
# install docker following the guide: https://docs.docker.com/install/linux/docker-ce/ubuntu/
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
- sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- sudo apt-get -y update
- sudo apt-get -y install docker-ce docker-ce-cli containerd.io
- sudo systemctl enable docker
# install docker-compose following the guide: https://docs.docker.com/compose/install/
- sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- sudo chmod +x /usr/local/bin/docker-compose
power_state:
mode: reboot
message: Restarting after installing docker & docker-compose
The VM is Ubuntu 20.04
Technically I want the "test" user to be able to pull the latest code from my git repo and (re-)deploy the website (in /home/test/website) using docker-compose. Is it possible that the user does not have sudo permissions (I don't want to have it have elevated permissions). And secondly: how do I create a root account with a separate SSH key (and would this be a safe setup)?
The Terraform code that produces the VM.
resource "scaleway_instance_server" "app_server" {
type = var.instance_type
image = "ubuntu-focal"
name = var.instance_name
enable_ipv6 = true
tags = [ "FocalFossa", "MyUbuntuInstance" ]
root_volume {
size_in_gb = 20
delete_on_termination = true
}
lifecycle {
create_before_destroy = true
}
ip_id = scaleway_instance_ip.public_ip.id
security_group_id = scaleway_instance_security_group.www.id
# cloud init: setup
cloud_init = file("${path.module}/cloud-init.yml")
}
Help is much appreciated.

Is it possible that the user does not have sudo permissions (I don't want to have it have elevated permissions).
Anything run by cloud-init is run as root, including the bootcmd/runcmd commands. To run things as a different user, you can use sudo in your runcmd.
sudo -u test whoami >> /var/tmp/run_cmd
would write test to /var/tmp/run_cmd.
And secondly: how do I create a root account with a separate SSH key (and would this be a safe setup)?
Your users section would something look like this.
users:
- default
# the docker service account
- name: test
shell: /bin/bash
home: /home/test
groups: docker
sudo: ALL=(ALL) NOPASSWD:ALL
lock_passwd: true
ssh-authorized-keys:
- ssh-rsa my-public-key
- name: root
ssh-authorized-keys:
- ssh-rsa root-public-key
disable_root: false
Is it safe? I think that's debatable, but there's a reason root login is disabled by default. It should be possible to ssh into the default user and then sudo su for your root access needs.
Also, just FYI, the ssh_import_id: None in your config was raising an exception in the cloud-init log because it was trying to import an ssh id for user None.

Related

Docker jrcs/letsencrypt-nginx-proxy-companion doesn't generate a proper certificate

I'm following a tutorial to deploy Wordpress using Docker on a Ubuntu server. The tutorial is in this website.
It's important to mention that I already have two subdomains at this point, one for the Wordpress site and another for the phpMyAdmin site.
However the letsencrypt certificates seem to not be generated properly. I can access the website via http, but not https, and when I look at the certificate it doesn't look correct. In fact it doesn't seem to have one for my website.
To make everything easier I created a script to run all the steps fast:
#!/bin/bash
web_dir=/srv/www
myusername=root
domain_name=subdomain.domain.com
website_folder=/srv/www/$domain_name
nginx_proxy_repo=https://github.com/kassambara/nginx-multiple-https-websites-on-one-server
nginx_folder=/srv/www/nginx-multiple-https-websites-on-one-server/nginx-proxy
final_nginx_folder=/srv/www/nginx-proxy
echo ---INSTALL REQUIRED COMPONENTS----
sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
sudo apt update
apt-cache policy docker-ce
sudo apt install docker-ce docker-compose git
sudo systemctl status docker
echo ---CREATE AND GIVE PERMISSIONS TO WEBSITES DIR----
sudo mkdir -p $web_dir
# 2. set your user as the owner
sudo chown -R $myusername $web_dir
# 3. set the web server as the group owner
sudo chgrp -R www-data $web_dir
# 4. 755 permissions for everything
sudo chmod -R 755 $web_dir
# 5. New files and folders inherit
# group ownership from the parent folder
chmod g+s $web_dir
echo ---INSTALL NGINX PROXY----
git clone $nginx_proxy_repo $web_dir
rm -rf $web_dir/nginx-proxy/nginx.tmpl
curl -s https://raw.githubusercontent.com/jwilder/nginx-proxy/master/nginx.tmpl > $web_dir/nginx-proxy/nginx.tmpl
cd $web_dir
rm -rf your-website-one.com your-website-two.com README.Rmd .gitignore .Rbuildignore .git README.md
echo ---INSTALL WORDPRESS----
cd $web_dir
git clone https://github.com/kassambara/wordpress-docker-compose $domain_name
echo ---CONFIGURE DOCKER COMPOSE FOR ONLINEHOST----
cd $website_folder
mv docker-compose-onlinehost.yml docker-compose.yml
echo ---FINAL TOUCHES----
cd $website_folder
vi ./setup-onlinehost.sh
chmod +x setup-onlinehost.sh && ./setup-onlinehost.sh
vi .env
vi docker-compose.yml
cd $final_nginx_folder
docker network create nginx-proxy
docker-compose up -d
cd $final_nginx_folder
cd vhost.d
echo "client_max_body_size 64M;" > $domain_name
cd $website_folder
docker-compose up -d --build
docker-compose -f docker-compose.yml -f wp-auto-config.yml run --rm wp-auto-config
When the time comes I setup the setup-onlinehost.sh like this:
project_name="wordpress"
user_name="wordpress"
pass_word="wordpress"
email="mail#gmail.com"
website_title="My Blog"
website_url="https://subdomain.domain.com"
phmyadmin_url="sqlsubdomain.domain.com"
env_file=".env"
compose_file="docker-compose.yml"
Then I remove the redirectnonwww container from the docker-compose.yml file since I don't want the redirect non-www to www behavior.
Then after everything is completed, I can access the websites over http but not over https. When I try to access it over https I receive a message about This connection is not private and the certificate seems to be wrong at this point.
Also If I let continue my browser to visit the website I got to the Nginx 500 Internal Server Error.
If I look into the contents of nginx-proxy/certs I see listed the following items:
certs (folder)
default.crt
default.key
dhparam.pem
subdomain.domain.com (empty folder)
sqlsubdomain.domain.com (empty folder)
conf.d (folder)
docker-compose.yml
html
nginx.tmpl
vhost.d (folder)
subdomain.domain.com (file)
The contents of vhost.d/subdomain.domain.com are:
## Start of configuration add by letsencrypt container
location ^~ /.well-known/acme-challenge/ {
auth_basic off;
auth_request off;
allow all;
root /usr/share/nginx/html;
try_files $uri =404;
break;
}
## End of configuration add by letsencrypt container
client_max_body_size 64M;
I'm not sure if I'm doing something wrong or if I should be doing something else that is not listed on the tutorial.
The issue seemed to be the number of times I had requested a certificate for those specific domains. I tried the deploy multiple times to figure out how to do it properly for the deployment server and also to write a proper version of the script, that I requested many times a certificate for two specific domains.
The issue was resolved after I tried a different domain and subdomain.

Error when starting custom Airflow Docker Image GROUP_OR_COMMAND

I created a custom image with the following Dockerfile:
FROM apache/airflow:2.1.1-python3.8
USER root
RUN apt-get update \
&& apt-get -y install gcc gnupg2 \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update \
&& ACCEPT_EULA=Y apt-get -y install msodbcsql17 \
&& ACCEPT_EULA=Y apt-get -y install mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc \
&& echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc \
&& source ~/.bashrc
RUN apt-get -y install unixodbc-dev \
&& apt-get -y install python-pip \
&& pip install pyodbc
RUN echo -e “AIRFLOW_UID=$(id -u) \nAIRFLOW_GID=0” > .env
USER airflow
The image creates successfully, but when I try to run it, I get this error:
"airflow command error: the following arguments are required: GROUP_OR_COMMAND, see help above."
I have tried supplying a group ID with the --user, but I can't figure it out.
How can I start this custom Airflow Docker image?
Thanks!
First of all this line is wrong:
RUN echo -e “AIRFLOW_UID=$(id -u) \nAIRFLOW_GID=0” > .env
If you are running it with Docker Compose (I presume you took it from https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html), this is something you should run on "Host" machine, not in the image. Remove that line, it has no effect.
Secondly - it really depends what "command" you run. The "GROUP_OR_COMMAND" message you got is the output of "airflow" command. You have not copied the whole output of your command but this is a message you get when you try to run airflow without telling it what to do. When you run the image you will run by default the airflow command which has a number of subcommands that can be executed. So the "see help above" message tells you the very thing you should do - look at the help and see what subcommand you wanted to run (and possibly run it).
docker run -it apache/airflow:2.1.2
usage: airflow [-h] GROUP_OR_COMMAND ...
positional arguments:
GROUP_OR_COMMAND
Groups:
celery Celery components
config View configuration
connections Manage connections
dags Manage DAGs
db Database operations
jobs Manage jobs
kubernetes Tools to help run the KubernetesExecutor
pools Manage pools
providers Display providers
roles Manage roles
tasks Manage tasks
users Manage users
variables Manage variables
Commands:
cheat-sheet Display cheat sheet
info Show information about current Airflow and environment
kerberos Start a kerberos ticket renewer
plugins Dump information about loaded plugins
rotate-fernet-key
Rotate encrypted connection credentials and variables
scheduler Start a scheduler instance
sync-perm Update permissions for existing roles and optionally DAGs
version Show the version
webserver Start a Airflow webserver instance
optional arguments:
-h, --help show this help message and exit
airflow command error: the following arguments are required: GROUP_OR_COMMAND, see help above.
when you extend the official image, it will pass the parametor to "airflow" command which causing this problem. Check this out: https://airflow.apache.org/docs/docker-stack/entrypoint.html#entrypoint-commands

Pull from GCR inside GCE vm on Ubuntu 20.04

I haven't set up a GCE stack in a while, and I swear this gets more difficult over time.
So the setup's easy enough: Blank ubuntu VM, installed docker via snap. Now when I try a pull from GCR, I get
> docker pull gcr.io/.../image
Using default tag: latest
Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Fair enough. I checked my gcloud command:
> gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* ...-compute#developer.gserviceaccount.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
So the right service account is there. In IAM it's listed as an editor and for good measure, I added storage admin too.
Now I run
> gcloud auth configure-docker
WARNING: `docker-credential-gcloud` not in system PATH.
gcloud's Docker credential helper can be configured but it will not work until this is corrected.
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
After update, the following will be written to your Docker config file
located at [/home/y/.docker/config.json]:
{
"credHelpers": {
"gcr.io": "gcloud",
"marketplace.gcr.io": "gcloud",
"eu.gcr.io": "gcloud",
"us.gcr.io": "gcloud",
"staging-k8s.gcr.io": "gcloud",
"asia.gcr.io": "gcloud"
}
}
Do you want to continue (Y/n)?
Docker configuration file updated.
And according to gcp's documentation, the warning is fine. gcloud can be used as an alternative to the standalone helper. But still: the pull fails. Bummer.
According to the documentation, sudo is a bad idea. So I tried adding my user to the docker group and apparently that clashes with snap. I ran
> sudo addgroup --system docker
> sudo adduser $USER docker
> newgrp docker
> sudo snap disable docker
> sudo snap enable docker
So now I can use docker with my account.
The issue still persists though. I also tried the standalone helper with
> VERSION=2.0.0
> OS=linux # or "darwin" for OSX, "windows" for Windows.
> ARCH=amd64 # or "386" for 32-bit OSs, "arm64" for ARM 64.
> curl -fsSL "https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v${VERSION}/docker-credential-gcr_${OS}_${ARCH}-${VERSION}.tar.gz" | tar xz --to-stdout ./docker-credential-gcr | sudo tee /usr/local/bin/docker-credential-gcr && sudo chmod +x /usr/local/bin/docker-credential-gcr
> docker-credential-gcr configure-docker
I've been troubleshooting this for too long, what's going on here?
Snap seems to have caused the issues here. Somewhere between snap-specific configuration files for the helpers and the snap-install gcloud SDK, the error happened. I went with a fresh installation and apt only:
sudo snap remove google-cloud-sdk
sudo apt update; sudo apt upgrade -y
sudo apt install docker.io
sudo curl -L --fail https://github.com/docker/compose/releases/download/1.25.5/run.sh -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker -v
sudo usermod -a -G docker $USER
## new shell
# exit
curl https://sdk.cloud.google.com | bash
gcloud auth configure-docker
. ~/.bashrc
sudo ln -s $(which gcloud) /usr/bin/
gcloud auth configure-docker

"rsync" was not detected as installed in your guest machine

I'm trying to setup Vagrant with docker as a provider but when running
vagrant up --provider=docker --debug
I get this error:
"rsync" was not detected as installed in your guest machine. This
is required for rsync synced folders to work. In addition to this,
Vagrant doesn't know how to automatically install rsync for your
machine, so you must do this manually.
Full log here:
http://pastebin.com/zCTSqibM
Vagrantfile
require 'yaml'
Vagrant.configure("2") do |config|
user_config = YAML.load_file 'user_config.yml'
config.vm.provider "docker" do |d|
d.build_dir = "."
d.has_ssh = true
d.ports = user_config['port_mapping']
d.create_args = ["--dns=127.0.0.1","--dns=8.8.8.8", "--dns=8.8.4.4"]
d.build_args = ['--no-cache=true'] end
config.vm.hostname = "dev"
config.ssh.username = "it" config.ssh.port = 22 config.ssh.private_key_path = ["./initial_ssh_key", user_config['ssh_private_key_path']] config.ssh.forward_agent = true
end
Dockerfile
FROM debian:jessie MAINTAINER IT <it#email.com>
RUN echo 'exit 0' > /usr/sbin/policy-rc.d
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
RUN apt-get update RUN apt-get upgrade -y RUN apt-get install sudo apt-utils -y
RUN apt-get -y install sysvinit-core sysvinit sysvinit-utils RUN cp /usr/share/sysvinit/inittab /etc/inittab RUN apt-get remove -y --purge
--auto-remove systemd libpam-systemd systemd-sysv
RUN apt-get install ssh -y
RUN addgroup --system it RUN adduser --system --disabled-password
--uid 1000 --shell /bin/bash --home /home/it it RUN adduser it it RUN adduser it sudo
RUN echo "it ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
ADD initial_ssh_key.pub /home/it/.ssh/authorized_keys RUN chown it:it /home/it/ -R RUN echo "Host * \n\tStrictHostKeyChecking no" >> /etc/ssh/ssh_config
CMD exec /sbin/init
Note:
I'm on Mac OS X 10.12 and I've installed vagrant, virtualbox and docker I have rsync installed and added to my PATH in the host machine.
Also, the same vagrant and docker configs works perfectly on a ubuntu host.
How do I install rsync in the guest machine? Or is something else wrong with my config? Any ideas?
You may want to give the alternative boot2docker box a try: https://github.com/dduportal/boot2docker-vagrant-box
as it contains rsync while the hashicorp/boot2docker, which is used by default, seems to lack this!
If doing so, you must add the follwong line to your docker provider config (of course adopted to your system):
d.vagrant_vagrantfile = "../path/to/Vagrantfile"
This is because you're changing the docker provider host vm as described in the vagrant docker provider documentation.
Try adding rsync to your Docker file, somewhere in one of your apt-get lines. Linux hosts use NFS by default, that's why it works on your Ubuntu.
Normally Vagrant tries to install rsync on a guest machine, if that fails - it notifies you with that error message. More info on vagrant website (3rd paragraph in "Prerequisites" chapter)

Vagrant - Rails Not Installed

I recently had to destroy and recreate my Vagrant instance. Now I can't run any rails command as it says Rails is not installed. When I did
Vagrant Up
I got the following error
default: /tmp/vagrant-shell: line 1: /home/vagrant/.rvm/scripts/rvm: No such file or directory
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
My Provision.sh file contains the following:
echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main " | sudo tee -a /etc/apt/sources.list.d/pgdg.list
sudo wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get remove postgresql-client-9.1 postgresql-client-common postgresql-client postgresql-common -y
sudo apt-get install postgresql-9.3 postgresql-client-9.3 libpq-dev curl git build-essential libxslt-dev libxml2-dev -y
wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
curl -sSL https://get.rvm.io | bash -s stable --ruby
cat << EOF | sudo tee -a /home/vagrant/.bashrc
cd /vagrant
EOF
echo '# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust' | sudo tee /etc/postgresql/9.3/main/pg_hba.conf
echo 'machine api.heroku.com
login *****
password ****
machine code.heroku.com
login *****
password *****
' | tee /home/vagrant/.netrc
echo 'ssh-rsa ***** vagrant#precise32
' | tee /home/vagrant/.ssh/id_*****.pub
chmod 0600 /home/vagrant/.netrc
sudo -u postgres psql -c "create user ***** createdb createuser password '*****';"
sudo /etc/init.d/postgresql restart
I have seen some answers (not specific to Vagrant) suggesting that I must have installed rvm using sudo or as root and need to remove it and then get rvm again. I have tried to do that butI'm not sure how it applies to a vagrant box and at any rate I must have done it wrong as it hasn't worked.
Is there something I need to correct/add to my provision.sh file or to my Vagrantfile?
Vagrant runs the provisioning file as root, so you would have indeed installed rvm as root unless you specified otherwise*. This was quite confusing for me as well (also a newbie), I would install things during provisioning and they would "disappear". In fact, they were all being installed / set as root.
*Or, you manually installed rvm when ssh'd into the machine, which I'll touch on more below.
You can switch your user using su -c "source /home/vagrant/myapp/vagrant/user-config.sh" vagrant
What goes in the "" is any command you want to execute. In this case, we're switching to a separate shell file user-config.sh that contains all the commands that should not be run as root, such as installing RVM.
I also sense somewhat of a conceptual misunderstanding. Each time you do vagrant destroy your entire virtual machine is destroyed, hard drives and all. The next time you do vagrant up, everything is rebuilt from scratch. If you had ssh'd in and installed things, they'll no longer be there.
This means that all of your install and config goes into the provisioning file, and you shouldn't be installing things manually after the fact. You should be able to vagrant destroy any time you want.
Take a read through https://coderwall.com/p/uzkokw/configure-the-vagrant-login-user-during-provisioning-using-the-shell-provider once more, I'm hoping it makes more sense this time around.
May be this link helps you to install rvm using Vagrant.
RVM_Vagrant

Resources