I run Rstudio server mostly from an EC2 instance. However, I´d also like to run it from a cluster at work. They tell me that I can setup docker with rstudio and make it run. Now, I´d also like the Rstudios both on EC2 and work to have the same packages and the same versions available. Any idea how I can do this? Can I have both version point to a dropbox folder? In that case, how can I mount a dropbox folder?
You should setup a docker repository on dockerhub or aws ec2 container service (ecs). ECS is a managed service that allows you to easily deploy docker containers onto a cluster of 1 or more ec2 instances that are running the ecs agent (an aws program that helps that cluster work with the ecs). The Dockerfile should install all packages that you need at build time of the image. I suggest referencing the aws ecs documentation, which includes a walkthrough to get you going very quickly (assuming you have an idea of how docker works): https://aws.amazon.com/documentation/ecs/
You should then always run from that docker image, whether you are running on a local or remote machine. One key advantage of docker is that it keeps your application's environments the same (assuming you use the same build of the image) regardless of the host environment.
I am not sure why would not always run on ECS (we have multiple analysts using RStudio, and ECS lets us provision cpu/memory resources to each one, as well as autoscale as needed). You could install docker on EC2 and manage it that way, but probably easier to just install the ecs agent (or use the ecs optimized ec2 ami which has it preinstalled - the docs above walk through configuring it), and use ECS to launch rstudio services.
Related
I have built a docker image that when run, it registers itself as a GitHub Runner. This runner will, amongst other things, be used to build and push images to GitHub Container Registry. I don't want to deploy the containers to GKE or Compute, as I don't want the overhead of managing those resources. I would prefer to deploy the containers to Google Cloud Run. I've scoured the docs for help but I can't seem to find the answers to the following question:
Can I run 'docker in docker' when the container is deployed to GCP Cloud Run?
How do I specify the volume mount required when deploying the container to Google Cloud Run, i.e. the usual mapping with docker run would be:
-v /var/run/docker.sock:/var/run/docker.sock
I never tested but it's possible that the current Cloud Run sandbox prevent this king of use. And I don't really know the use case for this!
You can't mount volume in Cloud Run, it's stateless. You only have a in-memory file system in the /tmp directory (and it's in memory, size correctly your Cloud Run instance memory to take this into account). You can connect your instance to 3rd party product, such as Google Cloud Storage or databases, but no volume mountable on Cloud Run (for now)
If you have these requirement you can maybe have a look to autopilot and deploy directly your container on fully managed K8S.
My application is built using 3 Docker services:
backend (React)
frontend (Node.js)
nginx (routing traffic)
Up until now I was manually logging into an own Digital Ocean server, cloning the repository and launching the services with docker-compose build && docker-compose up.
I want to automate the process from now on.
Given Gitlab CI/CD Pipelines and the runners, what would be the best approach to automatically deploy the code to Digital Ocean server?
[WHAT I WAS THINKING OF, might seem very "beginner"]
Idea 1: Once a commit was pushed to master -> Gitlab runner will build the services and then copy it over to the DO server via scp. Problem: how do you launch the services? Do you connect to the DO server via ssh from the runner and then run the start script there?
Idea 2: Register a worker on the DO server just so when it pulls the data from Gitlab it has the code on the DO server itself. It just has to build them and run. But this approach is not scalable and seems hacky.
I am looking for some thinking guidelines or a step-by-step approach.
One of the benefits of using Docker in a production-deployment scenario is that you don't separately scp your application code; everything you need is built into the image.
If you're using an automation system like Ansible that can directly run containers on remote hosts then this is straightforward. Your CI system builds Docker images, tags them with some unique version stamp, and pushes them to a repository (Docker Hub, something provided by your cloud provider, one you run yourself). It then triggers the automation system to tell it to start containers with the image you built. (In the case of Ansible, it runs over ssh, so this is more or less equivalent to the other ssh-based options; tools like Chef or Salt Stack require a dedicated agent on the target system.)
If you don't have an automation system like that but you do have ssh and Docker Compose installed on the target system, then you can copy only the docker-compose.yml file to the target host, and then launch it.
TAG=...
docker push myname/myimage:$TAG
scp docker-compose.yml root#remote:
ssh root#remote env TAG=$TAG docker-compose up -d
A further option is to use a dedicated cluster manager like Kubernetes, and talk to its API; then the cluster will pull the updated containers itself, and you don't have to ssh anything. At the scale you're discussing this is probably much heavier weight than you need.
All the solutions I know to deploy a Docker to EC2 involves running it inside a wrapping Ubuntu.
I want to deploy my Ubuntu docker to EC2 so it will be a standalone EC2 image running by itself.
Is that feasible?
You can not launch EC2 with docker image, EC2 uses AWS AMI to launch instance.
One to way to launch your docker image directly with fargate, which does not manage any instance but will run your image as a standalone container.
AWS Fargate is a compute engine for Amazon ECS that allows you to run
containers without having to manage servers or clusters. With AWS
Fargate, you no longer have to provision, configure, and scale
clusters of virtual machines to run containers. This removes the need
to choose server types, decide when to scale your clusters, or
optimize cluster packing. AWS Fargate removes the need for you to
interact with or think about servers or clusters.
We are developing spring boot application which is currently deploying in AWS manually. For that, first we build docker image through Dockerfile and then connect to AWS EC2 instance from laptop & then pull the image and then we use docker run to start it. But we want to automate the process using gitlab CI/CD.
We created .gitlab-ci.yml, build stage builds spring-boot application and generates jar file. Package stage then build docker images using Dockerfile from source code and then push the image to registry.
Now i don't know how to finish deploy stage. Most of the tutorials explains only about deploying into Google cloud provider. I use below steps to deploy the docker image...
ssh -i "spring-boot.pem" ubuntu#ec2-IP_address.compute-2.amazonaws.com
sudo docker pull username/spring-boot:v1
sudo docker run -d -p 80:8080 username/spring-boot:v1
Can anybody help me to add above steps into deploy stage. Do I need to add pem file into source to connect to ec2 instance.
Or is there any easy way to deploy docker in ec2 from gitlab ci/cd.
First thing, If there is ssh then it's mean you must provide the key or password by default unless you allow access to everyone.
Do I need to add pem file into source to connect to ec2 instance?
Yes, you should provide the key for ssh.
Or is there any easy way to deploy Docker in ec2 from gitlab ci/cd?
Yes, there is the easiest way to do that but for that, you need to use ECS, the specially designed for Docker container and you can manage your deployment through API instead of doing ssh to the ec2 server.
ECS is designed for running Docker container, Some of the big Advantage of ECS over ec2 is you do not need to worry about container management, scalability and availability, ECS will take care of it. provide ECR which is like docker registry but it's private and with in-network.
deploy-docker-containers
I do have a (Python Flask) application that I want to deploy using GitLab CI and Docker to my VPS.
On my server I want to have a production version and a staging version of my application. Both of them require a MongoDB connection.
My plan is to automatically build the application on GitLab and push it to GitLab's Docker Registry. If I want to deploy the application to staging or production I do a docker pull, docker rm and docker run.
The plan is to store the config (e. g. secret_key) in .production.env (and .staging.env) and pass it to application using docker run --env-file ./env.list
I already have MongoDB installed on my server and both environments of the applications shall use the same MongoDB instance, but a different database name (configured in .env).
Is that the best practice for deploying my application? Do you have any recommendations? Thanks!
Here's my configuration that's worked reasonably well in different organizations and project sizes:
To build:
The applications are located in a git repository (GitLab in your case). Each application brings its own Dockerfile.
I use Jenkins for building, you can, of course, use any other CD tooling. Jenkins pulls the application's repository, builds the docker image and publishes it into a private Docker repository (Nexus, in my case).
To deploy:
I have one central, application-independent repository that has a docker-compose file (or possibly multiple files that extend one central file for different environments). This file contains all service definitions and references the docker images in my Nexus repo.
If I am using secrets, I store them in a HashiCorp Vault instance. Jenkins pulls them, and writes them into an .env file. The docker-compose file can reference the individual environment variables.
Jenkins pulls the docker-compose repo and, in my case via scp, uploads the docker-compose file(s) and the .env file to my server(s).
It then triggers a docker-compose up (for smaller applications) or re-deploys a docker stack into a swarm (for larger applications).
Jenkins removes everything from the target server(s).
If you like it, you can do step 3. via Docker Machine. I feel, however, its benefits don't warrant use in my cases.
One thing I can recommend, as I've done it in production several times is to deploy Docker Swarm with TLS Encrypted endpoints. This link talks about how to secure the swarm via certificate. It's a bit of work, but what it will allow you to do is define services for your applications.
The services, once online can have multiple replicas and whenever you update a service (IE deploy a new image) the swarm will take care of making sure one is online at all times.
docker service update <service name> --image <new image name>
Some VPS servers actually have Kubernetes as a service (Like Digital Ocean) If they do, it's more preferable. Gitlab actually has an autodevops feature and can remotely manage your Kubernetes cluster, but you could also manually deploy with kubectl.