Definitely missing something, and could use some quick assistance!
Simply, how do you deploy a Docker image to an Airflow DAG for running jobs? Does anyone have a simple example of deploying a Google container and running it via Airflow/Composer?
You can use the Docker Operator, included in the core Airflow repository.
If pulling an image from a private registry, you'll need to set a connection config with the relevant credentials and pass it to the docker_conn_id param.
Related
I have .env file for my docker-compose, and was able to run using "docker-compose up"
Now I pushed to cloud registry, and want to Cloud Run
How can I supply the various environemnt variables?
I did create secrets in secret manager, but how can I integrate both, so that my container starts reading all those needed secrets?
Note: My docker-compose is an app with database, but I can split them as 2 containers, if needed, but they still need secrets
Edit: Added secret references.
EDIT:
I am unable to run my container
If env file X=x , and docker-compose environemnt app.prop=${X}
then should I create secret X or x?
Is Cloud run using Dockerfile or docker-compose? I image pushed is built from docker-compose only. Sorry I am getting confused (not assuming trivial things as it is not working)
It is not possible to use docker-compose on Cloud Run, as it is designed for individual stateless containers. My suggestion is to create an image from your application service, upload the image to Google Container Registry in order to use it for your Cloud Run service, and connect it to Cloud SQL following the attached documentation. You can store database credentials with Secret Manager and pass it to your Cloud Run service as environment variables (check this documentation).
as we can see in https://hub.docker.com/r/hyperledger/composer-rest-server/tags all the docker containers in Hyperledger composer rest server images are removed. how are we supposed to deploy the rest server,as the official documentation says doing so this way.
Composer has been sunset (it's been deprecated for a while) and it looks like all the docker hub images have been removed now.
Any documentation referring to using the docker images (rest server, cli, playground) won't work.
The source code for composer https://github.com/hyperledger/composer contains the Dockerfile's so you can always use them to build your own local docker image.
i'm new to amazon web services, and im involved in project where we dicided to user serverless architecture which consist of lambda(nodejs), dynamo DB, cognito etc, as a DevOps engineer, im trying to figure out how to to do CD/CD for the project.
I've read multiple articles, they mentioning fargate and other services which i understand, but when it comes to docker and ECS im bit confused, *
i dont know if we push the image to ECS and write the dockerfile so that my lambda functions can run, or we just push the image to ECS so that the cluster of lambda functions can run?
please anyone with a clear explanantion please assist.
Thank you
i dont know if we push the image to ECS and write the dockerfile so
that my lambda functions can run, or we just push the image to ECS so
that the cluster of lambda functions can run?
You push Docker images to AWS Elastic Container Registry (ECR). ECS can then pull those images to deploy docker containers in either EC2 or Fargate.
ECS, ECR and Docker are totally unrelated to AWS Lambda. You don't run docker containers in AWS Lambda.
Implemented similar using Jenkins.. We have create docker image using jenkins as part of CI CD pipeline.. then we store those docker images in ECR. In ECS we create task definition and container instance, then container instance will refer to the ECR images which we created as part of CICD.
Dokcerfile was part of docker image.
I am trying to run a Docker image through a Google Cloud proxy and despite my best efforts Google Cloud continues giving me this error:
Can't create logging client: google: could not find default
credentials. See
https://developers.google.com/accounts/docs/application-default-credentials
for more information.
Whenever I try to run my Docker image using this command:
sudo docker run dc701c583cdb
I have tried updating my GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of my key file.
I have successfully logged in to Google Cloud using the gcloud auth application-default login command.
I've defined and associated my project in Google Cloud.
I am attempting this in order to run an open source project. I'm quite sure I created the Docker image correctly. I have a feeling the issue is coming from the fact that I am not correctly connecting the existing project to my Google Cloud.
Any advice would be greatly appreciated. I am using Docker 18.06.1-ce and Google Cloud-SDK 219.0.1. Running on a virtual linux machine with Ubuntu 18.04.
When running the google/cloud-sdk container from Docker Hub in a newly-created Ubuntu 18.04 instance, the container's gcloud automatically inherits the instance's user configuration. Give it a try: run that container and then run gcloud info inside of it.
As such, I believe you might be doing something wrong. I recommend you take a look at the aforementioned container to see how that can be made to work.
I started playing with Docker Cloud and am trying to deploy a tomav/docker-mailserver container to an EC2 instance. The EC2 and dockercloud-agent seems to work fine for container deployment.
The docker-compose.yml uses hostname and domainname parameters which are required to properly configure it, but I can't find their equivalent in Docker Cloud's interface.
One of them is using the container auto-generated name, which I need to override.
Anybody knows if I am missing something ? Or is it not possible yet ?
Thank you for your help !
What you want is a stack file, roughly equivalent to docker-compose