Trouble Deploying Docker image Flask AWS Beanstalker - machine-learning

I am trying to deploy a machine learning model using flask, then I used docker to build an image, and finally Travis for CI\CD and deploy it on AWS Beanstalk.
the Flask application runs well on localhost and docker image run command.
My entire project is here: end2endML
I am having the following problem > 502 Bad Gateway
Until now :
It is running on 64bit Amazon Linux 2/3.4.15 -I don't find any other option-
I have read about the port 80 or 5000 I have tried both and they failed
I read the Log and received this message
I have a Dockerfile and Docker-compose.yml for deployment. and they look like this:
FROM python:3.7-slim-bullseye
WORKDIR /app
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
# Install dependencies:
COPY . /app
RUN pip install -r /app/requirements/requirements.txt
EXPOSE 5000
# Run the application:
CMD ["python", "/app/classification_model/application.py"]
---> My docker Compose
# Docker compose for Amazon
version: '3'
services:
aws_deployment:
# This function forces Dockerfile
build:
# Files (look at the current dir)
context: .
# Look for this file
dockerfile: Dockerfile
ports:
- "80:5000"
--> My Travis file
#autom install a copy of docker
services:
- docker
# Do before : -f force -t anynameyouwantforyourimage
before_install:
- docker build -t raudez77/end2end-machine-learning -f Dockerfile .
# Run test
# Script contents all script to run
script:
- docker run raudez77/end2end-machine-learning python test.py -- --coverage
#Deploy into Amazon ElasticBeanstalk
deploy:
# Use this Saas for deployment
provider: elasticbeanstalk
region: "us-west-2"
# Application name
app: "docker_titanic"
# Enviroment
env: "Dockertitanic-env"
# Go Services- S3 - look for yout us-west-2 elastickbeanstalk
bucket_name: "elasticbeanstalk-us-west-2-082124557004"
bucket_path : "docker_titanic"
# Create Acces key in Travis
access_key_id: $AWS_ACCESS_KEY
secret_access_key: "$AWS_SECRET_KEY"
on:
branch: master
skip_cleanup: true ```

Related

Gitlab CI npm cannot resolve module

Totally new to Gitlab and CI in general, so apologies for the lack of understanding. I have a repo, which is NuxtJS based, with a Dockerfile. The end goal of the pipeline is to build and push this repo to my docker account. The Dockerfile is relatively straight forward, containing an npm install and npm run build. I'm using a custom docker image as my runner, based on docker:20.10.17-dind-alpine3.16 with ansible, terraform and kubectl installed.
When building the project's docker image on my local machine, I receive no issues, however in gitlab, when running the npm run build command, I get the following error:
Module not found: Error: Can't resolve '../node_modules/vue-confirm-dialog' in '/usr/src/nuxt-app/plugins'
Here is my yml file:
stages:
- docker
docker:
stage: docker
image: <my-runner-image>
services:
- "docker:dind"
before_script:
- docker login -u $DOCKER_REGISTRY_USER -p $DOCKER_REGISTRY_PASSWORD
script:
- docker build -t <my-repo> .
- docker push <my-repo>
Any suggestions are greatly appreciated
--EDIT--
As requested, here is the project's Dockerfile:
FROM node:lts-alpine3.15
# create destination directory
RUN mkdir -p /usr/src/nuxt-app
WORKDIR /usr/src/nuxt-app
# update and install dependency
RUN apk update && apk upgrade
RUN apk add git
# copy the app, note .dockerignore
COPY . /usr/src/nuxt-app/
RUN npm install
RUN npm run build
EXPOSE 3000
ENV NUXT_HOST=0.0.0.0
ENV NUXT_PORT=3000
CMD [ "npm", "start" ]

Env vars lost when building docker image from Gitlab CI

I'm trying to build my React / NodeJS project using Docker and Gitlab CI.
When I build manually my images, I use .env file containing env vars, and everything is fine.
docker build --no-cache -f client/docker/local/Dockerfile . -t espace_client_client:local
docker build --no-cache -f server/docker/local/Dockerfile . -t espace_client_api:local
But when deploying with Gitlab, I can build successfully the image, but when I run it, env vars are empty in the client.
Here is my gitlab CI:
image: node:10.15
variables:
REGISTRY_PACKAGE_CLIENT_NAME: registry.gitlab.com/company/espace_client/client
REGISTRY_PACKAGE_API_NAME: registry.gitlab.com/company/espace_client/api
REGISTRY_URL: https://registry.gitlab.com
DOCKER_DRIVER: overlay
# Client Side
REACT_APP_API_URL: https://api.espace-client.company.fr
REACT_APP_DB_NAME: company
REACT_APP_INFLUX: https://influx-prod.company.fr
REACT_APP_INFLUX_LOGIN: admin
REACT_APP_HOUR_GMT: 2
stages:
- publish
docker-push-client:
stage: publish
before_script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $REGISTRY_URL
image: docker:stable
services:
- docker:dind
script:
- docker build --no-cache -f client/docker/prod/Dockerfile . -t $REGISTRY_PACKAGE_CLIENT_NAME:latest
- docker push $REGISTRY_PACKAGE_CLIENT_NAME:latest
Here is the Dockerfile for the client
FROM node:10.15-alpine
WORKDIR /app
COPY package*.json ./
ENV NODE_ENV production
RUN npm -g install serve && npm install
COPY . .
RUN npm run build
EXPOSE 3000
CMD [ "serve", "build", "-l", "3000" ]
Why is there such a difference between the 2 process ?
According to your answer in comments, GitLab CI/CD environment variables doesn't solve your issue. Gitlab CI environment is actual only in context of GitLab Runner that builds and|or deploys your app.
So, if you are going to propagate Env vars to the app, there are several ways to deliver variables from .gitlab-cy.ymlto your app:
ENV instruction Dockerfile
E.g.
FROM node:10.15-alpine
WORKDIR /app
COPY package*.json ./
ENV NODE_ENV production
ENV REACT_APP_API_URL: https://api.espace-client.company.fr
ENV REACT_APP_DB_NAME: company
ENV REACT_APP_INFLUX: https://influx-prod.company.fr
ENV REACT_APP_INFLUX_LOGIN: admin
ENV REACT_APP_HOUR_GMT: 2
RUN npm -g install serve && npm install
COPY . .
RUN npm run build
EXPOSE 3000
CMD [ "serve", "build", "-l", "3000" ]
docker-compose environment directive
web:
environment:
- NODE_ENV=production
- REACT_APP_API_URL=https://api.espace-client.company.fr
- REACT_APP_DB_NAME=company
- REACT_APP_INFLUX=https://influx-prod.company.fr
- REACT_APP_INFLUX_LOGIN=admin
- REACT_APP_HOUR_GMT=2
Docker run -e
(Not your case, just for information)
docker -e REACT_APP_DB_NAME="company"
P.S. Try Gitlab CI variables
There is convenient way to store variables outside of your code: Custom environment variables
You can set them up easily from the UI. That can be very powerful as it can be used for scripting without the need to specify the value itself.
(source: gitlab.com)

Docker Compose task in VSTS : Couldn't connect to Docker daemon at http+docker://localhost - is it running?

Installed docker 18.03 on vsts agent box(self-hosted VSTS agent)
The user under which the agent is running has been added to the docker group.
When I try to build using Docker Compose task in VSTS, the build fails with error:
Couldn't connect to Docker daemon at http+docker://localhost - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
Couldn't connect to Docker daemon at http+docker://localhost - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
/usr/local/bin/docker-compose failed with return code: 1
I have been stuck in this for few hours, any help will be awesome.
One more note: docker compose works perfectly fine from the agent box, but when the build is triggered by VSTS task I get this error.
docker-compose file:
version: '3'
services:
some-api:
build:
context: .
dockerfile: .docker/dockerfile1
image: some.azurecr.io/some-api:latest
container_name: 'some-api'
ports:
- '8080:80'
some-website:
build:
context: .
dockerfile: .docker/dockerfile2
image: some.azurecr.io/some-website:latest
container_name: 'some-website'
ports:
- '3434:3434'
dockerfile -api
FROM microsoft/dotnet AS build
# Docker image container .NET Core SDK
COPY .api/ ./some-api
WORKDIR /some-api
RUN dotnet restore; dotnet publish -o out
# final image
FROM microsoft/aspnetcore
# .NET Core runtime-only image
COPY --from=build /some-api/out /some-api
WORKDIR /some-api
EXPOSE 80
ENTRYPOINT [ "dotnet", "some.dll" ]
dockerfile-website
#----------------------
### STAGE 1: BUILD ###
#---------------------
# Building node from LTS version
FROM node:8.11.1 as builder
# Installing npm to remove warnings and optimize the container build process
# One of many warnings: npm WARN notice [SECURITY] deep-extend has 1 low vulnerability.
#Go here for more details: https://nodesecurity.io/advisories?search=deep-extend&version=0.5.0 -
#Run `npm i npm#latest -g` to upgrade your npm version, and then `npm audit` to get more info.
RUN npm install npm#latest -g
# Copying all necessary files required for npm install
COPY package.json ./
# Install npm dependencies in a different folder to optimize container build process
RUN npm install
# Create application directory and copy node modules to it
RUN mkdir /some-website
RUN cp -R ./node_modules ./some-website
# Setting application directory as work directory
WORKDIR /some-website
# Copying application code to container application directory
COPY . .
# Building the angular app
RUN npm run build.prod
#--------------------------------------------------
### STAGE 2: Setup nginx and Deploy application ###
#--------------------------------------------------
FROM nginx:latest
## Copy defualt ngninx configuration file
COPY default.conf /etc/nginx/conf.d
## Remove default nginx website
RUN rm -rf /usr/share/nginx/hmtl/*
# Copy dist folder from the builder to nginx public folder(STAGE 1)
COPY --from=builder /some-website/dist/prod /usr/share/nginx/html
CMD ["nginx","-g","daemon off;"]
Thanks
The issue was user permissions. So after adding a user to the docker group,
sudo usermod -aG docker $USER
logging out and logging in didn't work. I had to reboot my ubuntu server in order for permissions to take effect.

Setting build args in docker-compose.yml using env_file

I'm trying to use Docker and Docker Compose to create a containerized app. I have a PubNub account, which allows me to use different API keys for different environments (dev, test, prod). To help me build images for this, I am trying to use build args set with an env_file.
It's not working.
WARNING: The PUB_KEY variable is not set. Defaulting to a blank string.
WARNING: The SUB_KEY variable is not set. Defaulting to a blank string.
Questions:
What mistake am I making in setting the build args?
How do I fix it?
Is this a good way to set ENV variables for the containers scan and flask?
At the very bottom is an IntelliJ IDE screenshot, or the text code is just below.
Here is the docker-compose.yml content:
version: '3.6'
services:
scan:
env_file:
- sample.env
build:
context: .
dockerfile: Dockerfile
args:
pub_key: $PUB_KEY
sub_key: $SUB_KEY
target: scan
image: bt-beacon/scan:v1
flask:
env_file:
- sample.env
build:
context: .
dockerfile: Dockerfile
args:
pub_key: $PUB_KEY
sub_key: $SUB_KEY
target: flask
image: bt-beacon/flask:v1
ports:
- "5000:5000"
And the Dockerfile:
# --- BASE NODE ---
FROM python:3.6-jessie as base
ARG pub_key
ARG sub_key
RUN test -n "$pub_key"
RUN test -n "$sub_key"
# --- SCAN NODE ---
FROM base as scan
ENV PUB_KEY=$pub_key
ENV SUB_KEY=$sub_key
COPY app/requirements.scan.txt /
RUN apt-get update
RUN apt-get -y install bluetooth bluez bluez-hcidump python-bluez python-numpy python3-dev libbluetooth-dev libcap2-bin
RUN pip install -r /requirements.scan.txt
RUN setcap 'cap_net_raw,cap_net_admin+eip' $(readlink -f $(which python))
COPY app/src /app
WORKDIR /app
CMD ["./scan.py", "$pub_key", "$sub_key"]
# -- FLASK APP ---
FROM base as flask
ENV SUB_KEY=$sub_key
COPY app/requirements.flask.txt /
COPY app/src /app
RUN pip install -r /requirements.flask.txt
WORKDIR /app
EXPOSE 5000
CMD ["flask", "run"]
Finally, sample.env:
# PubNub app keys here
PUB_KEY=xyz1
SUB_KEY=xyz2
env_file can only set environment variables inside a service container. Variables from env_file cannot be injected into docker-compose.yml itself.
You have such options (described there in detail):
inject these variables into the shell, from which you run docker-compose up
create .env file containing these variables (syntax identical to your sample.env)
Personally I would separate image building process and container launching process (take away image building responsibility from docker-compose to external script, then building process can be configured easily).

Docker images using circleci

I am working on integrating CI/CD pipeline using Docker. How can i use dockercompose file to build and create container?
I have tried it in putting Dockerfile, and docker-compose.yml, but none of them works.
Below is docker-compose file :
FROM ruby:2.2
EXPOSE 8000
RUN mkdir -p /usr/src/app
COPY . /usr/src/app
WORKDIR /usr/src/app
RUN bundle install
CMD ["ruby","app.rb"]
RUN docker build -t itsruby:itsruby .
RUN docker run -d itsruby:itsruby
Below is docker-compose.yml
version: 2
jobs:
build:
docker:
- image: circleci/ruby:2.2
steps:
- checkout
- run: CMD ["ruby","app.rb"]
- run: |
docker build -t itsruby:itsruby .
docker run -d itsruby:itsruby
test:
docker:
- image: circleci/ruby:2.2
steps:
- checkout
- run: CMD ["ruby","app.rb"]
The build is getting failed in circle/ci.

Resources