I have very simple flask project with just one end point that I was able to deploy into AWS using Elastic Beanstalk
The only exposed end point goes to S3 retrieves a csv file and publish data in raw format, this configuration is working fine, so I know the roles and permissions at elastic beanstalk work correctly to reach the S3 bucket.
from flask import Flask
from flask_restful import Resource, Api
import io
import boto3
import pandas as pd
application = Flask(__name__)
api = Api(application)
s3 = boto3.resource('s3')
bucket = 'my_bucket'
key = 'my_file.csv'
class Home(Resource):
def get(self):
try:
s3 = boto3.resource('s3')
obj = s3.Object(bucket, key).get()['Body']
data = pd.read_csv(io.BytesIO(obj.read()))
print("S3 Object loaded.")
except:
print("S3 Object could not be opened.")
print(data)
csv = data.to_csv(index=False)
return csv
#End points definition and application raise up
api.add_resource(Home, '/')
if __name__ == '__main__':
application.run(host='0.0.0.0')
Now I'm trying to move that to a container so I created a Dockerfile to encapsulate the minimal app:
# syntax=docker/dockerfile:1
FROM python:3.8-slim-buster
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY . .
CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]
Since I don't have additional volumes or anything extra my Dockerrun.aws.json is barely empty
{
"AWSEBDockerrunVersion": "1"
}
I'm missing something to procure the access to the S3 bucket from inside the container?
While debugging I figured out that I was not exposing the port at the dockerfile and therefore it was not able to deploy the container correctly. I also added python as an entry point and the script name as the cmd.
Finally after some investigation also realized that the container inherits all role permissions that the host has so there is not need to do any additional task
My docker file end up looking like this:
# syntax=docker/dockerfile:1
FROM python:3.8-slim-buster
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
EXPOSE 5000
COPY . .
ENTRYPOINT [ "python" ]
CMD [ "app.py" ]
Related
I have a NextJS application that accesses a database from the API.
When in development, I have a .env file that contains the host, port, username, password, and database. After running npm run dev, the API functionality works as expected. Even if I run npm run build and npm run start on my local machine, it works correctly.
The problem comes after I push to Github and have Github build the app and deploy it as a docker container. For some reason, the docker build does not accept my environment variables loaded through an attached .env file.
To further elaborate, the environment variables are either on the dev machine or attached to the docker container on the production server. They are mounted to the same place (the root directory of the project: /my-site/.env) but the production API does not work.
The environment variables are included in /lib/db.tsx:
import mysql from "mysql2/promise";
const executeQuery = async (query: string) => {
try {
const db = await mysql.createConnection({
host: process.env.MYSQL_HOST,
database: process.env.MYSQL_DATABASE,
user: process.env.MYSQL_USER,
password: process.env.MYSQL_PASSWORD,
});
const [results] = await db.execute(query, []);
db.end();
return results;
} catch (error) {
return { error };
}
};
export default executeQuery;
This file is included in the API endpoints as:
import executeQuery from "../../../../lib/db";
Again, since it works on the development computer, I think there is an issue is with the building of the docker container.
Here is my included Dockerfile:
FROM node:lts as dependencies
WORKDIR /my-site
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
FROM node:lts as builder
WORKDIR /my-site
COPY . .
COPY --from=dependencies /my-site/node_modules ./node_modules
RUN yarn build
FROM node:lts as runner
WORKDIR /my-site
ENV NODE_ENV production
# If you are using a custom next.config.js file, uncomment this line.
# COPY --from=builder /my-site/next.config.js ./
COPY --from=builder /my-site/public ./public
COPY --from=builder /my-site/.next ./.next
COPY --from=builder /my-site/node_modules ./node_modules
COPY --from=builder /my-site/package.json ./package.json
EXPOSE 3000
CMD ["yarn", "start"]
Any and all assistance is greatly appreciated!
Edit: Other things I have attempted:
Add them as environment variables to the docker container in the docker compose file (under environment) and verified that they are accessible inside of the container using echo $MYSQL_USER.
Mounting the .env file inside of the .next folder
Mounting the .env file inside of the .next/server folder
I ended up solving my own issue after hours of trying to figure it out.
My solution was to create a .env.production file and commit it to git.
I also adjusted my Dockerfile to include: COPY --from=builder /my-site/.env.production ./
I am not a fan of that solution, as it involves pushing secrets to a repo, but it works.
My docker container's file structure is the following File structure
Below is my Dockerfile
# syntax=docker/dockerfile:1
FROM python:3.9
WORKDIR /app
RUN pip3 install poetry
ADD ../services/ingest/ ./services/ingest/
ADD ../protos/datapipeline/ ./protos/datapipeline/
WORKDIR /app/services/ingest
RUN poetry install
CMD [ "poetry", "run", "python", "ingest/processor.py"]
I copied the services/ingest and protos/datapipeline folders in such a way that they should be in the same level as in my local machine.
The project also uses poetry with the following line for accessing the python proto files within the pyproject.toml located in the services/ingest folder
datapipeline = {path = "../../protos/datapipeline"}
the container should then go to /app/services/ingest and launch services/ingest/ingest/processor.py using the poetry enviroment which has datapipeline.datapipeline as a module. But running it this way the program cannot locate it even though it works outside the container and produces the following error.
Exception has occurred: ModuleNotFoundError
No module named 'datapipeline.datapipeline'
File "/app/services/ingest/ingest/processor.py", line 9, in <module>
import datapipeline.ingest
Any help or resources would be greatly appreciated on getting this working.
File is imported the following way in processor.py
import datapipeline.ingest
I'm running a simple flask app inside a docker container. At the end of the Dockerfile I want to add a new user that doesn't have permissions to read files owned by the docker root user. But when I do that the new user sandbox can still access /etc/passwd and all files copied before it's creation like app.py.
Dockerfile:
FROM python:3
WORKDIR /var/www
COPY app.py .
RUN pip3 install flask
RUN useradd sandbox
USER sandbox
CMD [ "python3", "app.py" ]
App.py
from flask import Flask
app = Flask(__name__)
#app.route('/')
def root_handler():
return "Hello: {} {}".format(getpass.getuser(), '<3')
#app.route('/read')
def read_source():
with open("/etc/passwd", "r") as f:
return f.read()
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8080)
Do I have to manually set all permissions with RUN chmod ... or is there a more elegant way?
So I am learning out docker for the first time and was wondering if I am doing this in the correct format for my flask app, as a lot of documentation online for the WOKRDIR command is changing dir into "/app" however my main file to run the app is run.py which would be the same directory as the actual docker file. However, WORKDIR doesn't let me do "WORKDIR ." to use the current DIR.
Can someone clarify if I have my docker file set up correctly?
(I also plan to host this on Heroku if that matters)
File structure
Docker file
# start by pulling the python image
FROM python:3.8-alpine
# copy the requirements file into the image
COPY ./requirements.txt /requirements.txt
# Don't need to switch working directory
# WORKDIR
# install the dependencies and packages in the requirements file
RUN pip install -r requirements.txt
# copy every content from the local file to the image
COPY . /app
# configure the container to run in an executed manner
ENTRYPOINT [ "python" ]
CMD ["run.py" ]
I'm struggling with this issue:
I've a fastAPI app with a couple of routes that are called by other services and I've to put up a testing environment for the customers to try out the features.
I've containerized the whole api project and it works very well.
Then I've published on Heroku via container stack the whole thing and now it doesn't save the files in the /response directory.
My Dockerfile is:
FROM python:3.9-slim
COPY ./src /app/src
RUN mkdir /app/responses
COPY ./requirements.txt /app
COPY ./start.sh /app
COPY ./templates /app/templates
COPY ./static /app/static
WORKDIR /app
RUN pip install -r requirements.txt
RUN chmod a+x ./start.sh
#EXPOSE 8000
CMD ["./start.sh"]
My fastAPI app has mounted the static file directory (which is not working either) and the responses directory as follows:
from fastapi import FastAPI, File, UploadFile, Request, Form
from fastapi.responses import HTMLResponse, FileResponse
from fastapi.staticfiles import StaticFiles
from fastapi.templating import Jinja2Templates
app.mount("/static", StaticFiles(directory="static"), name="static")
app.mount("/responses", StaticFiles(directory="responses"), name="responses")
templates = Jinja2Templates(directory="templates")
And then I try to write the files with this code:
filename = fiscalCode + "/" + fiscalCode +"_" +timestamp+".html"
filepath = os.path.join(RESPONSE_FOLDER, filename)
print(filepath)
if not os.path.exists(filepath):
os.makedirs(os.path.dirname(filepath), exist_ok=True)
with open(filepath, "w+") as f:
f.write(file.file.read().decode("utf-8"))
I know that Heroku filesystem is ephemeral but this is not an issue: the file must be available only for a couple of minutes for testing purposes.
I'm stuck bcs on other applications deployed via Heroku and Procfile (so without the Docker shenanigans) I had no issues at all in writing and retrieving files.
Thanks for any idea.
I found that, even in Heroku remote shell, I can't see the files created, I think because of the ephemeral file system.
Nevertheless all the functions are working fine, and transitional files are stored somewhere in the hyperuranium.
So if you don't find your actual files, they are written on (I suppose) a caching memory or something like that.