How to pass environment variable to Dockerfile through Pulumi? - docker

Question

I have made a simple flask app to practice Pulumi.
It gets env variable set through Dockerfile and I intend to host it on AWS Fargate, and RDS Postgres as database.
Here is the Flask app:
import os
from flask import Flask, request
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = "postgresql://{}".format(
os.environ.get("DATABASE_URL")
)
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
db = SQLAlchemy(app)
migrate = Migrate(app, db)
class CarsModel(db.Model):
__tablename__ = "cars"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String())
model = db.Column(db.String())
doors = db.Column(db.Integer())
def __init__(self, name, model, doors):
self.name = name
self.model = model
self.doors = doors
def __repr__(self):
return f"<Car {self.name}>"
#app.route("/")
def hello():
return {"hello": "world"}
#app.route("/cars", methods=["POST", "GET"])
def handle_cars():
if request.method == "POST":
if request.is_json:
data = request.get_json()
new_car = CarsModel(
name=data["name"], model=data["model"], doors=data["doors"]
)
db.session.add(new_car)
db.session.commit()
return {"message": f"car {new_car.name} has been created successfully."}
else:
return {"error": "The request payload is not in JSON format"}
elif request.method == "GET":
cars = CarsModel.query.all()
results = [
{"name": car.name, "model": car.model, "doors": car.doors} for car in cars
]
return {"count": len(results), "cars": results, "message": "success"}
#app.route("/cars/<car_id>", methods=["GET", "PUT", "DELETE"])
def handle_car(car_id):
car = CarsModel.query.get_or_404(car_id)
if request.method == "GET":
response = {"name": car.name, "model": car.model, "doors": car.doors}
return {"message": "success", "car": response}
elif request.method == "PUT":
data = request.get_json()
car.name = data["name"]
car.model = data["model"]
car.doors = data["doors"]
db.session.add(car)
db.session.commit()
return {"message": f"car {car.name} successfully updated"}
elif request.method == "DELETE":
db.session.delete(car)
db.session.commit()
return {"message": f"Car {car.name} successfully deleted."}
if __name__ == "__main__":
app.run(host="0.0.0.0", port=8000)
Here is the Dockerfile:
# Use an official Python runtime as a parent image
FROM python:3.8
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
COPY . /app
# Install any needed packages specified in requirements.txt
RUN pip install --trusted-host pypi.python.org -r requirements.txt
ENV FLASK_APP main.py
ENV DATABASE_URL localhost
RUN flask db init
RUN flask db migrate
RUN flask db upgrade
# Make port 80 available to the world outside this container
EXPOSE 8000
# Run app.py when the container launches
CMD ["python", "main.py"]
Here is the index.ts file for Pulumi:
import * as awsx from "#pulumi/awsx";
import * as aws from "#pulumi/aws";
import * as pulumi from "#pulumi/pulumi";
const vpc = new awsx.ec2.Vpc("custom");
// Step 1: Create an ECS Fargate cluster.
const cluster = new awsx.ecs.Cluster("first_cluster", { vpc });
const securityGroupIds = cluster.securityGroups.map(g => g.id);
const dbSubnets = new aws.rds.SubnetGroup("dbsubnets", {
subnetIds: vpc.publicSubnetIds,
});
const db = new aws.rds.Instance("postgresdb", {
engine: "postgres",
instanceClass: "db.t2.micro",
allocatedStorage: 20,
dbSubnetGroupName: dbSubnets.id,
vpcSecurityGroupIds: securityGroupIds,
name: "dummy",
username: "dummy",
password: "123456789",
publiclyAccessible: true,
skipFinalSnapshot: true,
});
const hosts = pulumi.all([db.endpoint.apply(e => e)]);
const environment = hosts.apply(([postgresHost]) => [
{ name: "DATABASE_URL", value: postgresHost },
]);
// Step 2: Define the Networking for our service.
const alb = new awsx.elasticloadbalancingv2.ApplicationLoadBalancer(
"net-lb", { external: true, securityGroups: cluster.securityGroups, vpc });
const atg = alb.createTargetGroup(
"app-tg", { port: 8000, deregistrationDelay: 0 });
const web = atg.createListener("web", { port: 80, external: true });
// Step 3: Build and publish a Docker image to a private ECR registry.
const img = awsx.ecs.Image.fromPath("app-img", "./app");
// Step 4: Create a Fargate service task that can scale out.
const appService = new awsx.ecs.FargateService("app-svc", {
cluster,
taskDefinitionArgs: {
container: {
image: img,
cpu: 102 /*10% of 1024*/,
memory: 50 /*MB*/,
portMappings: [web],
environment: environment,
},
},
desiredCount: 5,
}, { dependsOn: [db] });
// Step 5: Export the Internet address for the service.
export const url = web.endpoint.hostname;
Now, when I do pulumi up, I get this:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Cannot assign requested address
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
(Background on this error at: http://sqlalche.me/e/e3q8)
at /Users/myuser/projects/practice/pulumi/simple_flask_app/node_modules/#pulumi/docker.ts:546:15
at Generator.next (<anonymous>)
at fulfilled (/Users/myuser/projects/practice/pulumi/simple_flask_app/node_modules/#pulumi/docker/docker.js:18:58)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
error: The command '/bin/sh -c flask db migrate' returned a non-zero code: 1
Now, I know that its because its trying to connect to localhost as that is the default, but how to pass in the host name of db resource?
Thanks
UPDATE 1: Tried removing ENV DATABASE_URL localhost
After removing ENV DATABASE_URL localhost:
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 652, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 490, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "None" to address: Name or service not known
(Background on this error at: http://sqlalche.me/e/e3q8)
at /Users/myuser/projects/practice/pulumi/simple_flask_app/node_modules/#pulumi/docker.ts:546:15
at Generator.next (<anonymous>)
at fulfilled (/Users/myuser/projects/practice/pulumi/simple_flask_app/node_modules/#pulumi/docker/docker.js:18:58)

Answer 1

I’d consider this bad practice to run the migrations during the docker build. What happens if the build fails afterwards? How can you control which changes are applied to which environment? I think there are better solutions to this problem.
Those migrations could also be applied when the container boots up in fargate by e.g. putting those commands into an entrypoint script or executing the migration in the process startup (basically in your main.py) like described here: https://flask-migrate.readthedocs.io/en/latest/#command-reference
Another reason for not doing it during the pulumi up is that this would also require a firewall rule allowing your local machine to access the database (might be already “solved“ with your publiclyAccessible setting though).
If you still want to keep this action in the build, you need a different way of providing the database url to step 3. The env is only used during step 4 (setting up fargate). For step 3 you could leverage build args (https://docs.docker.com/engine/reference/builder/#arg) and pass them via pulumi like so https://www.pulumi.com/docs/reference/pkg/docker/image/#dockerbuild
Keep in mind that this adds some security issues because you open up the database to the public which wouldn’t be necessary otherwise. So I’d definitely go with a different approach as described above.