I am trying to set-up a simple dagster container with the following Dockerfile:
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM --platform=linux/amd64 python:3.8-slim
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
ENV DAGSTER_HOME=/dagster
ENV DAGIT_HOME=0.0.0.0
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /dagster
COPY ./dagster-sample /dagster
# Creates a non-root user with an explicit UID and adds permission to access the /dagster folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
RUN adduser -u 5678 --disabled-password --gecos "" dagsteruser && chown -R dagsteruser /dagster
USER dagsteruser
EXPOSE 3000
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["dagit", "-p", "3000"]
Concerning the dagster contents, they are pristine, I simply created a workspace within my home folder and created a dagster project according to the documentation:
pip install dagster
dagster project scaffold --name my-dagster-project
The image is created via visual studio's extension, which translates to this command :
docker image build --pull --file '/home/user1/workspaces/dagster-sample-wrapper/Dockerfile' --tag 'dagstersamplewrapper:latest' --label 'com.microsoft.created-by=visual-studio-code' '/home/user1/workspaces/dagster-sample-wrapper'
The container is started via this command:
docker run -d -p 3000:3000 -it dagstersamplewrapper
Here are the contents of the running container
2022-12-26 03:32:29 2022-12-26 02:32:29 +0000 - dagster - INFO - Started Dagster code server for module dagster_sample in process 10
2022-12-26 03:32:29
2022-12-26 03:32:29 Telemetry:
2022-12-26 03:32:29
2022-12-26 03:32:29 As an open source project, we collect usage statistics to inform development priorities. For more
2022-12-26 03:32:29 information, read https://docs.dagster.io/install#telemetry.
2022-12-26 03:32:29
2022-12-26 03:32:29 We will not see or store solid definitions, pipeline definitions, modes, resources, context, or
2022-12-26 03:32:29 any data that is processed within solids and pipelines.
2022-12-26 03:32:29
2022-12-26 03:32:29 To opt-out, add the following to $DAGSTER_HOME/dagster.yaml, creating that file if necessary:
2022-12-26 03:32:29
2022-12-26 03:32:29 telemetry:
2022-12-26 03:32:29 enabled: false
2022-12-26 03:32:29
2022-12-26 03:32:29
2022-12-26 03:32:29 Welcome to Dagster!
2022-12-26 03:32:29
2022-12-26 03:32:29 If you have any questions or would like to engage with the Dagster team, please join us on Slack
2022-12-26 03:32:29
2022-12-26 03:32:29 2022-12-26 02:32:29 +0000 - dagit - INFO - Serving dagit on http://127.0.0.1:3000 in process 1
Here's a wget -O- "http://127.0.0.1:3000" 2>&1 command's output that I ran inside the container
--2022-12-26 03:53:59-- http://127.0.0.1:3000/
Connecting to 127.0.0.1:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 900 [text/html]
Saving to: ‘STDOUT’
<!doctype html><html lang="en"><head><meta charset="utf-8"/><meta name="viewport" content="width=device-width,initial-scale=1,shrink-to-fit=no"/><meta name="theme-color" content="#000000"/><script type="application/json" id="initialization-data">{"pathPrefix": "", "telemetryEnabled": "True"}</script><script nonce="03fa0b7cb58d417ca2662ec5d0ed2c68">__webpack_nonce__="03fa0b7cb58d417ca2662ec5d0ed2c68"</script><link rel="manifest" href="/manifest.json" crossorigin="use-credentials"/><link rel="icon" type="image/png" href="/favicon.png"/><link rel="icon" type="image/svg+xml" href="/favicon.svg"/><title>Dagit</title><script defer="defer" src="/static/js/main.e20f9d2d.js" nonce="03fa0b7cb58d417ca2662ec5d0ed2c68"></script><link href="/static/css/main.24e9b352.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
0K 100% 284M=0s
2022-12-26 03:53:59 (284 MB/s) - written to stdout [900/900]
I am running wsl2 on a windows 11 host.
I've tried accessing the dagit platform via the following URIs in my host's browser:
localhost:3000 (I know this is wrong)
172.26.221.133:3000 (WSL's IP returned via ifconfig)
172.17.0.2:3000 (The docker container IP adress, found via docker inspect --format '{{ .NetworkSettings.IPAddress }}' <CONTAINER ID>)
192.168.1.7:3000 (Some IP I found suggested here)
192.168.99.100:3000 (Some IP I found suggested here)
Calling http://172.17.0.2:3000 returns : The connection has timed out after 40 seconds, whereas localhost:3000 returns The connection was reset right away.
Could you advise on how to access the container's IP, and how to find it?
EDIT: I have searched for a while, but could not find a use-case where dagit is accessed by anything else except dagit -h 0.0.0.0...
Try changing your last line in Dockerfile to
#CMD ["dagit", "-p", "3000"]
ENTRYPOINT ["dagit", "-h", "0.0.0.0", "-p", "3000"]
Then, you get this log
2022-12-26 04:17:21 +0000 - dagit - INFO - Serving dagit on http://0.0.0.0:3000 in process 1
And check from your HOST ( after 1min )
( !! ) you need more than 512m of memory to run it. Otherwise, your container may die.
http://localhost:3000/dagit_info
Dockerfile
FROM --platform=linux/amd64 python:3.8-slim
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
WORKDIR /dagster
# Install pip requirements
RUN pip install dagster
RUN dagster project from-example \
--name my-dagster-project \
--example quickstart_etl
RUN cd my-dagster-project && pip install -e ".[dev]"
WORKDIR /dagster/my-dagster-project
RUN touch dagster.yaml #just to avoid warnings
ENV DAGSTER_HOME=/dagster/my-dagster-project
#ENV DAGIT_HOME=0.0.0.0
EXPOSE 3000
ENTRYPOINT ["dagit", "-h", "0.0.0.0", "-p", "3000"]
How to find container's IP ?
docker inspect --format '{{ .NetworkSettings.IPAddress }}' CONTAINER_NAME
Normally it is not necessary to access a container by ip directly. If you have several containers and they must interact, it is usual to use a network
https://docs.docker.com/config/containers/container-networking/
You could set any port from host to container
More detail here:
From inside of a Docker container, how do I connect to the localhost of the machine?
Related
I have a fastapi python script that works on codespaces when I use the following command:
uvicorn main:fast_API_app --reload
The following code appears and my api's work fine:
INFO: Will watch for changes in these directories: ['/workspaces/WebAPI']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [3229] using WatchFiles
INFO: Started server process [3241]
INFO: Waiting for application startup.
INFO: Application startup complete.
resulting in
Running it in github codespaces works fine
However, when I turn this into a docker container, running it results in a 502 Bad Gateway
terminal:
docker container run <username>/<container name>:v0.0.1
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Whether i select port 8000 to be public or private in github codespaces makes no difference.
Below is my Dockerfile which is used to build the image.
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.10-slim
EXPOSE 8000
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /app
COPY . /app
# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["uvicorn", "main:fast_API_app", "--host", "0.0.0.0", "--port","8000"]
It results in the following error:
image of the error
It does not show an error code.
What I already tried (but potentially also did wrong):
I tried exposing different ports
tried running gunicorn instead of uvicorn
Searched on Stackoverflow.com for docker codespaces and bad gateway
Toggling the port forward to public / private
Changing the Port Protocol to https
rebuilding the container several times
I started using Redis 6.0 in docker container recently and faced with the following issue, suddenly: I see in logs that Redis container started to try writing to /var/spool/cron directory.
It is the second time I face the problem, first time It happened tonight (in several hours after launch). If I restart container, everything is fine again.
I have found link which stated that it happens so, because Redis container was hacked:
The link
Is it true and how can I solve the issue?
My Dockerfile:
FROM redis:6.0-alpine
WORKDIR /usr/src/app
RUN apk add --no-cache tzdata
ENV TZ=Europe/Moscow
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
COPY redis.conf /usr/local/etc/redis/redis.conf
CMD [ "redis-server", "/usr/local/etc/redis/redis.conf" ]
RUN chmod 0755 /usr/src/app/
RUN chmod 0755 /etc/crontabs/
RUN chmod 0755 /data/
Redis Config key points:
# bind 192.168.1.100 10.0.0.1
# bind 127.0.0.1 ::1
protected-mode no # I thought that if I have user protected-mode works by default
port 6379
dbfilename dump.rdb
dir ./
user someusername allcommands allkeys on >somelongpassword
I run docker conatainer the following way:
docker run -p 6379:6379 -v myvol:/usr/src/app --name redis -d --network mynet redis
I attached print screen of my terminal window.
Never expose unprotected Redis to the Internet!
In your case, although you set ACL for someusername, you didn't disable the default user. And with protected-mode set to no, you expose your Redis instance to everyone.
You need to set protected-mode to yes. Also you can disable the default user with the following config:
user default off
before I start: I have searched this question already and implemented what the "Solutions" were and it did not help (setting host to 0.0.0.0). So, with that out of the way,
Directory structure
|-- osr
| |-- __init__.py
|-- requirements.txt
|-- Dockerfile
|-- docker-compose.yml
Dockerfile:
FROM python:3.7.5-buster
EXPOSE 5000 # i have tried with and without this
ENV INSTALL_PATH /osr
ENV FLASK_APP osr
ENV FLASK_ENV development
RUN mkdir -p $INSTALL_PATH
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . .
CMD [ "flask", "run", "--host=0.0.0.0"]
docker-compose.yml:
version: '3.8'
services:
web:
build: .
ports:
- '5000:5000'
expose:
- '5000'
volumes:
- .:/osr
__ init __.py
import os
from flask import Flask
def create_app(test_config=None):
app = Flask(__name__, instance_relative_config=True)
app.config.from_mapping(
SECRET_KEY='dev'
)
#app.route('/hello')
def hello():
return 'Hello, World!'
return app
docker-compose build web
docker-compose run web
* Serving Flask app "osr" (lazy loading)
* Environment: development
* Debug mode: on
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 225-441-434
all of the below return "hmm can't reach this page"
http://localhost
http://localhost:5000/hello
http://127.0.0.1:5000/hello
I've even tried going to the containers IP with
docker exec -it 49e677 bash
ip add | grep global
inet 172.21.0.2/16 brd 172.21.255.255 scope global eth0
http://127.21.0.2:5000/hello # nothing
Nothing.
I'm sure the app itself works fine as I can simply run the app directly.
$env:FLASK_APP="osr"
$env:FLASK_ENV="development"
flask run --host=0.0.0.0
And it runs fine
EDIT: UPDATE
I am actually able to get to the container when I run it through simply the Dockerfile... using
docker run -it -p 5000:5000 osr_web # the container built by docker-compose build
With this, I am able to access the endpoint through localhost:5000/hello
So the issue appears to lie in spinning it up through docker-compose run
Does this help at all?
UPDATE 2
I have discovered that when I run docker ps -a I can see that Docker run actually exposes the port, but docker-compose run does not:
Are you sure app works fine itself? I tried to run your python __init__.py file and ended up with an error.
python osr/__init__.py
File "osr/__init__.py", line 11
def hello():
^
IndentationError: unexpected indent
It works after fixing the indentation error.
#app.route('/hello')
def hello():
return 'Hello, World!'
$ docker run -d -p 5000:5000 harik8/osr:latest
76628f86fecb61c0be4a969d3c91c5c575702ad8063b594a6c1c90b498ea25f1
$ curl http://127.0.0.1:5000/hello
Hello, World!
You can't run both docker and docker-compose in port 5000 at the same time. Either run one at a time or change the docker-compose/dockerfile host port.
$ docker ps -a | grep osr
8b885c4a9654 harik8/osr:latest "flask run --host=0.…" 12 seconds ago Up 11 seconds 0.0.0.0:5000->5000/tcp
$ docker ps -a | grep q5
70f38bf11e26 q5_web "flask run --host=0.…" About a minute ago Up 10 seconds 0.0.0.0:5001->5000/tcp
$ docker ps -a | grep q5
f9f6ba999109 q5_web "flask run --host=0.…" 5 minutes ago Up 5 minutes 0.0.0.0:5000->5000/tcp q5_web_1
$ docker ps -a | grep osr
93fb421333e4 harik8/osr:latest "flask run --host=0.…" 18 seconds ago Up 18 seconds 5000/tcp
I found the issue. For reference, I am running these versions:
docker-compose version 1.25.5, build 8a1c60f6
Docker version 19.03.8, build afacb8b
There were a couple issues: First and foremost to get the ports exposed I needed to run one of two options.
docker-compose up web
or
docker-compose run --service-ports web
Simply running docker-compose run web would not expose the ports.
Once this was finished, I was able to access the endpoint. However I started getting another odd error,
flask.cli.NoAppException
flask.cli.NoAppException: Failed to find Flask application or factory in module
"osr". Use "FLASK_APP=osr:name to specify one.
I had not experienced this simply using docker run -it -p 5000:5000 osr_web which was odd. However I noticed I had not set the work directory in the Dockerfile.
I changed the Dockerfile to this:
FROM python:3.7.5-buster
EXPOSE 5000
ENV INSTALL_PATH /osr
ENV FLASK_APP osr
ENV FLASK_ENV development
RUN mkdir -p $INSTALL_PATH
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
# added this line
WORKDIR $INSTALL_PATH
COPY . .
CMD [ "flask", "run", "--host=0.0.0.0"]
I believe you could get away without setting the WORK_DIR if you turn the flask application into a package and install it.
I am having some issues deploying to Cloud Run lately. When I am trying to deploy the below Dockerfile to Cloud Run, it ends up with the error Failed to start and then listen on the port defined by the PORT environment variable.:
FROM phpmyadmin/phpmyadmin:latest
EXPOSE 8080
RUN sed -i 's/80/${PORT}/g' /etc/apache2/sites-available/000-default.conf /etc/apache2/ports.conf
ENTRYPOINT [ "/docker-entrypoint.sh" ]
CMD [ "apache2-foreground" ]
The ENTRYPOINT and CMD were added separately even though the phpmyadmin/phpmyadmin:latest uses this same ENTRYPOINT and CMD to see if that would solve it, though it is not required. The same Docker image when deployed using docker run runs properly and listens on port 8080. Is there something I am doing wrong?
This is the command I use to deploy:
gcloud run deploy phpmyadmin --memory=1Gi --platform=managed \
--allow-unauthenticated --add-cloudsql-instances project_id:us-central1:db-name \
--region=us-central1 --image gcr.io/project_id/phpmyadmin:1.3 \
--update-env-vars PMA_HOST=localhost,PMA_SOCKET="/cloudsql/project_id:us-central1:db-name",PMA_ABSOLUTE_URI=phpmyadmin.domain.com
This is all I can find in the logs. (Have redacted some data):
https://gist.github.com/shanukk27/9dd4b3076c55307bd6e853a76e7a34e0
Cloud Run runtime environment seems to be slightly different than Docker run command. You can't use ENTRYPOINT and CMD in the same time
ENTRYPOINT [ "/docker-entrypoint.sh" ]
CMD [ "apache2-foreground" ]
It works with Docker Run (Why? Docker issue? Docker feature?) and not on Cloud Run (missing feature? bug?).
Use only one of them, for example:
ENTRYPOINT /docker-entrypoint.sh && apache2-foreground
EDIT
A strange remark shared by Shanu is the 2 command works with Wordpress deployment, and doesn't work here.
FROM wordpress:5.3.2-php7.3-apache
EXPOSE 8080
# Copy custom entrypoint from repo
COPY cloud-run-entrypoint.sh /usr/local/bin/
# Change apache listening port and set permission for docker entrypoint
RUN sed -i 's/80/${PORT}/g' /etc/apache2/sites-available/000-default.conf /etc/apache2/ports.conf && \
chmod +x /usr/local/bin/cloud-run-entrypoint.sh
# Wordpress conf
COPY wordpress/. /var/www/html/
# Custom entrypoint
ENTRYPOINT ["cloud-run-entrypoint.sh","docker-entrypoint.sh"]
# Start apache when docker container starts
CMD ["apache2-foreground"]
The problem is solved here, but the reason is not clear
Note to Googler (Steren? Ahmet?): Can you share more details on this behavior?
Based on this guide:
https://shekhargulati.com/2019/01/18/dockerizing-a-vue-js-application/
I have created a sample VueJS app and created a docker image:
docker build -t myapp .
based on the below Dockerfile:
# base image
FROM node:10.15.0
# set working directory
RUN mkdir /usr/src/app
WORKDIR /usr/src/app
# add `/usr/src/app/node_modules/.bin` to $PATH
ENV PATH /usr/src/app/node_modules/.bin:$PATH
# install and cache app dependencies
COPY package.json /usr/src/app/package.json
RUN npm install
RUN npm install -g #vue/cli
# start app
CMD ["npm", "run", "serve"]
Next I run a docker container with:
docker run -it -v ${PWD}:/usr/src/app -v /usr/src/app/node_modules -p 5000:5000 myapp
and get this (successfull) output:
DONE Compiled successfully in 4644ms 4:05:10 PM
No type errors found
No lint errors found
Version: typescript 3.4.3, tslint 5.15.0
Time: 4235ms
App running at:
- Local: http://localhost:8080/
It seems you are running Vue CLI inside a container.
Access the dev server via http://localhost:<your container's external mapped port>/
Note that the development build is not optimized.
To create a production build, run npm run build.
I then try to access the application from my browser on: http://localhost:5000/ but I just get a The connection was reset error.
I have also tried to inspect the port information on the running container with:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
755d2745bce2 myapp "npm run serve" 22 seconds ago Up 18 seconds 0.0.0.0:5000->5000/tcp confident_mirzakhani
$ docker port confident_mirzakhani
5000/tcp -> 0.0.0.0:5000
But that basically confirms the port info I passed to the run command.
Any suggestion on how to access the VueJS application in the container from the browser on my host?