Related
I have been trying to use a custom dockerfile for mounting dags and plugins as follows:
FROM apache/airflow:2.3.0-python3.7
COPY ./dags/ /opt/airflow/dags/
COPY ./plugins/ /opt/airflow/plugins/docker push
COPY requirements.txt .
RUN pip install -r requirements.txt
EXPOSE 5555
which I am building as:
docker build -f base.dockerfile --pull --tag lqc-airflow:0.0.1 .
minikube image load lqc-airflow:0.0.1
and then doing a helm install
helm upgrade $RELEASE_NAME apache-airflow/airflow --namespace $NAMESPACE --set images.airflow.repository=lqc-airflow --set images.airflow.tag=0.0.1
which however is making just the airflow-worker-0 pod fail due to the following error:
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 38, in main
args.func(args)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/celery_command.py", line 130, in worker
session = celery_app.backend.ResultSession()
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 109, in ResultSession
**self.engine_options)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/session.py", line 88, in session_factory
self.prepare_models(engine)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/session.py", line 72, in prepare_models
ResultModelBase.metadata.create_all(engine)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 4745, in create_all
ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 3007, in _run_ddl_visitor
with self.begin() as conn:
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2923, in begin
conn = self.connect(close_with_result=close_with_result)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 3095, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 91, in __init__
else engine.raw_connection()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 3174, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 3145, in _wrap_pool_connect
e, dialect, self
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2004, in _handle_dbapi_exception_noconnection
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 3141, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 301, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 755, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 419, in checkout
rec = pool._do_get()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 259, in _do_get
return self._create_connection()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 247, in _create_connection
return _ConnectionRecord(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 362, in __init__
self.__connect(first_connect_check=True)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 599, in __connect
connection = pool._invoke_creator(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/create.py", line 578, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 583, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.7/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "postgres" to address: Temporary failure in name resolution
I am just following the reading advisory from airflow: https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html
please note that there are no such name resolution errors if I dont use my custom docker file. Kindly help!
thanks #Hussein also for your support, but I could solve it by myself. I saw that the logs of the airflow-migrations pods, complaint of a revision id not being found. So somehow I chanced upon: https://airflow.apache.org/docs/apache-airflow/stable/migrations-ref.html
and there I saw the airflow tag which was above/carrying that alembic revision.
My revision id was ecb43d2a1842 and thus the changes to my docker file were:
FROM apache/airflow:2.4.3
COPY ./dags/ /opt/airflow/dags/
COPY ./plugins/ /opt/airflow/plugins/
COPY requirements.txt .
RUN pip install -r requirements.txt
thus 2.4.3 was the catch.
As I have stated in the title, I'm having trouble installng Frappe/ERPNext on my Docker. I followed instructions from their official Github repo here.
After opening frappe_docker folder in VSCode, when trying to execute the bench init command I got an error. Below are the full code:
frappe#84badc593d50:/workspace/development$ bench init --skip-redis-config-generation --frappe-branch version-14 frappe-bench
Traceback (most recent call last):
File "/home/frappe/.bench/bench/commands/make.py", line 68, in init
init(
File "/home/frappe/.bench/bench/utils/render.py", line 105, in wrapper_fn
return fn(*args, **kwargs)
File "/home/frappe/.bench/bench/utils/system.py", line 63, in init
bench.setup.dirs()
File "/home/frappe/.bench/bench/utils/render.py", line 126, in wrapper_fn
return fn(*args, **kwargs)
File "/home/frappe/.bench/bench/bench.py", line 337, in dirs
os.makedirs(self.bench.name, exist_ok=True)
File "/home/frappe/.pyenv/versions/3.10.5/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: 'frappe-bench'
I have had user added to docker group as per instruction and have followed it step by step but I got the error above. I've tried run chmod -R frappe:frappe * at the frappe folder but the error still persist.
My machine is running Pop!_OS 20.04 and Docker Desktop 4.16.1
My bad guys, I found the answer.
All I need to do was run chown frappe:frappe /workspace/development/ and all is well. I can run bench init --skip-redis-config-generation --frappe-branch version-14 frappe-bench without a problem after that.
I'm new to docker and I'm trying to follow this simple "getting started" tutorial https://docs.docker.com/compose/gettingstarted/ using a newly first time installation of docker (for Windows 10) dowloaded from here: https://hub.docker.com/editions/community/docker-ce-desktop-windows/.
At step 4 of this tutorial i get this error:
PS D:\composetest> docker-compose up
Building web
Traceback (most recent call last):
File "site-packages\docker\credentials\store.py", line 80, in _execute
File "subprocess.py", line 395, in check_output
File "subprocess.py", line 487, in run
subprocess.CalledProcessError: Command '['C:\\Program Files\\Docker\\Docker\\resources\\bin\\docker-credential-desktop.EXE', 'list']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 6, in <module>
File "compose\cli\main.py", line 72, in main
File "compose\cli\main.py", line 128, in perform_command
File "compose\cli\main.py", line 1078, in up
File "compose\cli\main.py", line 1074, in up
File "compose\project.py", line 548, in up
File "compose\service.py", line 367, in ensure_image_exists
File "compose\service.py", line 1106, in build
File "site-packages\docker\api\build.py", line 261, in build
File "site-packages\docker\api\build.py", line 308, in _set_auth_headers
File "site-packages\docker\auth.py", line 302, in get_all_credentials
File "site-packages\docker\credentials\store.py", line 71, in list
File "site-packages\docker\credentials\store.py", line 93, in _execute
docker.credentials.errors.StoreError: Credentials store docker-credential-desktop exited with "error listing credentials - err: exit status 1, out: `Impossibile trovare elemento.`".
[13284] Failed to execute script docker-compose
EDIT: The accepted solution in docker-compose unable to start, unfortunately, did not work.
What is going wrong?
I have the same issue...
Try:
$nano ~/.docker/config.json
In this file change credsStore to credStore
Now run your docker-compose. If its not works try sudo
try to add to the environment path :
C:\Program Files\Docker\Docker\resources\bin
If you are using WSL2 run this command
sudo ln -s /mnt/c/Program\ Files/Docker/Docker/resources/bin/docker-credential-desktop.exe /usr/bin/docker-credential-desktop.exe
My CKAN Docker install seems perfect except that the ckan container exits immediately after a restart.strong textI have repeatedly tried a rebuild of the docker containers. I see no errors except a reference to the deprecation of python2. Please advise on any issues that I may have overlooked or something missing. This installation is on Ubuntu 19.10.
docker log is as follows:
Command 'db' not known (you may need to run setup.py egg_info)
Known commands:
create Create the file layout for a Python distribution
Traceback (most recent call last):
File "/usr/local/bin/ckan-paster", line 8, in <module>
sys.exit(run())
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 806, in run
command.load().summary))
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2321, in load
return self.resolve()
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2327, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/lib/ckan/venv/src/ckan/ckanext/datapusher/cli.py", line 7, in <module>
import ckan.lib.cli as cli
File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 17, in <module>
from ckan.cli import load_config as _get_config
File "/usr/lib/ckan/venv/src/ckan/ckan/cli/__init__.py", line 8, in <module>
from configparser import ConfigParser
ImportError: No module named configparser
Command 'db' not known (you may need to run setup.py egg_info)
Known commands:
create Create the file layout for a Python distribution
Traceback (most recent call last):
File "/usr/local/bin/ckan-paster", line 8, in <module>
sys.exit(run())
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 806, in run
command.load().summary))
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2321, in load
return self.resolve()
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2327, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/lib/ckan/venv/src/ckan/ckanext/datapusher/cli.py", line 7, in <module>
import ckan.lib.cli as cli
File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 17, in <module>
from ckan.cli import load_config as _get_config
File "/usr/lib/ckan/venv/src/ckan/ckan/cli/__init__.py", line 8, in <module>
from configparser import ConfigParser
ImportError: No module named configparser
Traceback (most recent call last):
File "/usr/local/bin/ckan-paster", line 8, in <module>
Command 'db' not known (you may need to run setup.py egg_info)
Known commands:
create Create the file layout for a Python distribution
sys.exit(run())
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 806, in run
command.load().summary))
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2321, in load
return self.resolve()
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2327, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/lib/ckan/venv/src/ckan/ckanext/datapusher/cli.py", line 7, in <module>
import ckan.lib.cli as cli
File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 17, in <module>
from ckan.cli import load_config as _get_config
File "/usr/lib/ckan/venv/src/ckan/ckan/cli/__init__.py", line 8, in <module>
from configparser import ConfigParser
ImportError: No module named configparser
You are using python 2.7 but importing configparser, which is Python 3 module. Check which version you are pooling from the Docker, I think you need 2.8.x tag as the latest stable.
I'm using Google Cloud Registry, associating with Docker using gcloud auth configure-docker.
https://cloud.google.com/sdk/gcloud/reference/auth/configure-docker
However, when my computer is offline and I run docker-compose up I get an error where it tries to communicate/authenticate with Google.
how can I use docker offline now that I've started using GCR?
$ docker-compose up --build --force-recreate -d
Building solr
ERROR: (gcloud.auth.docker-helper) There was a problem refreshing your current auth tokens: Unable to find the server at www.googleapis.com
Please run:
$ gcloud auth login
to obtain new credentials, or if you have already logged in with a
different account:
$ gcloud config set account ACCOUNT
to select an already authenticated account to use.
Traceback (most recent call last):
File "site-packages/dockerpycreds/store.py", line 74, in _execute
File "subprocess.py", line 336, in check_output
File "subprocess.py", line 418, in run
subprocess.CalledProcessError: Command '['/Users/me/google-cloud-sdk/bin/docker-credential-gcloud', 'get']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "site-packages/docker/auth.py", line 129, in _resolve_authconfig_credstore
File "site-packages/dockerpycreds/store.py", line 35, in get
File "site-packages/dockerpycreds/store.py", line 87, in _execute
dockerpycreds.errors.StoreError: Credentials store docker-credential-gcloud exited with "".
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 6, in <module>
File "compose/cli/main.py", line 71, in main
File "compose/cli/main.py", line 127, in perform_command
File "compose/cli/main.py", line 1080, in up
File "compose/cli/main.py", line 1076, in up
File "compose/project.py", line 475, in up
File "compose/service.py", line 342, in ensure_image_exists
File "compose/service.py", line 1082, in build
File "site-packages/docker/api/build.py", line 251, in build
File "site-packages/docker/api/build.py", line 307, in _set_auth_headers
File "site-packages/docker/auth.py", line 96, in resolve_authconfig
File "site-packages/docker/auth.py", line 146, in _resolve_authconfig_credstore
docker.errors.DockerException: Credentials store error: StoreError('Credentials store docker-credential-gcloud exited with "".',)
[26419] Failed to execute script docker-compose
Docker-compose will read your YAML file to configure your application’s services , so if in your YAML you are using a Docker image or a personalized images and you don't have them on local, docker will try download them from the principal registry in your configuration, in this case Google Cloud Registry and if your are offline you will get an error.