Given this Dockerfile:
FROM python:3.8-slim-buster
EXPOSE 5000
ADD . /data/sample.csv
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /app
COPY . /app
I can run this flask app on my local machine, no errors. Pandas grabs the data from the csv and displays it in a return statement from the function (below):
from flask import Flask
import pandas as pd
app = Flask(__name__)
df = pd.read_csv("data\sample.csv")
lst_one = df['col1'].to_list()
lst_two = df['col2'].to_list()
lst_three = df['col3'].to_list()
lang = "World!"
#app.route('/')
def hello_world():
return f'Hello, {lang} {lst_one}, {lst_three}'
However, after creating Dockerfile and requirements.txt and building image, when running the container, this error persists. (Notice how the error message has the file path slashes going in the opposite direction from those in my code???)
[2021-04-30 01:34:04 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2021-04-30 01:34:04 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2021-04-30 01:34:04 +0000] [1] [INFO] Using worker: sync
[2021-04-30 01:34:04 +0000] [8] [INFO] Booting worker with pid: 8
[2021-04-30 01:34:05 +0000] [8] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 119, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.8/site-packages/gunicorn/util.py", line 358, in import_app
mod = importlib.import_module(module)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/app/app.py", line 5, in <module>
df = pd.read_csv("data\sample.csv")
File "/usr/local/lib/python3.8/site-packages/pandas/io/parsers.py", line 610, in read_csv
return _read(filepath_or_buffer, kwds)
File "/usr/local/lib/python3.8/site-packages/pandas/io/parsers.py", line 462, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/usr/local/lib/python3.8/site-packages/pandas/io/parsers.py", line 819, in __init__
self._engine = self._make_engine(self.engine)
File "/usr/local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1050, in _make_engine
return mapping[engine](self.f, **self.options) # type: ignore[call-arg]
File "/usr/local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1867, in __init__
self._open_handles(src, kwds)
File "/usr/local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1362, in _open_handles
self.handles = get_handle(
File "/usr/local/lib/python3.8/site-packages/pandas/io/common.py", line 642, in get_handle
handle = open(
***FileNotFoundError: [Errno 2] No such file or directory: 'data\\sample.csv'***
[2021-04-30 01:34:05 +0000] [8] [INFO] Worker exiting (pid: 8)
[2021-04-30 01:34:05 +0000] [1] [INFO] Shutting down: Master
[2021-04-30 01:34:05 +0000] [1] [INFO] Reason: Worker failed to boot
Tried a different path, an absolute path, everything and Docker just cannot recognize that this directory/file exists. The actual app I need to containerize has to be able to access data outside the container, eventually, but as a first step, I cannot even get Docker to recognize this data WITHIN the app. So, all you Docker experts, please step up and let me know what presumably simple thing I am obviously missing here. Thank you so much.
I'm trying to deploy a docker image to a remote server using docker-compose but I'm getting authentication issue:
Command:
docker-compose -H "ssh://root#host_ip" --tlscacert "~/.ssh/private_key" ps
Output:
sudo docker-compose -H "ssh://root#jenkins.evercam.io" --tlscacert "~/.ssh/id_jenkins" -f evercam_camera_server/docker-compose.yml ps
Traceback (most recent call last):
File "bin/docker-compose", line 3, in <module>
File "compose/cli/main.py", line 67, in main
File "compose/cli/main.py", line 123, in perform_command
File "compose/cli/command.py", line 69, in project_from_options
File "compose/cli/command.py", line 132, in get_project
File "compose/cli/docker_client.py", line 43, in get_client
File "compose/cli/docker_client.py", line 170, in docker_client
File "docker/api/client.py", line 164, in __init__
File "docker/transport/sshconn.py", line 113, in __init__
File "docker/transport/sshconn.py", line 121, in _connect
File "paramiko/client.py", line 446, in connect
File "paramiko/client.py", line 764, in _auth
File "paramiko/client.py", line 740, in _auth
File "paramiko/transport.py", line 1580, in auth_publickey
File "paramiko/auth_handler.py", line 250, in wait_for_response
paramiko.ssh_exception.AuthenticationException: Authentication failed.
[28861] Failed to execute script docker-compose
docker-compose version 1.27.4, build 40524192
docker version 20.10.5, build 55c4c88
ubuntu version 18.04
I tried with a recent version of docker compose which is 1.28.5 it's able to access the given host, but strangely it's asking the password five times!
Im trying to run use docker-compose to run the rasa server alongwith the actions server.
This is my docker-compose.yml
version: '3.0'
services:
rasa:
image: rasa/rasa:latest-full
ports:
- 5005:5005
volumes:
- ./:/app
command:
- x
action_server:
image: rasa/rasa-sdk:latest
volumes:
- ./actions:/app/actions
Currently i get two errors on doing docker-compose up
Failed to register package 'actions.actions'.
ModuleNotFoundError: No module named 'rasax'
Here is the complete trace:
docker-compose up
Recreating wlbot_rasa_1 ... done
Starting wlbot_action_server_1 ... done
Attaching to wlbot_action_server_1, wlbot_rasa_1
action_server_1 | 2019-08-22 19:38:10 INFO rasa_sdk.endpoint - Starting action endpoint server...
action_server_1 | 2019-08-22 19:38:10 ERROR rasa_sdk.executor - Failed to register package 'actions.actions'.
action_server_1 | Traceback (most recent call last):
action_server_1 | File "/app/rasa_sdk/executor.py", line 184, in register_package
action_server_1 | self._import_submodules(package)
action_server_1 | File "/app/rasa_sdk/executor.py", line 171, in _import_submodules
action_server_1 | package = importlib.import_module(package)
action_server_1 | File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
action_server_1 | return _bootstrap._gcd_import(name[level:], package, level)
action_server_1 | File "<frozen importlib._bootstrap>", line 994, in _gcd_import
action_server_1 | File "<frozen importlib._bootstrap>", line 971, in _find_and_load
action_server_1 | File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
action_server_1 | ModuleNotFoundError: No module named 'actions.actions'
action_server_1 | 2019-08-22 19:38:10 INFO rasa_sdk.endpoint - Action endpoint is up and running. on ('0.0.0.0', 5055)
rasa_1 | Traceback (most recent call last):
rasa_1 | File "/usr/local/bin/rasa", line 10, in <module>
rasa_1 | sys.exit(main())
rasa_1 | File "/usr/local/lib/python3.6/site-packages/rasa/__main__.py", line 76, in main
rasa_1 | cmdline_arguments.func(cmdline_arguments)
rasa_1 | File "/usr/local/lib/python3.6/site-packages/rasa/cli/x.py", line 291, in rasa_x
rasa_1 | run_locally(args)
rasa_1 | File "/usr/local/lib/python3.6/site-packages/rasa/cli/x.py", line 308, in run_locally
rasa_1 | from rasax.community import local # pytype: disable=import-error
rasa_1 | ModuleNotFoundError: No module named 'rasax'
wlbot_rasa_1 exited with code 1
I can't find in the docs about how to configure this for Rasa X instead of rasa run and also specify a particular model.
Please suggest. Thanks.
In your docker-compose for rasa, you have to pull image rasa/rasa-x (which contains rasa + rasa x). For the actions, try to specify the file (not the directory) like "./actions.py:/app/actions/actions.py"
I successfully installed docker and nvidia-docker on ubuntu 18.04
I pull this image from NVIDIA's GPU cloud
https://ngc.nvidia.com/catalog/containers/nvidia:caffe
and ran it with this command
nvidia-docker run -it --rm -v /home/stefan/Dropbox:/data -p 8888:8888 nvcr.io/nvidia/caffe:19.03-py2 sh
The container gives me a shell prompt and it seems to work, for example
# nvidia-smi
results in
Sat Mar 30 21:03:30 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.39 Driver Version: 418.39 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 105... On | 00000000:01:00.0 On | N/A |
| 20% 30C P8 N/A / 75W | 441MiB / 4038MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
It sees my wimpy gpu. I try to run jupyter with this command
#jupyter-notebook
but I get
[I 21:05:18.088 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
Traceback (most recent call last):
File "/usr/local/bin/jupyter-notebook", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-7>", line 2, in initialize
File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/notebook/notebookapp.py", line 1628, in initialize
self.init_webapp()
File "/usr/local/lib/python2.7/dist-packages/notebook/notebookapp.py", line 1407, in init_webapp
self.http_server.listen(port, self.ip)
File "/usr/local/lib/python2.7/dist-packages/tornado/tcpserver.py", line 143, in listen
sockets = bind_sockets(port, address=address)
File "/usr/local/lib/python2.7/dist-packages/tornado/netutil.py", line 168, in bind_sockets
sock.bind(sockaddr)
File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 99] Cannot assign requested address
I know jupyter is installed in the container because when I type
#jupyter --version
I get
4.4.0
Typing
# jupyter
gives
usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir]
[--paths] [--json]
[subcommand]
jupyter: error: one of the arguments --version subcommand --config-dir --data-dir --runtime-dir --paths is required
I have several notebooks in the host directory I attached to the container
# ls
NBA.ipynb exponents.ipynb hello_deep_learning-master
but nothing seems to work
# jupyter NBA.ipynb
Error executing Jupyter command 'NBA.ipynb': [Errno 2] No such file or directory
# jupyter notebook NBA.ipynb
Traceback (most recent call last):
File "/usr/local/bin/jupyter-notebook", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-7>", line 2, in initialize
File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/notebook/notebookapp.py", line 1628, in initialize
self.init_webapp()
File "/usr/local/lib/python2.7/dist-packages/notebook/notebookapp.py", line 1407, in init_webapp
self.http_server.listen(port, self.ip)
File "/usr/local/lib/python2.7/dist-packages/tornado/tcpserver.py", line 143, in listen
sockets = bind_sockets(port, address=address)
File "/usr/local/lib/python2.7/dist-packages/tornado/netutil.py", line 168, in bind_sockets
sock.bind(sockaddr)
File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 99] Cannot assign requested address
# jupyter-notebook NBA.ipynb
Traceback (most recent call last):
File "/usr/local/bin/jupyter-notebook", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-7>", line 2, in initialize
File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/notebook/notebookapp.py", line 1628, in initialize
self.init_webapp()
File "/usr/local/lib/python2.7/dist-packages/notebook/notebookapp.py", line 1407, in init_webapp
self.http_server.listen(port, self.ip)
File "/usr/local/lib/python2.7/dist-packages/tornado/tcpserver.py", line 143, in listen
sockets = bind_sockets(port, address=address)
File "/usr/local/lib/python2.7/dist-packages/tornado/netutil.py", line 168, in bind_sockets
sock.bind(sockaddr)
File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 99] Cannot assign requested address
I think it's a syntax issue because this works
docker run -it --rm -v ~/Dropbox:/tf/notebooks -p 8888:8888 tensorflow/tensorflow:latest-py3-jupyter
It starts the jupyter server in the container and in a browser I can open a notebook at 127.0.0.1 which shows a directory where I can see a folder named 'notebooks' which contains my Dropbox contents. Just as expected since I mounted my dropbox folder as a volume in the command above.
But if I type this
docker run -it --rm -v ~/Dropbox:/tf/notebooks -p 8888:8888 tensorflow/tensorflow:latest-py3-jupyter sh
I am in a shell but cannot start jupyter. I get the same error as I was before with the nvcr.io/nvidia/caffe image. How can I start jupyter AFTER I am in the running docker container shell?
I think I figured it out. At the shell prompt of the container , I type
jupyter notebook --ip=0.0.0.0 --allow-root
I'll leave this here in case any other noob like me has a similar problem. (Unless the moderator feels it should be edited or nuked)
I'm trying to run ckan in docker container following the steps in: http://docs.ckan.org/en/ckan-2.4.7/maintaining/installing/install-using-docker.html
Links for the images available from : https://hub.docker.com/u/ckan/ seems to be updated at the time of post this questions (2 days ago).
Well, I have followed the steps:
$ docker run -d --name db ckan/postgresql
$ docker run -d --name solr ckan/solr
$ docker run -d -p 80:80 --link db:db --link solr:solr ckan/ckan
And everythings ok, but the question is, How can I access to ckan portal???
Using docker inspect <ckan_image> I get something like this:
"NetworkSettings": {
"Bridge": "",
"SandboxID": "c66a4d1bb1a27c160f1655a9c660d24337e85053e8a8ad1e1a2c570ed217223e",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"5000/tcp": null,
"80/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "80"
}
]
},
"SandboxKey": "/var/run/docker/netns/c66a4d1bb1a2",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "7985fc49cc7795b668ca4dfc5812f0ffa40f305f29a7726b15947890051f2014",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:04",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "88de6de00bdbc9974e48021ff783378835fc99d09582b8f7ccaab363a605a499",
"EndpointID": "7985fc49cc7795b668ca4dfc5812f0ffa40f305f29a7726b15947890051f2014",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:04",
"DriverOpts": null
}
}
}
Port exposed are 5000 and 80 so, using this ip address and this port it should be accessible.
Am I missing something?
Accessing solr is correct in: http://localhost:8983/solr/#/
UPDATE 1
Following help of Tarun Lalwani and using docker-compose, I think there is a problem with ckan and solr. This is the output error:
See the second line:
Invalid URL u'http://:/solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json': No host supplied
ckan_1 | 2017-07-31 11:23:37,622 INFO [ckan.config.environment] Loading static files from public
****ckan_1 | 2017-07-31 11:23:37,916 ERROR [ckan.lib.search.common] Invalid URL u'http://:/solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json': No host supplied****
ckan_1 | Traceback (most recent call last):
ckan_1 | File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/common.py", line 57, in is_available
ckan_1 | conn.search(q="*:*", rows=1)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 720, in search
ckan_1 | response = self._select(params, handler=search_handler)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 418, in _select
ckan_1 | return self._send_request('get', path)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 366, in _send_request
ckan_1 | timeout=self.timeout)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py", line 515, in get
ckan_1 | return self.request('GET', url, **kwargs)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
ckan_1 | prep = self.prepare_request(req)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py", line 431, in prepare_request
ckan_1 | hooks=merge_hooks(request.hooks, self.hooks),
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/models.py", line 305, in prepare
ckan_1 | self.prepare_url(url, params)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/models.py", line 382, in prepare_url
ckan_1 | raise InvalidURL("Invalid URL %r: No host supplied" % url)
ckan_1 | InvalidURL: Invalid URL u'http://:/solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json': No host supplied
ckan_1 | 2017-07-31 11:23:38,106 WARNI [ckan.lib.search] Problems were found while connecting to the SOLR server
ckan_1 | 2017-07-31 11:23:38,183 INFO [ckan.config.environment] Loading templates from /usr/lib/ckan/default/src/ckan/ckan/templates
ckan_1 | Traceback (most recent call last):
ckan_1 | File "/usr/local/bin/ckan-paster", line 11, in <module>
ckan_1 | sys.exit(run())
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
ckan_1 | invoke(command, command_name, options, args[1:])
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
ckan_1 | exit_code = runner.run(args)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
ckan_1 | result = self.command()
ckan_1 | File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 337, in command
ckan_1 | self._load_config(cmd!='upgrade')
ckan_1 | File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 310, in _load_config
ckan_1 | self.site_user = load_config(self.options.config, load_site_user)
ckan_1 | File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 225, in load_config
ckan_1 | load_environment(conf.global_conf, conf.local_conf)
ckan_1 | File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 111, in load_environment
ckan_1 | p.load_all()
ckan_1 | File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 129, in load_all
ckan_1 | unload_all()
ckan_1 | File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 182, in unload_all
ckan_1 | unload(*reversed(_PLUGINS))
ckan_1 | File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 210, in unload
ckan_1 | plugins_update()
ckan_1 | File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 121, in plugins_update
ckan_1 | environment.update_config()
ckan_1 | File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 289, in update_config
ckan_1 | engine = sqlalchemy.engine_from_config(config, client_encoding='utf8')
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 428, in engine_from_config
ckan_1 | return create_engine(url, **options)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
ckan_1 | return strategy.create(*args, **kwargs)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 50, in create
ckan_1 | u = url.make_url(name_or_url)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 194, in make_url
ckan_1 | return _parse_rfc1738_args(name_or_url)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 240, in _parse_rfc1738_args
ckan_1 | return URL(name, **components)
ckan_1 | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 60, in __init__
ckan_1 | self.port = int(port)
ckan_1 | ValueError: invalid literal for int() with base 10: ''
After this I get:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d44df7bad12f ckan/solr "docker-entrypoint..." 2 hours ago Up 19 minutes 8983/tcp dockercompose_solr_1
8f0c6c815746 ckan/postgresql "docker-entrypoint..." 2 hours ago Up 19 minutes 5432/tcp dockercompose_db_1
The rest seems to be ok.
UPDATE 2
Updated my docker-compose file. I have been made several test. Finally this combination seems to work. Solr works ok (just change the public port with kitematic), postgresql has the database and table and I can access correctly but I get Internal Server Error, so I think I'm close.
My docker-compose file:
version: '3'
services:
solr:
container_name: solr
#Possible options solr
#image: milafrerichs/ckan_solr
#image: miguelbgouveia/solr-docker
image: ckan/solr:dev-v2.6
ports:
- "8983:8983/tcp"
db:
container_name: db
image: ckan/postgresql
ports:
- "5432:5432/tcp"
ckan:
container_name: ckan
image: ckan/ckan:dev-v2.6
depends_on:
- solr
- db
links:
- db:db
ports:
- "5000:5000"
- "80:80"
environment:
DATABASE_URL: "postgresql://ckan:ckan#db:5432/ckan"
SOLR_URL: "http://solr:8983/solr/ckan"
Looking at the apache2 logs, I have no see anything interesting.
This is the complete output:
$ docker-compose -f docker-compose-ckan.yml up
Attaching to db, solr, ckan
solr | Starting Solr on port 8983 from /opt/solr/server
solr |
solr | 0 INFO (main) [ ] o.e.j.u.log Logging initialized #2757ms
solr | 1711 INFO (main) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
db | running bootstrap script ... ok
db | performing post-bootstrap initialization ... ok
db | syncing data to disk ... ok
db |
ckan | Distribution already installed:
ckan | ckan 2.6.3 from /usr/lib/ckan/default/src/ckan
ckan | Creating /etc/ckan/default/ckan.ini
ckan | Now you should edit the config files
ckan | /etc/ckan/default/ckan.ini
ckan | Edited option sqlalchemy.url = "postgresql://ckan_default:pass#localhost/ckan_default"->"postgresql://ckan:ckan#db:5432/ckan" (section "app:main")
ckan | Edited option ckan.site_url = ""->"http://192.168.0.6" (section "app:main")
ckan | Option uncommented and set solr_url = "http://solr:8983/solr/ckan" (section "app:main")
ckan | Option uncommented and set ckan.storage_path = "/var/lib/ckan" (section "app:main")
ckan | Option uncommented and set email_to = "disabled#example.com" (section "app:main")
ckan | Option uncommented and set error_email_from = "ckan#95e87010bd4d" (section "app:main")
solr | 1803 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///opt/solr/server/contexts/] at interval 0
solr | 4046 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
solr | 4080 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext#o.e.j.w.WebAppContext#13a5fe33{/solr,file:///opt/solr/server/solr-webapp/webapp/,STARTING}{/opt/solr/server/solr-webapp/webapp} has uncovered http methods for path: /
solr | 4118 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): WebAppClassLoader=1740189450#67b92f0a
solr | 4163 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
solr | 4169 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /opt/solr/server/solr
solr | 4174 INFO (main) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/opt/solr/server/solr'
solr | 4179 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
solr | 4179 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /opt/solr/server/solr
db | LOG: autovacuum launcher started
solr | 4186 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /opt/solr/server/solr/solr.xml
solr | 4455 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /opt/solr/server/solr
db | done
db | server started
db | done
db | server stopped
db |
db | PostgreSQL init process complete; ready for start up.
db |
db | LOG: database system was shut down at 2017-08-01 22:58:52 UTC
db | LOG: MultiXact member wraparound protections are now enabled
db | LOG: database system is ready to accept connections
solr | 5404 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
solr | 6145 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
solr | 6153 INFO (main) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
solr | 6160 INFO (main) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
solr | 6163 INFO (main) [ ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
solr | 6165 INFO (main) [ ] o.a.s.c.CoreContainer No authentication plugin used.
solr | 6341 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /opt/solr/server/solr
solr | 6353 INFO (main) [ ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=ckan, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
solr | 6356 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Found core ckan in /opt/solr/server/solr/ckan
ckan | *** Running /etc/my_init.d/70_initdb...
**ckan | 2017-08-01 22:58:54,440 ERROR [pysolr] Failed to connect to server at 'http://solr:8983/solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json', are you sure that URL is correct? Checking it in a browser might help: HTTPConnectionPool(host='solr', port=8983): Max retries exceeded with url: /solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb54cbe9050>: Failed to establish a new connection: [Errno 111] Connection refused',))**
ckan | Traceback (most recent call last):
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 361, in _send_request
ckan | timeout=self.timeout)
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py", line 487, in get
ckan | return self.request('GET', url, **kwargs)
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
ckan | resp = self.send(prep, **send_kwargs)
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
ckan | r = adapter.send(request, **kwargs)
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/adapters.py", line 467, in send
ckan | raise ConnectionError(e, request=request)
**ckan | ConnectionError: HTTPConnectionPool(host='solr', port=8983): Max retries exceeded with url: /solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb54cbe9050>: Failed to establish a new connection: [Errno 111] Connection refused',))**
solr | 6399 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
solr | 6665 INFO (main) [ ] o.a.s.s.SolrDispatchFilter user.dir=/opt/solr/server
ckan | 2017-08-01 22:58:54,444 ERROR [ckan.lib.search.common] Failed to connect to server at 'http://solr:8983/solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json', are you sure that URL is correct? Checking it in a browser might help: HTTPConnectionPool(host='solr', port=8983): Max retries exceeded with url: /solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb54cbe9050>: Failed to establish a new connection: [Errno 111] Connection refused',))
ckan | Traceback (most recent call last):
ckan | File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/common.py", line 56, in is_available
ckan | conn.search(q="*:*", rows=1)
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 710, in search
ckan | response = self._select(params, handler=search_handler)
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 411, in _select
ckan | return self._send_request('get', path)
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 370, in _send_request
ckan | raise SolrError(error_message % params)
ckan | SolrError: Failed to connect to server at 'http://solr:8983/solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json', are you sure that URL is correct? Checking it in a browser might help: HTTPConnectionPool(host='solr', port=8983): Max retries exceeded with url: /solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb54cbe9050>: Failed to establish a new connection: [Errno 111] Connection refused',))
ckan | 2017-08-01 22:58:54,444 WARNI [ckan.lib.search] Problems were found while connecting to the SOLR server
ckan | 2017-08-01 22:58:55,458 ERROR [pysolr] Solr responded with an error (HTTP 503): [Reason: Error 503 {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore is loading,code=503}]
ckan | 2017-08-01 22:58:55,458 ERROR [ckan.lib.search.common] Solr responded with an error (HTTP 503): [Reason: Error 503 {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore is loading,code=503}]
ckan | Traceback (most recent call last):
ckan | File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/common.py", line 56, in is_available
ckan | conn.search(q="*:*", rows=1)
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 710, in search
ckan | response = self._select(params, handler=search_handler)
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 411, in _select
ckan | return self._send_request('get', path)
ckan | File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 386, in _send_request
ckan | raise SolrError(error_message % (resp.status_code, solr_message))
ckan | SolrError: Solr responded with an error (HTTP 503): [Reason: Error 503 {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore is loading,code=503}]
db | ERROR: relation "user" does not exist at character 465
db | STATEMENT: SELECT "user".password AS user_password, "user".id AS user_id, "user".name AS user_name, "user".openid AS user_openid, "user".fullname AS user_fullname, "user".email AS user_email, "user".apikey AS user_apikey, "user".created AS user_created, "user".reset_key AS user_reset_key, "user".about AS user_about, "user".activity_streams_email_notifications AS user_activity_streams_email_notifications, "user".sysadmin AS user_sysadmin, "user".state AS user_state
db | FROM "user"
db | WHERE "user".name = 'default' OR "user".id = 'default' ORDER BY "user".name
db | LIMIT 1
solr | 13480 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.r.RestManager Initializing 0 registered ManagedResources
db | ERROR: relation "user" does not exist at character 465
db | STATEMENT: SELECT "user".password AS user_password, "user".id AS user_id, "user".name AS user_name, "user".openid AS user_openid, "user".fullname AS user_fullname, "user".email AS user_email, "user".apikey AS user_apikey, "user".created AS user_created, "user".reset_key AS user_reset_key, "user".about AS user_about, "user".activity_streams_email_notifications AS user_activity_streams_email_notifications, "user".sysadmin AS user_sysadmin, "user".state AS user_state
db | FROM "user"
db | WHERE "user".name = 'default' OR "user".id = 'default' ORDER BY "user".name
db | LIMIT 1
solr | 13577 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
solr | 13640 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
solr | 13653 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.h.c.SpellCheckComponent No queryConverter defined, using default converter
solr | 13700 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.h.c.QueryElevationComponent Loading QueryElevation from: /opt/solr/server/solr/ckan/conf/elevate.xml
solr | 13914 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
solr | 14015 INFO (searcherExecutor-7-thread-1-processing-x:ckan) [ x:ckan] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher#2cd58256[ckan] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
solr | 14018 INFO (searcherExecutor-7-thread-1-processing-x:ckan) [ x:ckan] o.a.s.c.QuerySenderListener QuerySenderListener done.
solr | 14021 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
solr | 14023 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
solr | 14026 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
solr | 14035 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1574571440349380608
solr | 14020 INFO (searcherExecutor-7-thread-1-processing-x:ckan) [ x:ckan] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
solr | 14075 INFO (searcherExecutor-7-thread-1-processing-x:ckan) [ x:ckan] o.a.s.c.SolrCore [ckan] Registered new searcher Searcher#2cd58256[ckan] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
solr | 14088 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.u.UpdateLog Took 65.0ms to seed version buckets with highest version 1574571440349380608
db | WARNING: there is already a transaction in progress
ckan | 2017-08-01 22:58:55,460 WARNI [ckan.lib.search] Problems were found while connecting to the SOLR server
ckan | Initialising DB: SUCCESS
ckan | *** Running /etc/rc.local...
ckan | *** Booting runit daemon...
ckan | *** Runit started as PID 25
ckan | * Starting Postfix Mail Transport Agent postfix
ckan | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.4. Set the 'ServerName' directive globally to suppress this message
ckan | ...done.
solr | 14092 INFO (coreLoadExecutor-6-thread-1) [ x:ckan] o.a.s.c.CoreContainer registering core: ckan
solr exited with code 137
db | WARNING: there is no transaction in progress
db exited with code 137
Any suggestions?
Thanks
Use docker-compose which makes it easier to do these things. Below file would help you
version: '2'
services:
db:
image: ckan/postgresql
solr:
image: ckan/solr
ckan:
image: ckan
ports:
- "80:80"
- "5000:5000"
Do a docker-compose up to get this running. Then you can access the ckan at http://<HOSTIPofDocker>:80. If you are not able to access it then make sure to run docker-compose ps to check everything is running and check logs using docker-compose logs -f ckan
The docker installation docs are out of date. I'm currently working on an update to add install docs for Docker Compose in this pull request. Hope this helps!