Cannot see Ipython notebook interface on a browser using Docker Kitematic - docker

I found data science environment image on Kitematic and so I installed and tried use it. But Although I could successfully ran and logs say that
The IPython Notebook is running at: https://[all ip addresses on your system]:8888/
, I cannot open localhost:8888. Could someone help?
Docker port:8888
MAC IP port: 192.168.99.100:32768
Below is Container log on Kitematic.
Generating a 2048 bit RSA private key
...................................+++
........................................................+++
writing new private key to '/key.pem'
-----
[W 22:46:12.956 NotebookApp](B Unrecognized alias: '--matplotlib=inline', it will probably have no effect.
[I 22:46:12.966 NotebookApp](B Writing notebook server cookie secret to /root/.ipython/profile_default/security/notebook_cookie_secret
[I 22:46:12.968 NotebookApp](B Using MathJax from CDN: https://cdn.mathjax.org/mathjax/latest/MathJax.js
[I 22:46:13.013 NotebookApp](B Serving notebooks from local directory: /notebooks
[I 22:46:13.015 NotebookApp](B 0 active kernels
[I 22:46:13.015 NotebookApp](B The IPython Notebook is running at: https://[all ip addresses on your system]:8888/
[I 22:46:13.016 NotebookApp](B Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 22:46:49.788 NotebookApp](B SSL Error on 7 ('192.168.99.1', 63676): [Errno 1] _ssl.c:510: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
[E 22:46:49.797 NotebookApp](B Uncaught exception
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/http1connection.py", line 693, in _server_request_loop
ret = yield conn.read_response(request_delegate)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
value = future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 876, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/http1connection.py", line 168, in _read_message
quiet_exceptions=iostream.StreamClosedError)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
value = future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
SSLError: [Errno 1] _ssl.c:510: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

If you are not on Linux, you would need to port-forward 8888 to your actual host (Windows or Mac) through the VM.
See "Connect to a Service running inside a docker container from outside"
VBoxManage controlvm "default" natpf1 "tcp-port8888,tcp,,8888,,8888"
VBoxManage controlvm "default" natpf1 "udp-port8888,udp,,8888,,8888"
(Replace default with the name of your docker-machine: see docker-machine ls)
You can also try and use directly the IP of your docker machine:
docker-machine IP <name of VM>

Related

How to call non-interactively docker-compose run command - SSH?

Description of the situation : I have a Ubuntu VM running on remote computer B. I can access this computer B via SSH executed on my host (computer A).
When I enter an interactive shell via SSH on computer B, I can execute all my docker commands without any trouble, the steps are the following :
ssh user#remote_IP (enters interactively in the remote shell)
docker-compose run -d --name wfirst --user 1000:1000 container (compose a docker container and starts it)
This works perfectly, my container is mounted, up and running.
However, if I try to run that command in a non-interactive way by writing in my host terminal :
ssh user#remoteIP "cd /path_to_docker_file;docker-compose run -d --name wfirst --user 1000:1000 container"
The command does not succeed, my container is not mounted and does not run. I was able to get more information by using the "--verbose" flag on the docker-compose command.
Here the interesting part of the output of the successful method :
compose.config.config.find: Using configuration files: ./docker-compose.yml
WARNING: compose.config.environment.__getitem__: The DISPLAY variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The NO_PROXY variable is not set. Defaulting to a blank string.
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
urllib3.connectionpool._new_conn: Starting new HTTP connection (1): localhost:2375
urllib3.connectionpool._make_request: http://localhost:2375 "GET /v1.25/version HTTP/1.1" 200 758
compose.cli.command.get_client: docker-compose version 1.21.0, build unknown
docker-py version: 3.4.1
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1c 28 May 2019
compose.cli.command.get_client: Docker base_url: http://localhost:2375
...
We can see that a HTTP connection to docker is successfully established. The command then continues its execution and is able to create the container from the docker image.
Here the output of the failing method :
compose.config.config.find: Using configuration files: ./docker-compose.yml
compose.config.environment.__getitem__: The DISPLAY variable is not set. Defaulting to a blank string.
compose.config.environment.__getitem__: The NO_PROXY variable is not set. Defaulting to a blank string.
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1344, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 11, in <module>
load_entry_point('docker-compose==1.21.0', 'console_scripts', 'docker-compose')()
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 71, in main
command()
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 124, in perform_command
project = project_from_options('.', options)
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 41, in project_from_options
compatibility=options.get('--compatibility'),
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 121, in get_project
host=host, environment=environment
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 95, in get_client
version_info = six.iteritems(client.version())
File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 198, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
We can see that the HTTP connection cannot be established.
Why do I need to send my commands in a non-interactive way ? Originally, I want to send those commands using Jenkins (I added a SSH plugin in Jenkins), and I noticed that the docker commands were not working (same output as shown in this post). After a couple of tests, I realised that when Jenkins uses SSH, he sends the commands in a non-interactive way :
- ssh user#remote_IP "commands_to_execute"
This non-interactive way is not an issue for simple commands, but it appears to be an issue for some docker commands which require to be executed in an interactive shell I guess ?
Has someone found a work-around to successfully execute docker commands in a non-interactive shell ? Any help, hint, or alternative solutions will be greatly appreciated, as I tried a lot of things without any success so far..
Did you check if it is docker-engine running on tcp or by unix socket?
Maybe isn't reading right environment vars when you login by ssh u # h "command".
If it's running on unix socket (by default) try...
ssh user # remoteIP "cd /path_to_docker_file;DOCKER_HOST=unix:///var/run/docker.sock docker-compose run -d --name wfirst --user 1000: 1000 container"
or if is running on TCP, you can try ...
ssh user # remoteIP "cd /path_to_docker_file;DOCKER_HOST=tcp://127.0.0.1:2375 docker-compose run -d --name wfirst --user 1000: 1000 container"

PyHive Thrift transport exception: read 0 bytes

I'm trying to connect to Hive server-2 running inside docker container (from outside the container) via python (PyHive 0.5, python 2.7) using DB-API (asynchronous) example
from pyhive import hive
conn = hive.connect(host='172.17.0.2', port='10001', auth='NOSASL')
However, I'm getting following error
Traceback (most recent call last):
File "py_2.py", line 4, in <module>
conn = hive.connect(host='172.17.0.2', port='10001', auth='NOSASL')
File "/home/foodie/anaconda2/lib/python2.7/site-packages/pyhive/hive.py", line 64, in connect
return Connection(*args, **kwargs)
File "/home/foodie/anaconda2/lib/python2.7/site-packages/pyhive/hive.py", line 164, in __init__
response = self._client.OpenSession(open_session_req)
File "/home/foodie/anaconda2/lib/python2.7/site-packages/TCLIService/TCLIService.py", line 187, in OpenSession
return self.recv_OpenSession()
File "/home/foodie/anaconda2/lib/python2.7/site-packages/TCLIService/TCLIService.py", line 199, in recv_OpenSession
(fname, mtype, rseqid) = iprot.readMessageBegin()
File "/home/foodie/anaconda2/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 148, in readMessageBegin
name = self.trans.readAll(sz)
File "/home/foodie/anaconda2/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 60, in readAll
chunk = self.read(sz - have)
File "/home/foodie/anaconda2/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 161, in read
self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
File "/home/foodie/anaconda2/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 132, in read
message='TSocket read 0 bytes')
thrift.transport.TTransport.TTransportException: TSocket read 0 bytes
The docker image that I'm using is this (tag: mysql_corrected).
It runs following services (as outputted by jps command)
992 Master
1810 RunJar
259 DataNode
2611 Jps
584 ResourceManager
1576 RunJar
681 NodeManager
137 NameNode
426 SecondaryNameNode
1690 RunJar
732 HistoryServer
I'm launching the container using
docker run -it -p 8088:8088 -p 8042:8042 -p 4040:4040 -p 18080:18080 -p 10002:10002 -p 10000:10000 -e 3306 -e 9084 -h sandbox -v /home/foodie/docker/w1:/usr/tmp/test rohitbarnwal7/spark:mysql_corrected bash
Furthermore, I perform following steps to launch Hive server inside docker container
Start mysql service: service mysqld start
Switch to directory /usr/local/hive: cd $HIVE_HOME
Launch Hive metastore server: nohup bin/hive --service metastore &
Launch Hive server 2: hive --service hive-server2 (note that thrift-server port is already changed to 10001 in /usr/local/hive/conf/hive-site.xml)
Launch beeline shell: beeline
Connect beeline shell with Hive server-2: !connect jdbc:hive2://localhost:10001/default;transportMode=http;httpPath=cliservice
I've already tried the following things without any luck
Making python 2.7.3 as default python version inside docker container (original default is python 2.6.6, python 2.7.3 is installed inside container but isn't default)
Changing Hive server port to it's' default value: 10000
Trying to connect with Hive server by running same python script inside the container (it still gives the same error)

uwsgi in docker in vagrant in vmware - socket not being created

On my mac I am running Windows Server 2016 in VMware. In there I am running Ubuntu in vagrant/Virtual Box. In there I am trying to run a django app in a docker container with nginx/uwsgi.
uwsgi is failing to start with:
[uWSGI] getting INI configuration from /opt/django/CAPgraph/uwsgi.ini
*** Starting uWSGI 2.0.15 (64bit) on [Thu Aug 17 20:01:23 2017] ***
compiled with version: 6.4.0 20170805 on 17 August 2017 06:10:50
os: Linux-3.13.0-128-generic #177-Ubuntu SMP Tue Aug 8 11:40:23 UTC 2017
nodename: 37db4344b5ae
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 1
current working directory: /
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
chdir() to /opt/django/CAPgraph/
your memory page size is 4096 bytes
detected max file descriptor number: 524288
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
bind(): Operation not permitted [core/socket.c line 230]
In VMware the folder is set for sharing for everyone with write permission. That is mounted in the vagrant VM where it is 777, and in the docker container is it also 777. I can create files in the dir from all 3 places. But it seems uwsgi cannot create the socket.
I tried a short python script as a test from vagrant and that could not create a socket either:
vagrant#vagrant-ubuntu-trusty-64:/vagrant$ python -c "import socket as s; sock = s.socket(s.AF_UNIX); sock.bind('/vagrant/app.sock')"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 1] Operation not permitted
Anyone know how I can resolve this?
UPDATE: I changed the dir to /tmp where I can create a socket with my python script, but still uwsgi fails with the same error.
UPDATE 2: I created the socket in /tmp with my python script, chmod-ed it to 777 and still I get the same error from uwsgi.
Use any other folder other than /vagrant. I usually use /home/vagrant
The problem is that /vagrant is not same as a normal folder. If you execute the command mount | grep vagrant you will find it uses a vboxfs file system, which for some reason doesn't gel well with docker

Unable to use variable substitution in docker-compose

I use a mac to run docker and hence I am dependent on boot2docker or docker-machine . My application running in the container needs the docker host's ip.
I saw that the variable substitution feature of docker-compose is exactly what I need . When I try something like the below it works fine
extra_hosts:
- "somehost:162.242.195.82"
BUT when I try something like:
export DOCKER_HOST="64.88.225.66"
extra_hosts:
- "host:${DOCKER_HOST}"
I get :
$ docker-compose build --no-cache
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "/code/compose/cli/main.py", line 54, in main
File "/code/compose/cli/docopt_command.py", line 23, in sys_dispatch
File "/code/compose/cli/docopt_command.py", line 26, in dispatch
File "/code/compose/cli/main.py", line 169, in perform_command
File "/code/compose/cli/command.py", line 53, in project_from_options
File "/code/compose/cli/command.py", line 89, in get_project
File "/code/compose/cli/command.py", line 70, in get_client
File "/code/compose/cli/docker_client.py", line 28, in docker_client
File "/code/.tox/py27/lib/python2.7/site-packages/docker/client.py", line 58, in __init__
File "/code/.tox/py27/lib/python2.7/site-packages/docker/utils/utils.py", line 362, in parse_host
docker.errors.DockerException: Bind address needs a port: 64.88.225.66
docker-compose returned -1
I'm not sure why it wants a port. It wasn't happy when I gave it a port number too.
So, how can I get my docker host ip to my container? It would be nice if i can pass commands like boot2docker ip or docker-machine ip default in the extra_hosts section or maybe if docker-compose could execute a script to get this value it would be great !
The problem is that DOCKER_HOST is a variable used by docker-compose itself (https://docs.docker.com/compose/reference/overview/#docker-host) and the format is wrong (it needs a tcp:// scheme and a port I believe).
You'll have to choose a different variable to include it in extra_hosts:.
Otherwise you could just pass it in as an environment variable (environment: [DOCKER_HOST=]) and parse the ip address out of it in the container.

Can't connect Celery server to RabbitMQ on localhost

I am using Celery and RabbitMQ as a message queue, where each is encapsulated in it's own Docker image. When they are connected using the --link parameter in Docker, everything works fine. I've had this setup working for some time now. I want to separate them so that they run on different hosts, so I can no longer use the --link parameter for this. I am getting a gaierror: [Errno -2] Name or service not known when I try to connect using AMQP and don't understand why.
The server is simply using the rabbitmq container on DockerHub:
docker run --rm --name=qrabbit -p 5672:5672 rabbitmq
I can telnet to this successfully:
$ telnet 192.168.99.100 5672
Trying 192.168.99.100...
Connected to 192.168.99.100.
Escape character is '^]'.
abc
^D
AMQP Connection closed by foreign host.
$
... so I know the server is running.
My client looks like this:
import os
from logging import getLogger, StreamHandler, DEBUG
from serverlib import QueueServer, CeleryMonitor
from celery import Celery
from argparse import ArgumentParser
log = getLogger('server')
log.addHandler(StreamHandler())
log.setLevel(DEBUG)
broker_service_host = os.environ.get('MESSAGE_QUEUE_SERVICE_SERVICE_HOST')
broker = 'amqp://{0}'.format(broker_service_host)
host = ''
port = 8000
retry = 5
if __name__ == '__main__':
log.info('connecting to {0}, {1}:{2}, retry={3}'.format(broker, host, port, retry))
app = Celery(broker=broker)
monitor = CeleryMonitor(app, retry=retry)
server = QueueServer((host, port), app)
monitor.start()
try:
log.info('listening on {0}:{1}'.format(host, port))
server.serve_forever()
except KeyboardInterrupt:
log.info('shutdown requested')
except BaseException as e:
log.error(e)
finally:
monitor.shutdown()
I'm somewhat certain the external modules (QueueServer and CeleryMonitor) are not part of the problem, as it runs properly when I do the following:
$ docker run --rm --name=qmaster -e "MESSAGE_QUEUE_SERVICE_SERVICE_HOST=localhost" --link qrabbit:rabbit -p 80:8000 render-task-master
connecting to amqp://localhost, :8000, retry=5
listening on :8000
^Cshutdown requested
$
... but not if I do the following (without the --link parameter):
$ docker run --rm --name=qmaster -e "MESSAGE_QUEUE_SERVICE_SERVICE_HOST=localhost" -p 80:8000 render-task-master
connecting to amqp://localhost, :8000, retry=5
listening on :8000
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/home/celery/serverlib/celerymonitor.py", line 68, in run
'*': self.__state.event
File "/usr/local/lib/python2.7/site-packages/celery/events/__init__.py", line 287, in __init__
self.channel = maybe_channel(channel)
File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 1054, in maybe_channel
return channel.default_channel
File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 756, in default_channel
self.connection
File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
self._connection = self._establish_connection()
File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
conn = self.Connection(**opts)
File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 75, in __init__
socket.SOCK_STREAM, SOL_TCP):
gaierror: [Errno -2] Name or service not known
^Cshutdown requested
$
What is the difference between using and not using the --link parameter that might cause this error?
UPDATE:
I've narrowed it down to an error in the monitor class I created:
recv = self.app.events.Receiver(connection, handlers={
'task-received': self.registerTask,
'task-failed': self.retryTask,
'task-succeeded': self.deregisterTask,
# should process all events to have state up to date
'*': self.__state.event
})
When this is called, it sits for a few seconds (timeout?) and then throws an exception. Any idea why this wouldn't like the amqp URL specified as amqp://localhost but everything works correctly when I use the --link parameter?
Here's the whole method that call is in, for additional context:
def run(self):
log.info('run')
self.__state = self.app.events.State()
with self.app.connection() as connection:
log.info('got a connection')
recv = self.app.events.Receiver(connection, handlers={
'task-received': self.registerTask,
'task-failed': self.retryTask,
'task-succeeded': self.deregisterTask,
# should process all events to have state up to date
'*': self.__state.event
})
log.info('received receiver')
# Capture until shutdown requested
while not self.__shutdown:
log.info('main run loop')
try:
recv.capture(limit=None, timeout=1, wakeup=True)
except timeout:
# timeout exception is fired when nothing occurs
# during timeout. Just ignore it.
pass
I found the issue: I had set CELERY_BROKER_URL in the Docker environment in which the container was running and this was causing the backend to attempt to connect to a hostname that did not exist. Once I un-set the variable, everything hooked up properly in my environment.
$ docker inspect server
<... removed ...>
"Env": [
"MESSAGE_QUEUE_SERVICE_SERVICE_HOST=192.168.99.100",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"PYTHON_VERSION=2.7.10",
"PYTHON_PIP_VERSION=7.1.2",
"CELERY_VERSION=3.1.18",
"CELERY_BROKER_URL=amqp://guest#rabbit"
],
<... removed ...>

Resources