I'm trying to setup Apache Superset for Clickhouse.
My understanding so far is that I need to install SQLAlchemy for Clickhouse
https://github.com/xzkostyan/clickhouse-sqlalchemy
I'm in Ubuntu 16.04 LTS, and using the Docker vanilla version of Clickhouse and of Superset:
https://store.docker.com/community/images/yandex/clickhouse-server
https://hub.docker.com/r/amancevice/superset/
without special settings
Any idea how I can bridge the two docker containers with clickhouse-sqlalchemy ?
Where and how in that case to install that?
(if you have sample command line that I can reuse that will be great)
You don't need to bridge them: what you want is a superset server (that you happen to be running via docker) to connect to a clickhouse database (that you also happen to be running via docker).
You also shouldn't need to install SQLAlchemy for Clickhouse: looking at the dockerfile at https://hub.docker.com/r/amancevice/superset/~/dockerfile/ that image has already sqlalchemy-clickhouse installed for you.
Your steps should be as follow:
When you docker run --detach --name superset [options] amancevice/superset you should have your superset instance running at http://localhost:8088/
Similarly, when you run $ docker run -d --name some-clickhouse-server --ulimit nofile=262144:262144 -v /path/to/your/config.xml:/etc/clickhouse-server/config.xml yandex/clickhouse-server you should end-up with a clickhouse instance that you can access via SQLAlchemy at something like clickhouse://default:#some-clickhouse-server/test
You'd need to modify that connection URI based on your config.xml - and you should be able to double-check that it works by connecting to it in your python console.
You should then be able to connect superset to your clickhouse db in the same way you'd connect to any other DB: by navigating into Superset's menu > Sources > Databases > [new]
Consider using already prepared and configured docker-compose.yml which included in Apache Superset (see https://github.com/apache/superset/blob/master/docker-compose.yml).
To work with Clickhouse should be installed sqlalchemy driver. There are two ones:
clickhouse-sqlalchemy by xzkostyan
sqlalchemy-clickhouse by cloudflare.
I recommend using clickhouse-sqlalchemy because it is actually supported and evolute, it supports both available protocols to interact with ClickHouse - HTTP and TCP (native protocol).
Let's connect to one of the public ClickHouse:
either Demo Yandex CH
docker run -it --rm yandex/clickhouse-client:latest \
--host gh-api.clickhouse.tech --user explorer -s
or Demo Altinity.Cloud CH
docker run -it --rm yandex/clickhouse-client:latest \
--host github.demo.trial.altinity.cloud -s --user demo --password demo
download source code from repo https://github.com/apache/superset
execute the commands
cd superset-master
docker-compose up
# open the new terminal
docker-compose exec superset bash /app/docker/docker-init.sh
docker-compose exec superset pip install clickhouse-sqlalchemy
docker-compose restart
wait for containers to be started and the web app to be built (see the console output, webpack should finish its work)
browse URL http://localhost:8088 (use credentials admin / admin)
add the database using one of the connection string:
# connection string for Demo Yandex ClickHouse
clickhouse+native://explorer#gh-api.clickhouse.tech/default?secure=true
# connection string for Demo Altinity.Cloud CH
clickhouse+native://demo:demo#github.demo.trial.altinity.cloud/default?secure=true
See also https://stackoverflow.com/a/66006784/303298.
Related
I have been able to successfully run apache ignite with custom config using the command
docker run -it --net=host -v "pathToLocalDirectory"/config:/opt/ignite/apache-ignite/config -e "CONFIG_URI=file:///opt/ignite/apache-ignite/config/default-config.xml" apacheignite/ignite.
But when I run my java project in IntelliJ I get the message
"IP finder returned empty addresses list. Please check IP finder configuration and make sure multicast works on your network...".
Note: the java client project works if I run the ignite server using windows batch file.
Also, I have published 47500 port as well. the result is the same.
try running it using docker -run -it --net=host (don't mount the volumes).
If that doesn't work, it means that either something is incorrect w/your docker setup OR you are configuring discovery differently for clients and servers.
check the IP addresses listed in your client discovery section.
ssh into the container and check what is actually mounted?
run docker exec -it container-name /bin/bash
check: /opt/ignite/apache-ignite/config/default-config.xml is there and contains the correct discovery info.
Check that the ignite log (located in /opt/ignite/apache-ignite/work/log/) specifies that the correct config is being used.
It will have a line like so: [INFO][main][IgniteKernal] Config URL: file:/opt/ignite/apache-ignite/config/default-config.xml
If you don't see the mounted config file try mounting more simply.
docker run -d -v /local/dir/config.xml:/config-file.xml -e CONFIG_URI=/config-file.xml apacheignite/ignite
more info:
https://apacheignite.readme.io/docs/docker-deployment
https://apacheignite.readme.io/docs/tcpip-discovery
I have installed grafana via docker.
Is it possible to export and run grafana-cli on my host?
If you meant running Grafana with some plugins installed, you can do it by passing a list of plugin names to a variable called GF_INSTALL_PLUGINS.
sudo docker run -d -p 3000:3000 -e "GF_INSTALL_PLUGINS=gridprotectionalliance-openhistorian-datasource,gridprotectionalliance-osisoftpi-datasource" grafana/grafana
I did this on Grafana 4.x
Installing plugins for Grafana 3 "or above"
For a full automatic setup of your Grafana install with the plugins you want I would follow Ricardo's suggestion. Its much better if you can configure your entire container as wanted in a single hit like that.
However if you are just playing with the plugins and want to install some manually, then you can access a shell on the running docker instance from the host.
host:~$ docker exec -it grafana /bin/bash
... assuming you named the docker container "grafana" otherwise you will need to substitute the given container name. The shell prompt that returns will allow you to run the standard
root#3e04b4578ebe:/# grafana-cli plugins install ....
Be warned that it may tell you to run service grafana-server restart afterwards. In my experience that didn't work (Not sure it runs as a traditional service in the container). However if you exit the container, and restart the container from the host...
host:~$ docker restart grafana
That should restart the grafana service and your new plugins should be in place.
Grafana running in docker container
Docker installed on Windows 10
Test: command to display grafana-cli help
c:\>docker exec -it grafana grafana-cli --help
Tested with a version: Version 6.4.4 November 6, 2019
I've installed the tensorflow docker container on an ubuntu machine. The tensorflow docker setup instructions specify:
docker run -it b.gcr.io/tensorflow/tensorflow
This puts me into the docker container terminal, and I can run python and execute the Hello World example. I can also manually run .\run_jupyter.sh to start the jupyter notebook. However, I can't reach the notebook from host.
How do I start the jupyter notebook such that I can use the notebook from the host machine? Ideally I would like to use docker to launch the container and start jupyter in a single command.
For a Linux host Robert Graves answer will work, but for Mac OS X or Windows there is more to be done because docker runs in a virtual machine.
So to begin launch the docker shell (or any shell if you are using Linux) and run the following command to launch a new TensorFlow container:
docker run -p 8888:8888 -p 6006:6006 b.gcr.io/tensorflow/tensorflow ./run_jupyter.sh
Then for Mac OS X and Windows you need to do the following only once:
Open VirtualBox
Click on the docker vm (mine was automatically named "default")
Open the settings by clicking settings
In the network settings open the port forwarding dialog
Click the + symbol to add another port and connect a port from your mac to the VM by filling in the dialog as shown below. In this example I chose port 8810 because I run other notebooks using port 8888.
then open a browser and connect to http://localhost:8810 (or whichever port you set in the host port section
Make your fancy pants machine learning app!
My simple yet efficient workflow:
TL;DR version:
Open Docker Quickstart Terminal. If it is already open, run $ cd
Run this once: $ docker run -it -p 8888:8888 -p 6006:6006 -v /$(pwd)/tensorflow:/notebooks --name tf b.gcr.io/tensorflow/tensorflow
To start every time: $ docker start -i tf
If you are not on windows, you should probably change /$(pwd) to $(pwd)
You will get an empty folder named tensorflow in your home directory for use as a persistent storage of project files such as Ipython Notebooks and datasets.
Explanation:
cd for making sure you are in your home directory.
params:
-it stands for interactive, so you can interact with the container in the terminal environment.
-v host_folder:container_folder enables sharing a folder between the host and the container. The host folder should be inside your home directory. /$(pwd) translates to //c/Users/YOUR_USER_DIR in Windows 10. This folder is seen as notebooks directory in the container which is used by Ipython/Jupyter Notebook.
--name tf assigns the name tf to the container.
-p 8888:8888 -p 6006:6006 mapping ports of container to host, first pair for Jupyter notebook, the second one for Tensorboard
-i stands for interactive
Running TensorFlow on the cloud
After further reading of docker documentation I have a solution that works for me:
docker run -p 8888:8888 -p 6006:6006 b.gcr.io/tensorflow/tensorflow ./run_jupyter.sh
The -p 8888:8888 and -p 6006:6006 expose the container ports to the host on the same port number. If you just use -p 8888, a random port on the host will be assigned.
The ./run_jupyter.sh tells docker what to execute within the container.
With this command, I can use a browser on the host machine to connect to http://localhost:8888/ and access the jupyter notebook.
UPDATE:
After wrestling with docker on windows I switched back to a Ubuntu machine with docker. My notebook was being erased between docker sessions which makes sense after reading more docker documentation. Here is an updated command which also mounts a host directory within the container and starts jupyter pointing to that mounted directory. Now my notebook is saved on the host and will be available next time start up tensorflow.
docker run -p 8888:8888 -p 6006:6006 -v /home/rob/notebook:/notebook b.gcr.io/tensorflow/tensorflow sh -c "jupyter notebook /notebook"
Jupyter now has a ready to run Docker image for TensorFlow:
docker run -d -v $(pwd):/home/jovyan/work -p 8888:8888 jupyter/tensorflow-notebook
These steps worked for me if you are a total docker noob using a windows machine.
Versions: Windows 8.1, docker 1.10.3, tensorflow r0.7
Run Docker Quickstart Terminal
After it is loaded, note the ip address. If you can't find it use this docker-machine ip and make a note. Lets call it 'ip address'. Will look something like this: 192.168.99.104 (I made up this ip address)
Paste this command on the docker terminal:
docker run -p 8888:8888 -p 6006:6006 b.gcr.io/tensorflow/tensorflow.
If you are running this for the first time, it will download and install the image on this light weight vm. Then it should say 'The Jupyter notebook is running at ....' -> This is a good sign!
Open your browser at: <your ip address (see above)>:8888. Eg. 192.168.99.104:8888/
Hopefully you can see your ipython files.
To get this to run under hyper-v. Perform the following steps:
1) Create a docker virtual machine using https://blogs.msdn.microsoft.com/scicoria/2014/10/09/getting-docker-running-on-hyper-v-8-1-2012-r2/ this will get you a working docker container. You can connect to it via the console or via ssh. I'd put at least 8gb of memory since I'm sure this will use a lot of memory.
2) run "ifconfig" to determine the IP address of the Docker VM
3) On the docker shell prompt type:
docker run -p 8888:8888 -p 6006:6006 -it b.gcr.io/tensorflow/tensorflow
4) Connect to the Jupyter Workbench using http:/[ifconfig address]:8888/
To tidy up the things a little bit, I want to give some additional explanations because I also suffered a lot setting up docker with tensorflow. For this I refer to this video which is unfortunately not selfexplanatory in all cases.
I assume you already installed docker. The really interesting general part of the video starts at minute 0:44 where he finally started docker. Until there he only downloads the tensorflow repo into the folder, that he then mounts into the container. You can of course put anything else into the container and access it later in the docker VM.
First he runs the long docker command docker run –dit -v /c/Users/Jay/:/media/disk –p 8000 –p 8888 –p 6006 b.gcr.io/tensorflow/tensorflow. The “run” command starts containers. In this case it starts the container “b.gcr.io/tensorflow/tensorflow”, whose address is provided within the tensorflow docker installation tutorial. The container will be downloaded by docker if not already locally available.
Then he gives two additional kinds of arguments: He mounts a folder of the hostsystem at the given path to the container. DO NOT forget to give the partition in the beginning (eg. "/c/").
Additionally he declares ports being available later from the host machine with the params -p.
From all this command you get back the [CONTAINER_ID] of this container execution!
You can always see the currently running containers by running “docker ps” in the docker console. Your container created above should appear in this list with the same id.
Next Step: With your container running, you now want to execute something in it. In our case jupyter notebook or tensorflow or whatever: To do this you make docker execute the bash on the newly created container: docker exec –ti [CONTAINER_ID] bash. This command now starts a bash shell on your container. You see this because the “$” now changed to root#[CONTAINER_ID]:. From here is no way back. If you want to get back to the docker terminal, you have to start another fresh docker console like he is doing in minute 1:10. Now with a bash shell running in the container you can do whatever you want and execute Jupiter or tensorflow or whatever. The folder of the host system, you gave in the run command, should be available now under “/media/disk”.
Last step accessing the VM output. It still did not want to work out for me and I could not access my notebook. You still have to find the correct IP and Port to access the launched notebook, tensorboard session or whatever. First find out the main IP by using docker-machine –ls. In this list you get the URL. (If it is your only container it is called default.) You can leave away the port given here. Then from docker ps you get the list of forwarded ports. When there is written 0.0.0.32776->6006/tcp in the list, you can access it from the hostmachine by using the port given in the first place (Awkyard). So in my case the executed tensorboard in the container said “launched on port 6006”. Then from my hostmachine I needed to enter http://192.168.99.100:32776/ to access it.
-> And that’s it! It ran for me like this!
It gives you the terminal prompt:
FOR /f "tokens=*" %i IN ('docker-machine env --shell cmd vdocker') DO %i
docker run -it tensorflow/tensorflow:r0.9-devel
or
FOR /f "tokens=*" %i IN ('docker-machine env --shell cmd vdocker') DO %i
docker run -it b.gcr.io/tensorflow/tensorflow:latest-devel
You should have 'vdocker' or change vdocker to 'default'.
For some reason I ran into one additional problem that I needed to overcome beyond the examples provided, using the --ip flag:
nvidia-docker run --rm \
-p 8888:8888 -p 6006:6006 \
-v `pwd`:/root \
-it tensorflow/tensorflow:latest-devel-gpu-py3 sh -c "jupyter notebook --ip 0.0.0.0 ."
And then I can access via http://localhost:8888 from my machine. In some ways this makes sense; within the container you bind to 0.0.0.0 which represents all available addresses. But whether I need to do this seems to vary (e.g I've started notebooks using jupyter/scipy-notebook without having to do this).
In any case, the above command works for me, might be of use to others.
As an alternative to the official TensorFlow image, you can also use the ML Workspace Docker image. The ML Workspace is an open-source web IDE that combines Jupyter, VS Code, TensorFlow, and many other tools & libraries into one convenient Docker image. Deploying a single workspace instance is as simple as:
docker run -p 8080:8080 mltooling/ml-workspace:latest
All tools are accessible from the same port and integrated into the Jupyter UI. You can find the documentation here.
I'm trying to build a docker image of web2py on top of ubuntu. Given the docker file
#######################
# Web2py installation #
#######################
# Set the base image for this installation
FROM ubuntu
# File Author/ Mainteainer
MAINTAINER sandilya28
#Update the repository sources list
RUN apt-get update --assume-yes
########### BEGIN INSTALLATION #############
## Install Git first
RUN apt-get install git-core --assume-yes && \
cd /home/ && \
git clone --recursive https://github.com/web2py/web2py.git
## Install Python
RUN sudo apt-get install python --assume-yes
########## END INSTALLATION ################
# Expose the default port
EXPOSE 8000
WORKDIR /home/
By building an image using the above Dockerfile
docker build -t sandilya28/web2py .
Then by building a container using the above image
docker run --name my_web2py -p 8000:8000 -it sandilya28/web2py bash
The ip address of the host is
192.168.59.103
which can be found by using boot2docker ip
After creating the image I'm starting the web2py sever using
python web2py/web2py.py
and I'm trying to access the web2py GUI from 192.168.59.103:8000 but it is showing the page is not available.
How to access the GUI of web2py from the browser.
Creating a docker that runs the development webserver will leave you with a very slow solution as the webserver is single threaded and will also serve all static files. It's meant for development.
As you don't use https it will also disable the web2py admin interface: that's only available over http if you access it from localhost.
That being said, you can get your solution up and running by starting web2py with:
python web2py.py --nogui -a admin -i 0.0.0.0
All options are important as web2py needs to start the server without asking any questions and it needs to bind to external netwerk interface address.
When you want to use a production ready docker to run web2py you would need some additional components in your docker; nginx, uwsgi and supervisord would make it a lot faster and give you the options to enable https. Note: for bigger projects you would probably need python binding for MySql or PostgreSQL and a separate docker with the database.
An production example, without fancy DB support, can be found here:
https://github.com/acidjunk/docker-web2py
It can be installed from the docker hub with:
docker pulll acidjunk/web2py
Make sure to read the instructions as you'll need a web2py app; that will be mounted in the container. If you just want to start a web2py server to fiddle around with the example or welcome app you can use:
docker pull thehipbot/web2py
Start it with:
docker run -p 443:443 -p 80:80 thehipbot/web2py
Then fire up a browser to
https://192.168.59.103
Look at the example app I created on github:
Main features:
Stripped down version of the base w2p app
Dev mode friendly (admin console)
Served by Gunicorn (optimized for use in Docker containers)
Naked URL i.e. http://localhost:8080, no extra URL paths
Dockerfile + k8s
Hope this helps.
I've been following several different tutorials as well as the official one however whenever I try to install PostgreSQL within a container I get the following message afterwards
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I've looked through several questions here on SO and throughout the internet but no luck.
The problem is that the your application/project is trying to access the postgres socket file in the HOST machine (not docker container).
To solve it one would either have to explicitly ask for an tcp/ip connection while using the -p flag to set up a port for the postgres container, or share the unix socket with the HOST maching using the -v flag.
:NOTE:
Using the -v or --volume= flag means you are sharing some space between the HOST machine and the docker container. That means that if you have postgres installed on your host machine and its running you will probably run into issues.
Below I demonstrate how to run a postgres container that is both accessible from tcp/ip and unix socket. Also I am naming the container as postgres.
docker run -p 5432:5432 -v /var/run/postgresql:/var/run/postgresql -d --name postgres postgres
There are other solutions, but I find this one the most suitable. Finally if the application/project that needs access is also a container, it is better to just link them.
By default psql is trying to connect to server using UNIX socket. That's why we see /var/run/postgresql/.s.PGSQL.5432- a location of UNIX-socket descriptor.
If you run postgresql-server in docker with port binding so you have to tell psql to use TCP-socket. Just add host param (--host or -h):
psql -h localhost [any other params]
UPD. Or share UNIX socket descriptor with host (where psql will be started) as was shown in main answer. But I prefer to use TCP socket as easy managed approach.
FROM postgres:9.6
RUN apt-get update && apt-get install -q -y postgresql-9.6 postgresql-client-9.6 postgresql-contrib-9.6 postgresql-client-common postgresql-common
RUN echo postgres:postgres | chpasswd
RUN pg_createcluster 9.6 main --start
RUN /etc/init.d/postgresql start
RUN su -c "psql -c \"ALTER USER postgres PASSWORD 'postgres';\"" postgres
Here are instructions for fixing that error that should also work for your docker container: PostgreSQL error 'Could not connect to server: No such file or directory'
If that doesn't work for any reason, there are many of off-the-shelf postgresql docker containers you can look at for reference on the Docker Index: https://index.docker.io/search?q=postgresql
Many of the containers are built from trusted repos on github. So if you find one that seems like it meets your needs, you can review the source.
The Flynn project has also included a postgresql appliance that might be worth checking out: https://github.com/flynn/flynn-postgres
Run the below command to create a new container with PSQL running it it, which can be accessed from other containers/applications.
docker run --name postgresql-container -p 5432:5432 -e POSTGRES_PASSWORD=somePassword -d postgres
Now, export the connection-string or DB credentials from ur .env and use it in the application.
Refernce: detailed installion and running