I am using Mazerunner docker given by kenny Bastani to integrate neo4j and spark-graphx. I am able to process Movie graph that is given. Now I want to pull my own Twitter graph to Mazerunner docker. Can any one tell me how to pull a new graph to mazerunner docker. Thanks in advance.
-Narendra
There are a few ways to do this. Normally you would be able to mount a volume from your Docker host as the data directory for Neo4j.
Unfortunately there is a defect in Neo4j 2.2 that prevents this. You can find more details here: https://github.com/kbastani/docker-neo4j/issues/4
In order to work around this issue you can copy your graph.db directory from your host machine to the docker-neo4j container.
Run the steps below from the terminal. After you've started HDFS and Mazerunner containers, then start the docker-neo4j container and replace the user specific information with your own, i.e. /Users/User/neo4j-community-2.2.1/data. After the container starts, you will have root access inside the container via the shell. Run the next two commands to copy your mounted host volume's graph.db directory (the database directory you are importing) to the local volume.
[ ~ ]$ docker run -ti -p 7474:7474 -v /Users/User/neo4j-community-2.2.1/data:/opt/data-copy --name graphdb --link mazerunner:mazerunner --link hdfs:hdfs kbastani/docker-neo4j /bin/bash
[ root#f4800317575d:/var/lib/neo4j ]$ cp -r /opt/data-copy/graph.db /opt/data
[ root#f4800317575d:/var/lib/neo4j ]$ bin/neo4j start
Keep in mind that you'll need to copy the container's /opt/data directory back to the host system in order to make sure it is safe.
Hopefully the Neo4j 2.2 issue will be resolved soon.
Related
I cannot start tensorflow with image download from tensorflow
I used docker on windows 10 and for error ouput said this:
WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.
To avoid this, run the container by specifying your user's userid:
$ docker run -u $(id -u):$(id -g) args...
I try search a problem for google... but cannot found, my experience with docker is null
This is a warning specifying that to access/change the files created in the mounted directory you may require sudo and you may not be able to change such files as a non sudo user, since your docker container used sudo permissions while creating them.
A quick search shows that there are many blog references available, check these -
Docker creates files as root in mounted volume
Running a Docker container as a non-root user
Setup Docker for windows using windows subsystem linux
https://jtreminio.com/blog/running-docker-containers-as-current-host-user/
https://medium.com/better-programming/running-a-container-with-a-non-root-user-e35830d1f42a
https://docs.docker.com/install/linux/linux-postinstall/
I followed the standard Odoo container instructions on Docker to start the required postgres and odoo servers, and tried to pass host directories as persistent data storage for both as indicated in those instructions:
sudo mkdir /tmp/postgres /tmp/odoo
sudo docker run -d -v /tmp/postgres:/var/lib/postgresql/data/pgdata -e POSTGRES_USER=odoo -e POSTGRES_PASSWORD=odoo -e POSTGRES_DB=postgres --name db postgres:10
sudo docker run -v /tmp/odoo:/var/lib/odoo -p 8069:8069 --name odoo --link db:db -t odoo
The Odoo container shows messages that it starts up fine, but when I point my web browser at http://localhost:8069 I get no response from the server. By contrast, if I omit the -v argument from the Odoo docker run command, my web browser connects to the Odoo server fine, and everything works great.
I searched and see other people also struggling with getting the details of persistent data volumes working, e.g. Odoo development on Docker, Encountered errors while bringing up the project
This seems like a significant gap in Docker's standard use-case that users need better info on how to debug:
How to debug why the host volume mounting doesn't work for the odoo container, whereas it clearly does work for the postgres container? I'm not getting any insight from the log messages.
In particular, how to debug whether the container requires the host data volume to be pre-configured in some specific way, in order to work? For example, the fact that I can get the container to work without the -v option seems like it ought to be helpful, but also rather opaque. How can I use that success to inspect what those requirements actually are?
Docker is supposed to help you get a useful service running without needing to know the guts of its internals, e.g. how to set up its internal data directory. Mounting a persistent data volume from the host is a key part of that, e.g. so that users can snapshot, backup and restore their data using tools they already know.
I figured out some good debugging methods that both solved this problem and seem generally useful for figuring out Docker persistent data volume issues.
Test 1: can the container work with an empty Docker volume?
This is a really easy test: just create a new Docker volume and pass that in your -v argument (instead of a host directory absolute path):
sudo docker volume create hello
sudo docker run -v hello:/var/lib/odoo -p 8069:8069 --name odoo --link db:db -t odoo
The odoo container immediately worked successfully this way (i.e. my web browswer was able to connect to the Odoo server). This showed that it could work fine with an (initially) empty data directory. The obvious question then is why it didn't work with an empty host-directory volume. I had read that Docker containers can be persnickety about UID/GID ownership, so my next question was how do I figure out what it expects.
Test 2: inspect the running container's file system
I used docker exec to get an interactive bash shell in the running container:
sudo docker exec -ti odoo bash
Inside this shell I then looked at the data directory ownership, to get numeric UID and GID values:
ls -dn /var/lib/odoo
This showed me the UID/GID values were 101:101. (You can exit from this shell by just typing Control-D)
Test 3: re-run container with matching host-directory UID:GID
I then changed the ownership of my host directory to 101:101 and re-ran the odoo container with my host-directory mount:
sudo chown 101:101 /tmp/odoo
sudo docker stop odoo
sudo docker rm odoo
sudo docker run -v /tmp/odoo:/var/lib/odoo -p 8069:8069 --name odoo --link db:db -t odoo
Success! Finally the odoo container worked properly with a host-directory mount. While it's annoying the Odoo docker docs don't mention anything about this, it's easy to debug if you know how to use these basic tests.
After creating a docker image as follow:
PS> docker run -d -p 1433:1433 --name sql1 -v sql1data:C:/sqldata -e sa_password=MyPass123 -e ACCEPT_EULA=Y microsoft/mssql-server-windows-developer
I stopped my container and copied a backup file into my volume:
PS> docker cp .\DataBase.bak sql1:C:\data
After that I can no longer start my container, the error message is as follows:
Error response from daemon: container 5fe22f4ac151d7fc42541b9ad2142206c67b43579ec6814209287dbd786287dc encountered an error during Start: failure in a Windows system call: Le système de calcul s’est fermé de façon inattendue. (0xc0370106)
Error: failed to start containers: sql1
I can start and stop any other container, the problem occurs only after copying the file into the volume.
I'm using windows containers
my docker version is 18.06.0-ce-win72 (19098)
The only workaround i found is to not copy any files into my container volume.
Seems like it's because of files ownership and permissions. When you make a backup with copying the files and use those files for a new Docker Container, MYSQL Daemon in your Docker Container finds that the ownership and permissions of it's files are changed.
I think the best thing to do is to create a raw MySQL Docker Container and see who is the owner of your backup files in that container (i guess it must be 1000). then change the owner of your backup files to that user id and then create a Container with Volumes mapped to your backup files.
I have a dockerized web application that I'm running in a HA setup. I have a cron setup that runs dockup every midnight to backup my important information stored on other containers. Now I would like to backup and aggregate my logs from my web application too. Problem is, how do I that? If I use the VOLUME key in Dockerfile to expose /logs to the host machine, there would be a collision because there would be two /logs directories on the dockup container?
I have checked dockup. It does not have a /logs directory. Seems it uses /var/logs for log output.
$ docker run -it --name dockup borja/dockup bash
Otherwise, yes it would be a problem because the volume will be mounted under the mentioned name and also the current container processes will log to the folder. Not good.
Use a logging container like fluentd. In this tutorial it also offers writing to S3 buckets like dockup. Tutorial can be founder here.
Tweak your container, e.g. with symbolic links to log or relay the log to a different volume.
Access log not through containers but native docker and copy it to S3 yourself or running dockup on your local mounted log file.
$ docker logs container/name > logfile.log
$ docker run --rm \
--env-file env.txt \
-v $(pwd)/logfile.log:/customlogs/logfile.txt \
--name dockup borja/dockup
Now you can take the folder /customlogs/ as your backup path inside the env.txt.
I use this method below to port data out of one container.
docker run --volumes-from <data container> ubuntu tar -cO <volume path> | gzip -c > volume.tgz
But there is one problem with it is every time it performs a backup, there will be a zombie container left. What is the good way to get that id and remove the zombie container afterward.
Thanks
Apparently, you just want to be able to export volume data. To do that, you just need to start your initial container with a volume pointing to a directory on the host with the -v option. You can tar on the host without creating a container for it. Your current tactic seems a bit over-engineered ;)
The easy way to remove the container after executing the command, is to use the option --rm, from here
However, if you feel that the container you are creating will have data that you will need to
1. update in real time
2. access after the container has been created
then you may also mount a host directory as a container volume and access the contents of that directory from the host.
If you start a container using the -volume option, you can also call reference the directory created on this host
$ docker run -v /volume_directory ubuntu
$ container=$(docker ps -n=1 -q)
$ docker inspect -f '{{.Volumes}}' $container