I'm using a local implementation of TinkerPop with docker image tinkerpop/gremlin-server:3.4.1 in order to locally interact with the graph database in nodeJs.
I need to set the IDManager to ANY so that it can accept string values for custom vertex IDs (Right now it's only working with numeric types).
I know that I need to set the configuration of TinkerGraph gremlin.tinkergraph.vertexIdManager, but I'm not sure how in my docker-compose file, I can have it initialize with the correct configurations.
http://tinkerpop.apache.org/docs/current/reference/#_configuration_4
Anyone know how to do this?
Thanks
When you launch the container using a command such as
docker run --rm -p 8182:8182 tinkerpop/gremlin-server
You can optionally pass in the path to a YAML configuration file. Which would look like this:
docker run --rm -p 8182:8182 tinkerpop/gremlin-server conf/gremlin-server.yaml
That file is located inside the container in the opt/gremlin-server/conf folder. One option is to docker exec into the running container and edit the YAML and properties files and then create a new image from the modified one. You could also use docker cp to replace those files. While this will work, the downside is that you will have to do this each time you pull a newer version of the Gremlin Server image.
What you can try instead is to mount a local file system volume as part the docker command containing a YAML file that points to your own properties file in which you can add the ID manager lines:
gremlin.tinkergraph.vertexIdManager=ANY
gremlin.tinkergraph.edgeIdManager=ANY
That docker command will be something like this:
docker run --rm -p 8182:8182 -v $(pwd):/opt/gremlin-server/conf tinkerpop/gremlin-server conf/myfile.yaml
However, this may not work as the Gremlin Server startup script runs a sed command that creates a modified version of the YAML file and that requires write permissions to your local disk (this can be worked around as explained below). As a side note, that is done to fix up issues with IP addresses. The file permissions and user permissions need to be such that that sed command is able to run.
To work around docker now needing to have the ability to edit files on your local disk (rather than in the container's own ephemeral storage), at least on Linux systems, you can try using the --user parameter as shown below.
docker run --rm -p 8182:8182 --user $(id -u):$(id -g) -v $(pwd):/opt/gremlin-server/conf tinkerpop/gremlin-server conf/myfile.yaml
Note that for this to work, any files that Gremlin Server expects to read from the conf folder as part of its bootstrap process will now need to exist on your local disk, as we have re-mapped where the conf folder is. The files read during startup include the log4j-server.properties file and any scripts and properties files referenced by your YAML file. You can copy these files from the container itself (via docker exec or docker cp) or the Apache TinkerPop Github repo.
Related
what I want to accomplish
I want to use the functions written in the .vimrc placed on the host side within Docker.
what I did
Put the .vimrc file in the /home/akihiro directory on the host side.
When using the docker run command, mount the /home/akihiro directory on the host side and run the python file in Docker with Vim.
akihiro#akihiro-thinkpad-x1-carbon-5th:~$ docker run --rm -it -v /home/akihiro:/home --name test cnn_study:latest
As a result, the settings written in the .vimrc file did not work.
Next, I started a new container without mounting.
Created /home/akihiro directory in the container.
I left the container.
I copied only the /home/akihiro/.vimrc file on the host side into the container, and re-entered the container.
docker cp ./.vimrc 52b28f1ffea8:/home/akihiro
Started up a Python file using Vim.
As a result, the settings written in the .vimrc file did not work.
What you are doing is mapping the complete /home or /home/akihiro directory on the Container.
You can't do that for 2 reasons:
There are more files in that directory. What do you expect happens with them?
It's not like a OR-function is done on files in both folder.
Mapping the volume complete replaces the internal folder
My gut feeling says that mapping the direcory comes too late in the process.
The directory is already there in the Container and therefore cannot be overwritten.
(At least that's how I understand the strange effects I get with mapping sometimes)
What you should do is only map the file:
$ docker run --rm -it -v /home/akihiro/.vimrc:/home/akihiro/.vimrc --name test cnn_study:latest
I do the same with .bashrc (to set the prompt to the name of the Container)
I am trying to get a Matrix Synapse server running on my Synology NAS through docker.
When I run the generate command to get the intial homeserver.yaml, it does get generated :
$ sudo docker run -it --rm --mount type=volume,src=synapse-config,dst=/data -e SYNAPSE_SERVER_NAME=my.matrix.host -e SYNAPSE_REPORT_STATS=yes matrixdotorg/synapse:latest generate
Creating log config /data/my.matrix.host.log.config
Generating config file /data/homeserver.yaml
Generating signing key file /data/my.matrix.host.signing.key
A config file has been generated in '/data/homeserver.yaml' for server name 'my.matrix.host'. Please review this file and customise it to your needs.
So the file is generated in /volume1/#docker/volumes/synapse-config/_data but I don't know how to actually access this folder.
How can I specify within the docker generate command where I would like the /data data to reside ? I have created a folder (/volume1/synapse/data) to that effect but I don't know how to make sure the output of the docker generate actually goes there.
With your --mount you sepcify what should be mapped to /data within the container.
Use e.g. -v /volume1/synapse/data:/data instead to map it to some directory on the host filesystem.
The image you are trying to use supports this environment variable, according to the documentation:
`SYNAPSE_CONFIG_PATH`: path to the file to be generated. Defaults to <SYNAPSE_CONFIG_DIR>/homeserver.yaml
To inject this ENV, add -e SYNAPSE_SERVER_NAME=<desired-path> in your docker run command
I use mlflow in a docker environment as described in this example and I start my runs with mlflow run ..
I get output like this
2019/07/17 16:08:16 INFO mlflow.projects: === Building docker image mlflow-myproject-ab8e0e4 ===
2019/07/17 16:08:18 INFO mlflow.projects: === Created directory /var/folders/93/xt2vz36s7jd1fh9bkhkk9sgc0000gn/T/tmp1lxyqqw9 for downloading remote URIs passed to arguments of type 'path' ===
2019/07/17 16:08:18 INFO mlflow.projects: === Running command 'docker run
--rm -v /Users/foo/bar/mlruns:/mlflow/tmp/mlruns -e
MLFLOW_RUN_ID=ef21de61d8a6436b97b643e5cee64ae1 -e MLFLOW_TRACKING_URI=file:///mlflow/tmp/mlruns -e MLFLOW_EXPERIMENT_ID=0 mlflow-myproject-ab8e0e4 python train.py' in run with ID 'ef21de61d8a6436b97b643e5cee64ae1' ===
I would like to mount a docker volume named my_docker_volume to the container
at
the path /data. So instead of the docker run shown above, I would like to
use
docker run --rm --mount source=my_docker_volume,target=/data -v /Users/foo/bar/mlruns:/mlflow/tmp/mlruns -e MLFLOW_RUN_ID=ef21de61d8a6436b97b643e5cee64ae1 -e MLFLOW_TRACKING_URI=file:///mlflow/tmp/mlruns -e MLFLOW_EXPERIMENT_ID=0 mlflow-myproject-ab8e0e4 python train.py
I see that I could in principle run it once without mounted volume and then
copy the docker run ... and add --mount source=my_volume,target=/data but
I'd rather use something like
mlflow run --mount source=my_docker_volume,target=/data .
but this obviously doesn't work because --mount is not a parameter for
mlflow run.
What's the recommened way of mounting a docker volume then?
A similar issue has been brought up on the mlflow issue tracker, see "Access large data from within a Docker environment". An excerpt from it says:
However, MLFlow Docker environments currently only have access to data baked into the repository or image or must download a large dataset for each run.
...
A potential solution is to enable the user to mount a volume (e.g. local directory containing the data) into the Docker container.
Looks like this is feature others would benefit from too. Best course of action here would be to contribute support for mounts, or keep track of the issue until someone else implements it.
Why do you need to mount /data folder in the first place? There's another issue, a PR containing a fix related to storing artifacts in a custom location on host machine, could it be something you're looking for?
Finally, to avoid the above problem and facilitate volume mounting, I now run my experiments using three interacting docker containers. One that runs the machine learning code, one that runs an mlflow server and one that runs a postgresql server. I closely followed this walk-through article to set things up. It works nicely and docker-compose makes volume mounting easy. Metrics, parameters and meta data are stored in a database that is mounted to a local persistent volume. Artifacts are logged in the directory /mlflow or if you prefer in a docker volume.
Note: There's a typo in the cited walk-through article
In docker-compose.yml it shouldn't be
volumes:
- ./postgres-store:/var/lib/postgresql/data
which would bind a local folder named postgres-store.
Instead, to mount the docker volume postgres_store, you should use
volumes:
- postgres-store:/var/lib/postgresql/data
I am using Docker to deploy my ASP.NET Core Web API microservices, and am looking at the options for injecting configuration into each container. The standard way of using an appsettings.json file in the application root directory is not ideal, because as far as I can see, that means building the file into my docker images, which would then limit which environment the image could run in.
I want to build an image once which can they be provided configuration at runtime and rolled through the dev, test UAT and into Production without creating an image for each environment.
Options seem to be:
Providing config via environment variables. Seems a bit tedious.
Somehow mapping a path in the container to a standard location on the host server where appsettings.json sits, and getting the service to pick this up (how?)
May be possible to provide values on the docker run command line?
Does anyone have experience with this? Could you provide code samples/directions, particularly on option 2) which seems the best at the moment?
It's possible to create data volumes in the docker image/container. And also mount a host directory into a container. The host directory will then by accessible inside the container.
Adding a data volume
You can add a data volume to a container using the -v flag with the docker create and docker run command.
$ docker run -d -P --name web -v /webapp training/webapp python app.py
This will create a new volume inside a container at /webapp.
Mount a host directory as a data volume
In addition to creating a volume using the -v flag you can also mount a directory from your Docker engine’s host into a container.
$ docker run -d -P --name web -v /src/webapp:/webapp training/webapp python app.py
This command mounts the host directory, /src/webapp, into the container at /webapp.
Refer to the Docker Data Volumes
We are using other packaging system for now (not docker itself), but still have same issue - package can be deployed in any environment.
So, the way we are doing it now:
Use External configuration management system to hold and manage configuration per environment
Inject to our package the basic environment variables to hold the configuration management system connection details
This way we are not only allowing the package to run in almost any "known" environment, but also run-time configuration management.
When you are running docker, you can use environment variable options of the run command:
$ docker run -e "deep=purple" ...
I am new to docker. I want to run tinyproxy within docker. Here is the image I used to create a docker container: "https://hub.docker.com/r/dtgilles/tinyproxy/".
For some unknown reason, when I mount the log file to the host machine, I can see the .conf file, but I can't see log file and the proxy server seems doesn't work.
Here is the command I tried:
docker run -v $(pwd):/logs -p 8888:8888 -d --name tiny
dtgilles/tinyproxy
If I didn't mount the file, then every time when run a container, I need to change its config file inside container.
Does anyone has any ideas about saving the changes in container?
Question
How to save a change committed by/into a container?
Answer
The command docker commit creates a new image from a container's changes (from the man page).
Best Practice
You actually should not do this to save a configuration file. A Docker image is supposed to be immutable. This increases sharing, and image customization through mounted volume.
What you should do is create the configuration file on the host and share it at through parameters with docker run. This is done by using the option -v|--volume. Check the man page, you'll then be able to share files (or directories) between host and containers allowing to persists the data through different runs.