APOC installation not working in Neo4J docker container - docker

I have a docker container running with a Graph created. I am following this guide to installing APOC. I have copied the JAR file from /var/lib/neo4j/labs to /var/lib/neo4j/plugins and have restarted the container.
A screenshot of the instructions:
I also enabled dbms.security.procedures.unrestricted=apoc.* but the APOC calls do not work after restarting the container.
It always says,
"There is no procedure with the name apoc.help registered for this database instance. Please ensure you've spelled the procedure name correctly and that the procedure is properly deployed."
Is there anything I am missing?
My Neo4J version: 4.4.11
APOC versions I have tried are: apoc-4.4.0.8-core, apoc-4.4.0.6-core and apoc-4.4.0.9-core
Update 1
Script that produces the above output:
MATCH (n:FEATURE{name:'Update_Profile'})
CALL apoc.path.spanningTree(n,{maxLevel:15}) YIELD path
RETURN path
Second one:
CALL apoc.export.cypher.all("all-plain.cypher", {
format: "plain",
useOptimizations: {type: "UNWIND_BATCH", unwindBatchSize: 20}
})
YIELD file, batches, source, format, nodes, relationships, properties, time, rows, batchSize
RETURN file, batches, source, format, nodes, relationships, properties, time, rows, batchSize;
Update 2
I run the following command inside the docker container to copy the jar file having the current location at /var/lib/neo4j
cp labs/apoc-4.4.0.8-core.jar /var/lib/neo4j/plugins/
After this, I restarted the container using:
sudo docker container restart cybersage-neo4j

Why don't you use NEO4JLABS_PLUGINS environment variable to simply install APOC automatically like in the documentation?
docker run \
-p 7474:7474 -p 7687:7687 \
-v $PWD/data:/data -v $PWD/plugins:/plugins \
--name neo4j-apoc \
-e NEO4J_apoc_export_file_enabled=true \
-e NEO4J_apoc_import_file_enabled=true \
-e NEO4J_apoc_import_file_use__neo4j__config=true \
-e NEO4JLABS_PLUGINS=\[\"apoc\"\] \
-e NEO4J_dbms_security_procedures_unrestricted=apoc.\\\* \
neo4j:4.0

Related

Avoiding duplicated arguments when running a Docker container

I have a tensorflow training script which I want to run using a Docker container (based on the official TF GPU image). Although everything works just fine, running the container with the script is horribly verbose and ugly. The main problem is that my training script allows the user to specify various directories used during training, for input data, logging, generating output, etc. I don't want to have change what my users are used to, so the container needs to be informed of the location of these user-defined directories, so it can mount them. So I end up with something like this:
docker run \
-it --rm --gpus all -d \
--mount type=bind,source=/home/guest/datasets/my-dataset,target=/datasets/my-dataset \
--mount type=bind,source=/home/guest/my-scripts/config.json,target=/config.json \
-v /home/guest/my-scripts/logdir:/logdir \
-v /home/guest/my-scripts/generated:/generated \
train-image \
python train.py \
--data_dir /datasets/my-dataset \
--gpu 0 \
--logdir ./logdir \
--output ./generated \
--config_file ./config.json \
--num_epochs 250 \
--batch_size 128 \
--checkpoint_every 5 \
--generate True \
--resume False
In the above I am mounting a dataset from the host into the container, and also mounting a single config file config.json (which configures the TF model). I specify a logging directory logdir and an output directory generated as volumes. Each of these resources are also passed as parameters to the train.py script.
This is all very ugly, but I can't see another way of doing it. Of course I could put all this in a shell script, and provide command line arguments which set these duplicated values from the outside. But this doesn't seem a nice solution, because if I want to anything else with the container, for example check the logs, I would use the raw docker command.
I suspect this question will likely be tagged as opinion-based, but I've not found a good solution for this that I can recommend to my users.
As user Ron van der Heijden points out, one solution is to use docker-compose in combination with environment variables defined in an .env file. Nice answer.

Cypher init script Neo4j 4.2 docker not working

I am trying to refer to this link of documentation: https://neo4j.com/labs/apoc/4.2/operational/init-script/
This is my docker command:
docker run --rm \
--env NEO4J_AUTH="neo4j/test" \
--env NEO4JLABS_PLUGINS='["apoc"]' \
--env NEO4J_apoc_export_file_enabled="true" \
--env NEO4J_apoc_import_file_enabled="true" \
--env NEO4J_apoc_import_file_use__neo4j__config="true" \
--env NEO4J_dbms_directories_import="/" \
--env NEO4J_apoc_initializer_neo4j_1='CALL apoc.cypher.runSchemaFile("file:////var/lib/neo4j/db_init_scripts/db_ddls.cypher");' \
--env NEO4J_apoc_initializer_neo4j_2='CALL apoc.cypher.runFile("file:////var/lib/neo4j/db_init_scripts/db_schema.cypher");' \
--name neo4j \
neo4j:cus
where neo4j:cus is my custom image where I copied the required cypher files into neo4j:4.2 image
My db_ddl.cypher has create indexes:
CREATE CONSTRAINT idx_person_unq IF NOT EXISTS ON (p:Person) ASSERT p.name IS UNIQUE;
My db_schema.cypher has schema creation:
MERGE (p:Person {name: "Inital Person"});
When I try to start the container, I get below message:
Unrecognized setting. No declared setting with name: apoc.initializer.cypher.1
Unrecognized setting. No declared setting with name: apoc.initializer.cypher.2
When I try to use, older version of environment variables:
--env NEO4J_apoc_initializer_cypher_1='CALL apoc.cypher.runSchemaFile("file:////var/lib/neo4j/db_init_scripts/db_schema.cypher");' \
--env NEO4J_apoc_initializer_cypher_2='CALL apoc.cypher.runFile("file:////var/lib/neo4j/db_init_scripts/db_ddls.cypher");' \
I get this:
Unrecognized setting. No declared setting with name: apoc.initializer.cypher.1
Unrecognized setting. No declared setting with name: apoc.initializer.cypher.2
But when I try to run only one cypher:
--env NEO4J_apoc_initializer_cypher='CALL apoc.cypher.runSchemaFile("file:////var/lib/neo4j/db_init_scripts/db_ddls.cypher");' \
Then it works.
This is how my conf/neo4j.conf looks:
root#4f8955759b52:/var/lib/neo4j# grep apoc conf/neo4j.conf
#dbms.security.procedures.allowlist=apoc.coll.*,apoc.load.*,gds.*
apoc.initializer.neo4j.1=CALL apoc.cypher.runFile("file:////var/lib/neo4j/db_init_scripts/db_schema.cypher");
apoc.initializer.neo4j.0=CALL apoc.cypher.runSchemaFile("file:////var/lib/neo4j/db_init_scripts/db_ddls.cypher");
apoc.import.file.use_neo4j_config=true
apoc.import.file.enabled=true
apoc.export.file.enabled=true
dbms.security.procedures.unrestricted=apoc.*
Can anyone point what I am missing so that I can run both creation of indexes as well as initialize some schema as well?
Thanks
Well this behavior works only when you create a separate apoc.conf file and place it beside neo4j.conf
my apoc.conf now looks like this:
apoc.initializer.neo4j.1=CALL apoc.cypher.runSchemaFile("file:////var/lib/neo4j/db_init_scripts/db_ddls.cypher")
apoc.initializer.neo4j.2=CALL apoc.cypher.runFile("file:////var/lib/neo4j/db_init_scripts/db_schema.cypher");
And log files have outputs:
2021-05-26 18:51:06.068+0000 INFO [neo4j/c0fb7489] successfully initialized: CALL apoc.cypher.runSchemaFile("file:////var/lib/neo4j/db_init_scripts/db_ddls.cypher")
2021-05-26 18:51:06.882+0000 INFO Remote interface available at http://localhost:7474/
2021-05-26 18:51:06.892+0000 INFO Started.
2021-05-26 18:51:08.091+0000 INFO [neo4j/c0fb7489] successfully initialized: CALL apoc.cypher.runFile("file:////var/lib/neo4j/db_init_scripts/db_schema.cypher");

How to showvariable with gitversion Docker

I can successfully get the full json string with:
docker run --rm -v `pwd`:`pwd` gittools/gitversion-dotnetcore:linux-4.0.0 `pwd` -output json
which outputs to something like:
{
"Major":0,
"Minor":1,
"Patch":0,
"SemVer":"0.1.0-dev-2.1",
.
.
.
"CommitsSinceVersionSource":20,
"CommitsSinceVersionSourcePadded":"0020",
"CommitDate":"2020-05-28"
}
Since I am only interested in SemVer variable I try to use the -showvariable FullSemVer with:
docker run --rm -v `pwd`:`pwd` gittools/gitversion-dotnetcore:linux-4.0.0 `pwd` -output json -showvariable FullSemVer
But it fails with a quite long and nasty error log.
INFO [05/28/20 18:23:12:10] End: Loading version variables from disk cache (Took: 76.31ms)
ERROR [05/28/20 18:23:12:13] An unexpected error occurred:
System.NotImplementedException: The method or operation is not implemented.
I wonder if there is a way to use the -showvariable flag with the gitversion Docker container?
I think the problem is the path argument passed to GitVersion. pwd will give you the working directory on your host, not within the container. GitVersion is unfortunately not aware of the fact that it's executing within a container, so it needs to be provided with the volume directory /repo as the path to calculate a version number for. This is something we should consider changing in version 6.
I also can't remember when -showvariable was implemented, so to be on the safe side you should try with a newer version of our Docker containers. I can also recommend using the alpine container, as it's the smallest one we offer (only 83.9 MB). This works:
docker run \
--rm \
--volume "$(pwd):/repo" \
gittools/gitversion:5.3.4-linux-alpine.3.10-x64-netcoreapp3.1 \
/repo \
-output json \
-showvariable FullSemVer

Understanding the difference in sequence of ENTRYPOINT/CMD between Dockerfile and docker run

Docker noob here...
I am trying to build and run an IBM DataPower container from a Dockerfile, but it doesn't seem to work the same as when just running docker run and passing the same parameters in the terminal.
This works (docker run)
docker run -it \
-v $PWD/config:/drouter/config \
-e DATAPOWER_ACCEPT_LICENSE=true \
-e DATAPOWER_INTERACTIVE=true \
-e DATAPOWER_WORKER_THREADS=4 \
-p 9090:9090 \
--name mydatapower \
ibmcom/datapower
... the key part being that it mounts the ./config folder and the custom configuration is picked up by datapower running in the container.
This doesn't (Dockerfile)
Dockerfile:
FROM ibmcom/datapower
ENV DATAPOWER_ACCEPT_LICENSE=true
ENV DATAPOWER_INTERACTIVE=true
ENV DATAPOWER_WORKER_THREADS=4
EXPOSE 9090
COPY config/auto-startup.cfg /drouter/config/auto-startup.cfg
Build:
docker build -t local/datapower .
Run:
docker run -it \
-p 9090:9090 \
--name mydatapower local/datapower
The problem is that DataPower doesn't pick up the auto-startup.cfg file, so the additional config options doesn't get used. I know the source file path is correct because if I misspell the file name docker throws an error.
I have a theory that it might be running the inherited ENTRYPOINT or CMD before the config file is available. I don't know how to test or prove this. I don't know what the ENTRYPOINT or CMD is because the inherited image is not open source and I can't figure out how to find it.
Does that seem likely?
UPDATE:
The content of the auto-startup.cfg is:
top; co
ssh
web-mgmt
admin enabled
port 9090
exit
It simply enables the DataPower WebGUI.
The output when running it in the commandline with:
docker run -it -v $PWD/config:/drouter/config -v $PWD/local:/drouter/local -e DATAPOWER_ACCEPT_LICENSE=true -e DATAPOWER_INTERACTIVE=true -e DATAPOWER_WORKER_THREADS=4 -p 9091:9090 --name myconfigureddatapower ibmcom/datapower`
...contains this:
20170908T121729.015Z [0x8100006e][system][notice] : Executing startup configuration.
20170908T121729.970Z [0x00350014][mgmt][notice] web-mgmt(WebGUI-Settings): tid(303): Operational state up
...but with Dockerfile it doesn't. That's why I think the config files may be copied into place too late.
I've tried adding CMD ["/bin/drouter"] to the end of my Dockerfile to no avail.
I have tested your Dockerfile and it seems to be working. My auto-startup.cfg file is copied in the proper location and when I launch the container it's reading the file.
I get this output:
[root#ip-172-30-2-164 tmp]# docker run -ti -p 9090:9090 test
20170908T123728.818Z [0x8040006b][system][notice] logging target(default-log): Logging started.
20170908T123729.067Z [0x804000fe][system][notice] : Container instance UUID: 36bcca0e-6139-4694-91b0-2b7b66c3a498, Cores: 4, vCPUs: 4, CPU model: Intel(R) Xeon(R) CPU E5-2676 v3 # 2.40GHz, Memory: 16049.1MB, Platform: docker, OS: dpos, Edition: developers-limited, Up time: 0 minutes
20170908T123729.071Z [0x8040001c][system][notice] : DataPower IDG is on-line.
20170908T123729.071Z [0x8100006f][system][notice] : Executing default startup configuration.
20170908T123729.416Z [0x8100006d][system][notice] : Executing system configuration.
20170908T123729.417Z [0x8100006b][mgmt][notice] domain(default): tid(8143): Domain operational state is up.
708f98be1390
Unauthorized access prohibited.
20170908T123731.239Z [0x806000dd][system][notice] cert-monitor(Certificate Monitor): tid(399): Enabling Certificate Monitor to scan once every 1 days for soon to expire certificates
20170908T123731.552Z [0x8100006e][system][notice] : Executing startup configuration.
20170908T123732.436Z [0x8100003b][mgmt][notice] domain(default): Domain configured successfully.
20170908T123732.449Z [0x00350014][mgmt][notice] web-mgmt(WebGUI-Settings): tid(303): Operational state up
login:
To check that your file has been copied to the container you can run docker run -ti local/datapower sh to enter the container and then check the content of /drouter/config/.
Your base image command is: CMD ["/bin/drouter"] you can check it running docker history ibmcom/datapower.
UPDATE:
The drouter user in the container must be able to read the auto-startup.cfg file. You have 2 options:
set your local auto-startup.cfg with the proper permissions (chmod 644 config/autostart.cfg).
or add these line in the Dockerfile so drouter can read the file:
USER root
RUN chown drouter /drouter/config/auto-startup.cfg
USER drouter

Why is my systemd unit not reading env variables properly?

I am trying to run kubernetes on coreos. I am using fleet, setup-network-environment, and kube-register to register nodes. However, in my cloud-init file where I write my systemd unit files, the kubelet's unit file won't run this properly:
ExecStart=/opt/bin/kubelet \
--address=0.0.0.0 --port=10250 \
--hostname_override=${DEFAULT_IPV4} \
--allow_privileged=true \
--logtostderr=true \
--healthz_bind_address=0.0.0.0
Instead of my public ip, ${DEFAULT_IPV4} results in $default_ipv4, which also doesn't result in the ip. I know --host-name-override should just take a string, and it works when I run this line from command line. There are other unit files where ${ENV_VAR} works fine. Why is it that for the kubelet's unit file, it just breaks?
EDIT 1
/etc/network-environment
LO_IPV4=127.0.0.1
ENS33_IPV4=192.168.195.242
DEFAULT_IPV4=192.168.195.242
ENS34_IPV4=172.22.22.238
EDIT 2
kubelet unit file
- name: kube-kubelet.service
command: start
content: |
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Requires=setup-network-environment.service
After=setup-network-environment.service
[Service]
EnvironmentFile=/etc/network-environment
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kubelet -z /opt/bin/kubelet https://storage.googleapis.com/kubernetes-release/release/v0.18.2/bin/linux/amd64/kubelet
ExecStartPre=/usr/bin/chmod +x /opt/bin/kubelet
# wait for kubernetes master to be up and ready
ExecStartPre=/opt/bin/wupiao 172.22.22.10 8080
ExecStart=/opt/bin/kubelet \
--address=0.0.0.0 \
--port=10250 \
--hostname_override=172.22.22.21 \
--api_servers=172.22.22.10:8080 \
--allow_privileged=true \
--logtostderr=true \
--healthz_bind_address=0.0.0.0 \
--healthz_port=10248
Restart=always
RestartSec=10
The Exec*=command is not a shell command. In my experimenting it was not very good at figuring out where the variable was unless it was by itself. I went and looked at some examples online and they always show the environment variable by itself. So, given a file like /tmp/myfile:
ENV=1.2.3.4
These [Service] definitions won't do what you think:
EnvironmentFile=/tmp/myfile
ExecStart=echo M$ENV
ExecStart=echo $ENV:8080
but, this will work on a line by itself:
EnvironmentFile=/tmp/myfile
ExecStart=echo $ENV
That doesn't help much when trying to pass an argument, like:
EnvironmentFile=/tmp/myfile
ExecStart=echo --myarg=http://$ENV:8080/v2
To accomplish passing the argument I had to put the entire myarg in a string in /tmp/myfile:
ENV="--myarg=http://1.2.3.4:8080/v2"
Finally I could can get my argument passed:
EnvironmentFile=/tmp/myfile
ExecStart=echo $ENV
It would seem the issue was in the version of coreos in the vagrant box. After an update of the vagrant box the environment variable was able to resolve to the proper value.

Resources