Elasticsearch docker image with data persistence - docker

Question

I am having an issue with data persistence on my Elasticsearch docker image on my linux AWS EC2 machine.
I am launching the container like so:
docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 \
-v $PWD/elasticsearch/data:/usr/share/elasticsearch/data \
-e "discovery.type=single-node" \
docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4
The issue is with the -v $PWD/elasticsearch/data:/usr/share/elasticsearch/data line. On Mac everything works correctly and I can persist my data after bringing down the container, but on the linux machine I get permission errors on the /usr/share/elasticsearch/data directory in the container.
Error (line 3 is the critical part):
[2018-07-06T00:39:35,479][INFO ][o.e.n.Node ] [] initializing ...
[2018-07-06T00:39:35,503][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:244) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
... 6 more
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:223) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0_161]
at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_161]
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:209) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
... 6 more
What do I need to add to make this work on linux?

Answer 1

This will work.
Set permission to:
sudo mkdir -p $PWD/elasticsearch/data
sudo chmod 777 -R $PWD/elasticsearch/data
Then:
docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 \
-v $PWD/elasticsearch/data:/usr/share/elasticsearch/data \
-e "discovery.type=single-node" \
--name elasticsearch \
docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4

Answer 2

I found a solution by investigating the ownership of the folders from inside and outside the docker container. It starting working by running the command sudo chown -R 1000:root $PWD/elasticsearch/data before launching the container so that, in the container, docker thought it owned the directory.
Why does the same docker run create two different results on two different machines? Isn't the point of docker to be one size fits all?
This will work for now but I would like a better solution because I'm not sure if mine will always work as I don't know if 1000 will be the UID for docker every time.

Answer 3

In you docker-compose.yml, you can specify the user with user: $USER, like this:
elasticsearch:
user: $USER
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.5
volumes:
- /srv/graylog/es_data:/usr/share/elasticsearch/data
This is also solves the problem, and you don't need to run chown command.

Answer 4

Why this error happens?
The Elasticsearch can't be started using root user, it's probably an extra security the developer team gave us. Therefore, the docker image for Elasticsearch checks if the current container user is root, if it is, it changes the user to elasticsearch:elasticsearch or 1000:1000.
Possible solution
You can change the directory owner to 1000:1000. It will work. But what if you can't change the owner? I had this problem recently, I was trying to map a NFS directory to docker and couldn't change the owner in NFS.
Final solution
the docker-entrypoint.sh located at /usr/local/bin/docker-entrypoint.sh is the responsible for checking and changing root user to elasticsearch. It can be overriden with a simple custom image like the following:
FROM docker.elastic.co/elasticsearch/elasticsearch:7.11.0
COPY ./docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
Below there's an example of a modified docker-entrypoint.sh currently working for Elasticsearch 7.11.0. You can always get the docker-entrypoint.sh by starting a test container and using something like
docker cp <elasticsearch-container>:/usr/local/bin/docker-entrypoint.sh ./docker-entrypoint.sh
Modified docker-entrypoint.sh
#!/bin/bash
set -e
# Files created by Elasticsearch should always be group writable too
umask 0002
run_as_other_user_if_needed() {
if [[ "$(id -u)" == "0" ]]; then
# If running as root, drop to specified UID and run command
exec chroot --userspec=<uid>:<gid> / "${#}"
else
# Either we are running in Openshift with random uid and are a member of the root group
# or with a custom --user
exec "${#}"
fi
}
# Allow user specify custom CMD, maybe bin/elasticsearch itself
# for example to directly specify `-E` style parameters for elasticsearch on k8s
# or simply to run /bin/bash to check the image
if [[ "$1" != "eswrapper" ]]; then
if [[ "$(id -u)" == "0" && $(basename "$1") == "elasticsearch" ]]; then
# centos:7 chroot doesn't have the `--skip-chdir` option and
# changes our CWD.
# Rewrite CMD args to replace $1 with `elasticsearch` explicitly,
# so that we are backwards compatible with the docs
# from the previous Elasticsearch versions<6
# and configuration option D:
# https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html#_d_override_the_image_8217_s_default_ulink_url_https_docs_docker_com_engine_reference_run_cmd_default_command_or_options_cmd_ulink
# Without this, user could specify `elasticsearch -E x.y=z` but
# `bin/elasticsearch -E x.y=z` would not work.
set -- "elasticsearch" "${#:2}"
# Use chroot to switch to UID 1000 / GID 0
exec chroot --userspec=<uid>:<gid> / "$#"
else
# User probably wants to run something else, like /bin/bash, with another uid forced (Openshift?)
exec "$#"
fi
fi
# Allow environment variables to be set by creating a file with the
# contents, and setting an environment variable with the suffix _FILE to
# point to it. This can be used to provide secrets to a container, without
# the values being specified explicitly when running the container.
#
# This is also sourced in elasticsearch-env, and is only needed here
# as well because we use ELASTIC_PASSWORD below. Sourcing this script
# is idempotent.
source /usr/share/elasticsearch/bin/elasticsearch-env-from-file
if [[ -f bin/elasticsearch-users ]]; then
# Check for the ELASTIC_PASSWORD environment variable to set the
# bootstrap password for Security.
#
# This is only required for the first node in a cluster with Security
# enabled, but we have no way of knowing which node we are yet. We'll just
# honor the variable if it's present.
if [[ -n "$ELASTIC_PASSWORD" ]]; then
[[ -f /usr/share/elasticsearch/config/elasticsearch.keystore ]] || (run_as_other_user_if_needed elasticsearch-keystore create)
if ! (run_as_other_user_if_needed elasticsearch-keystore has-passwd --silent) ; then
# keystore is unencrypted
if ! (run_as_other_user_if_needed elasticsearch-keystore list | grep -q '^bootstrap.password$'); then
(run_as_other_user_if_needed echo "$ELASTIC_PASSWORD" | elasticsearch-keystore add -x 'bootstrap.password')
fi
else
# keystore requires password
if ! (run_as_other_user_if_needed echo "$KEYSTORE_PASSWORD" \
| elasticsearch-keystore list | grep -q '^bootstrap.password$') ; then
COMMANDS="$(printf "%s\n%s" "$KEYSTORE_PASSWORD" "$ELASTIC_PASSWORD")"
(run_as_other_user_if_needed echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password')
fi
fi
fi
fi
if [[ "$(id -u)" == "0" ]]; then
# If requested and running as root, mutate the ownership of bind-mounts
if [[ -n "$TAKE_FILE_OWNERSHIP" ]]; then
chown -R <uid>:<gid> /usr/share/elasticsearch/{data,logs}
fi
fi
if [[ -n "$ES_LOG_STYLE" ]]; then
case "$ES_LOG_STYLE" in
console)
# This is the default. Nothing to do.
;;
file)
# Overwrite the default config with the stack config
mv /usr/share/elasticsearch/config/log4j2.file.properties /usr/share/elasticsearch/config/log4j2.properties
;;
*)
echo "ERROR: ES_LOG_STYLE set to [$ES_LOG_STYLE]. Expected [console] or [file]" >&2
exit 1 ;;
esac
fi
# Signal forwarding and child reaping is handled by `tini`, which is the
# actual entrypoint of the container
run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch <<<"$KEYSTORE_PASSWORD"
notice that the comments in the code are the original ones, it may be confusing they saying the code will change the user to elasticsearch.

Answer 5

This solved it for me:
sudo chown 1000:1000 -R $PWD/elasticsearch/data
The reason is that ES now runs with user 1000 and needs the directory to have permissions for user 1000

Answer 6

In case somebody land here with the same problem as me.
java.lang.IllegalStateException: failed to obtain node locks, tried
[[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations
are not writable or multiple nodes were started without increasing
[node.max_local_storage_nodes] (was [1])?
And created 2+ containers with docker-compose.yml, check that the volumes are different for each container. It was that in my case.