Couchbase with Docker Compose: Unable to insert - DurabilityImpossibleError - docker

I'm trying to set up Couchbase as part of a collection of servers using Docker-Compose. The sole purpose of this is for local application development.
The problem is that, once set up, I'm unable to write to the database. Insert and Upsert operations give me a DurabilityImpossibleError.
Docker compose file:
version: '3.4'
services:
...
couchbase-db:
image: couchbase/server
volumes:
- ./docker-data/couchbase/node3:/opt/couchbase/var
- ./provision/couchbase:/opt/startup/
ports:
- 8091:8091
- 8092:8092
- 8093:8093
- 8094:8094
- 11210:11210
The startup bash script, run after building, is an attempt to perform database setup without requiring a manual step:
#!/bin/bash
# Enables job control
set -m
# Enables error propagation
set -e
# Run the server and send it to the background
/entrypoint.sh couchbase-server &
# Check if couchbase server is up
check_db() {
curl --silent http://${1}:8091/pools > /dev/null
echo $?
}
# Variable used in echo
i=1
# Echo with
log() {
echo "[$i] [$(date +"%T")] $#"
i=`expr $i + 1`
}
# Wait until main server is ready
until [[ $(check_db 127.0.0.1) = 0 ]]; do
>&2 log "Waiting for Couchbase Server to be available ..."
sleep 1
done
couchbase-cli cluster-init -c localhost:8091 \
--cluster-username Administrator --cluster-password password \
--cluster-password password --services data,index,query --cluster-ramsize 512 \
--cluster-index-ramsize 256 || true
couchbase-cli setting-cluster -c localhost:8091 -u Administrator -p password \
--cluster-username Administrator --cluster-password password \
--cluster-password password --cluster-ramsize 512 \
--cluster-index-ramsize 256;
couchbase-cli setting-cluster -c localhost:8091 \
-u Administrator -p password --cluster-username Administrator \
--cluster-password password --cluster-ramsize 512 --cluster-index-ramsize 256;
curl -v POST http://localhost:8091/pools/default/buckets \
-u Administrator:password \
-d name=organisations \
-d bucketType=couchbase \
-d ramQuotaMB=512 \
-d durabilityMinLevel=majorityAndPersistActive
curl -v -X POST -u Administrator:password \
http://localhost:8091/settings/indexes \
-d indexerThreads=4 \
-d logLevel=verbose \
-d maxRollbackPoints=10 \
-d storageMode=plasma \
-d memorySnapshotInterval=150 \
-d stableSnapshotInterval=40000
# Need to wait until query service is ready to process N1QL queries
echo "$(date +"%T") Waiting ........."
sleep 20
# Create bucket1 indexes
echo "$(date +"%T") Create bucket1 indexes ........."
cbq -u Administrator -p password -s "CREATE PRIMARY INDEX idx_primary ON \`organisations\`;"
cbq -u Administrator -p password -s "CREATE INDEX idx_type ON \`organisations\`(_type);"
If I try to add a document via the web interface, I get:
Errors from server: ["Unexpected server error, request logged."]
If I try to add a document via the JavaScript SDK, I get:
DurabilityImpossibleError durability impossible
details:
{
name: 'DurabilityImpossibleError',
cause: LibcouchbaseError: libcouchbase error 308
at Object.translateCppError
(/app/node_modules/couchbase/dist/bindingutilities.js:174:21)
at /app/node_modules/couchbase/dist/connection.js:245:54 {
code: 308
},
context: KeyValueErrorContext {
status_code: 0,
opaque: 0,
cas: CbCas {
'0': <Buffer 00 00 00 00 00 00 00 00>
},
key: '22738bd4-7972-4370-85a3-71399d96ef05',
bucket: '',
collection: '',
scope: '',
context: '',
ref: ''
}
}
I've also attempted to send the following settings with the insert/upsert, to no effect:
insertOptions: {
durabilityLevel: 0,
durabilityPersistTo: 1,
durabilityReplicateTo: 0,
timeout: 5000,
},
My most recent attempt at a fix was to build a cluster of 3 nodes within docker-compose, and call the API commands to "add server" as part of a build script. However, "add server" takes a static IP, so the second time I run the servers, the IPs change and the database becomes unresponsive. I do get a functioning database on that first run though.
I'm looking for either a fix for a single node system (or an idea of where I'm going wrong), or a way of getting a cluster working in Docker-Compose after a down/up cycle. Anything that will give me a stable environment to develop in.
Thanks!

The bucket is created with -d durabilityMinLevel=majorityAndPersistActive, By default bucket enables replica with 1.
For single node cluster you will not have enough data nodes to satisfy durability (https://docs.couchbase.com/server/current/learn/data/durability.html). You can disable replica via UI and rebalance to take affect or change the bucket setting not include Minimum durability.
I have no idea about the 3 node docker compose error.

Related

SSH keys for Docker executor

I have created an image where I run some tasks.
I want to be able to push some files to a remote server that runs Windows Server 2022.
The gitlab-runner runs on an Ubuntu machine.
I managed to do that using shell executors. But now I want to do the same inside a docker container.
Using the following guide
https://docs.gitlab.com/ee/ci/ssh_keys/#ssh-keys-when-using-the-docker-executor
I don't understand in which user I will create the keys.
In a shell executor case I used gitlab-runner user in which I created a pair of keys. I added the public key to the server that I want to push files to and it worked.
However, I added the same private key into the gitlab CI/CD variable as the guide suggests.
Then inside the job I added the following:
before_script:
- 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
script:
- scp -P <port> myfile.txt username#ip:remote_path
But the job fails with errors
Host key verification failed.
lost connection
Should I use the same private key from gitlab-runner user?
PS: The echo "$SSH_PRIVATE_KEY" works. I can see the key I added in the gitlab CI/CD variable.
I used something similar in my CI process, works like a charm, I recall I've used some base64 formatted runner key due to some formatting errors:
- echo $GITLAB_RUNNER_SSH_KEY | base64 -d > $HOME/.ssh/runner_key
- chmod -R 600 ~/.ssh
- eval $(ssh-agent -s)
- ssh-add $HOME/.ssh/runner_key

How to Import Streamsets pipeline in Dockerfile without container exiting

I am trying to import a pipeline into streamsets, during container start up, by using the Docker CMD command in Dockerfile. The image builds, but while creating the container there is no error but it exits with code 0. So it never comes up. Here is what I did:
Dockerfile:
FROM streamsets/datacollector:3.18.1
COPY myPipeline.json /pipelinejsonlocation/
EXPOSE 18630
ENTRYPOINT ["/bin/sh"]
CMD ["/opt/streamsets-datacollector-3.18.1/bin/streamsets","cli","-U", "http://localhost:18630", \
"-u", \
"admin", \
"-p", \
"admin", \
"store", \
"import", \
"-n", \
"myPipeline", \
"--stack", \
"-f", \
"/pipelinejsonlocation/myPipeline.json"]
Build image:
docker build -t cmp/sdc .
Run image:
docker run -p 18630:18630 -d --name sdc cmp/sdc
This outputs the container id. But the container is in the Exited status as shown below.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
537adb1b05ab cmp/sdc "/bin/sh /opt/stream…" 5 seconds ago Exited (0) 3 seconds ago sdc
When I do not specify the CMD command in the Dockerfile, the streamsets container spins up and then when I run the streamsets import command in the running container in shell, it works. But how do I get it done during provisioning itself? Is there something I am missing in the Dockerfile?
In your Dockerfile you overwrite the default CMD and ENTRYPOINT from the StreamSets Data Collector Dockerfile. So the container only executes your command during startup and exits without errors afterwards. This is the reason why your container is in Exited (0) status.
In general this is good and expected behavior. If you want to keep your container alive you need to execute another command in the foreground, which never ends. But unfortunately, you cannot run multiple CMDs in your docker file.
I dug a little deeper. The default entry point of the image is ENTRYPOINT ["/docker-entrypoint.sh"]. This script sets up a few things and starts the Data Collector.
It is required that the Data Collector is running before the pipeline is imported. So a solution could be to copy the default docker-entrypoint.sh and modify it to start the Data Collector and import the pipeline afterwards. You could to it like this:
Dockerfile:
FROM streamsets/datacollector:3.18.1
COPY myPipeline.json /pipelinejsonlocation/
# Replace docker-entrypoint.sh
COPY docker-entrypoint.sh /docker-entrypoint.sh
EXPOSE 18630
docker-entrypoint.sh (https://github.com/streamsets/datacollector-docker/blob/master/docker-entrypoint.sh):
#!/bin/bash
#
# Copyright 2017 StreamSets Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
set -e
# We translate environment variables to sdc.properties and rewrite them.
set_conf() {
if [ $# -ne 2 ]; then
echo "set_conf requires two arguments: <key> <value>"
exit 1
fi
if [ -z "$SDC_CONF" ]; then
echo "SDC_CONF is not set."
exit 1
fi
grep -q "^$1" ${SDC_CONF}/sdc.properties && sed 's|^#\?\('"$1"'=\).*|\1'"$2"'|' -i ${SDC_CONF}/sdc.properties || echo -e "\n$1=$2" >> ${SDC_CONF}/sdc.properties
}
# support arbitrary user IDs
# ref: https://docs.openshift.com/container-platform/3.3/creating_images/guidelines.html#openshift-container-platform-specific-guidelines
if ! whoami &> /dev/null; then
if [ -w /etc/passwd ]; then
echo "${SDC_USER:-sdc}:x:$(id -u):0:${SDC_USER:-sdc} user:${HOME}:/sbin/nologin" >> /etc/passwd
fi
fi
# In some environments such as Marathon $HOST and $PORT0 can be used to
# determine the correct external URL to reach SDC.
if [ ! -z "$HOST" ] && [ ! -z "$PORT0" ] && [ -z "$SDC_CONF_SDC_BASE_HTTP_URL" ]; then
export SDC_CONF_SDC_BASE_HTTP_URL="http://${HOST}:${PORT0}"
fi
for e in $(env); do
key=${e%=*}
value=${e#*=}
if [[ $key == SDC_CONF_* ]]; then
lowercase=$(echo $key | tr '[:upper:]' '[:lower:]')
key=$(echo ${lowercase#*sdc_conf_} | sed 's|_|.|g')
set_conf $key $value
fi
done
# MODIFICATIONS:
#exec "${SDC_DIST}/bin/streamsets" "$#"
check_data_collector_status () {
watch -n 1 ${SDC_DIST}/bin/streamsets cli -U http://localhost:18630 ping | grep -q 'version' && echo "Data Collector has started!" && import_pipeline
}
function import_pipeline () {
sleep 1
echo "Start to import pipeline"
${SDC_DIST}/bin/streamsets cli -U http://localhost:18630 -u admin -p admin store import -n myPipeline --stack -f /pipelinejsonlocation/myPipeline.json
echo "Finished importing pipeline"
}
# Start checking if Data Collector is up (in background) and start Data Collector
check_data_collector_status & ${SDC_DIST}/bin/streamsets $#
I commented out the last line exec "${SDC_DIST}/bin/streamsets" "$#" of the default docker-entrypoint.sh and added two functions. check_data_collector_status () pings the Data Collector service until it is available. import_pipeline () imports your pipeline.
check_data_collector_status () runs in background and ${SDC_DIST}/bin/streamsets $# is started in foreground as before. So the pipeline is imported after the Data Collector service is started.
Run this image with sleep command:
docker run -p 18630:18630 -d --name sdc cmp/sdc sleep 300
300 is the time to sleep in seconds.
Then exec your script manually within the docker container and find out what's wrong.

supervisor manages a process which is started up with a shell script

I'm using supervisord to run multi-service in a container. I want a ldap service for my web application. So I installed and started opendj with the follow info,
Dockerfile
RUN dpkg -i $APP_HOME/packages/opendj_3.0.0-1_all.deb && \
/opt/opendj/setup \
--cli \
--backendType je \
--baseDN dc=test,dc=net \
--ldapPort 389 \
--adminConnectorPort 4444 \
--rootUserDN cn=Directory\ Manager \
--rootUserPassword 123456 \
--no-prompt \
--noPropertiesFile \
--acceptLicense \
--doNotStart
supervisord.conf
[program:ldap]
command=/opt/opendj/bin/start-ds
priority=1
When running my customized imgae, I got the following exiting message for ldap.
2020-05-25 06:46:03,486 INFO exited: ldap (exit status 0; expected)
Logging into the container to get all process status info with supervisorctl status all and ps -aux respectively.
$supervisorctl status all
ldap EXITED May 25 06:46 AM
$ps -aux
root 97 3.4 5.9 3489048 240248 pts/0 Sl 06:15 0:08 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -server -Dorg.opends.server.scriptName=start-ds org.opends.server.core.DirectoryServer --configClass org.opends.server.extensions.ConfigFileHandler
I found the ldap program starting up with start-ds shell script, that is, that start-ds shell process exited, but the ldap server which isn't controlled by supervisor is running.
If stopping supervisor subprocesses, the ldap server can't be stopped gracefully.
So my question is how to configure to make the supervisor to control the ldap server process which is started up by start-ds.
There is a --nodetach option that you should use in such cases: https://github.com/ForgeRock/opendj-community-edition/blob/master/resource/bin/start-ds#L60
Reference Doc says:
Options
The start-ds command takes the following options:
-N | --nodetach
Do not detach from the terminal and continue running in the foreground. This option cannot be used with the -t, --timeout option.
Default: false
The statement in start-ds.sh file is:
exec "${OPENDJ_JAVA_BIN}" ${OPENDJ_JAVA_ARGS} ${SCRIPT_NAME_ARG} \
org.opends.server.core.DirectoryServer \
--configClass org.opends.server.extensions.ConfigFileHandler \
--configFile "${CONFIG_FILE}" "${#}"
start-ds script will append this option when run /opt/opendj/bin/start-ds -N
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -server -Dorg.opends.server.scriptName=start-ds org.opends.server.core.DirectoryServer --configClass org.opends.server.extensions.ConfigFileHandler --configFile /opt/opendj/config/config.ldif -N

Can the kafka connectors be configured via env variables passed when launching docker? Or curl is the only way?

This is the docker image we use to host docker-connect with the plugins
FROM confluentinc/cp-kafka-connect:5.3.1
ENV CONNECT_PLUGIN_PATH=/usr/share/java
# JDBC-MariaDB
RUN wget -nv -P /usr/share/java/kafka-connect-jdbc/ https://downloads.mariadb.com/Connectors/java/connector-java-2.4.4/mariadb-java-client-2.4.4.jar
# SNMP Source
RUN wget -nv -P /tmp/ https://github.com/name/kafka-connect-snmp/releases/download/0.0.1.11/kafka-connect-snmp-0.0.1.11.tar.gz
RUN mkdir /tmp/kafka-connect-snmp && tar -xf /tmp/kafka-connect-snmp-0.0.1.11.tar.gz -C /tmp/kafka-connect-snmp/
RUN mv /tmp/kafka-connect-snmp/usr/share/kafka-connect/kafka-connect-snmp /usr/share/java/
I run this docker via docker-compose and then I have specified some common env variables defined here https://docs.confluent.io/current/installation/docker/config-reference.html#kafka-connect-configuration
But I also would like to specify connector related config from the env variable also, example I have done this
- CONNECT_NAME=snmp-connector
- CONNECT_CONNECTOR_CLASS=com.github.jcustenborder.kafka.connect.snmp.SnmpTrapSourceConnector
- CONNECT_TOPIC=fm_snmp
What I am trying to do it, instead of calling
curl -X POST -H "Content-Type: application/json" --data '{"name":"","config":{"connector.class":"com.github.jcustenborder.kafka.connect.snmp.SnmpTrapSourceConnector","topic":"fm_snmp"}}' http://localhost:8083/connectors
I want to just specify it via env variables, BUT!! unfortunately its not working. So when I try seeing list of active connectors curl -localhost:8083/connectors/ , then I dont see it listed there.
So finally, my question can I configure it via env variables or only curl is the way?
You can't pass it as environment variables, but you can specify it as part of your Docker startup by passing in a custom command. Here's an example of doing it with Docker Compose. If you're calling docker run itself you'd need to rework this into an appropriate structure:
kafka-connect:
image: confluentinc/cp-kafka-connect:5.3.1
environment:
CONNECT_REST_PORT: 18083
CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"
[…]
volumes:
- $PWD/scripts:/scripts
command:
- bash
- -c
- |
/etc/confluent/docker/run &
echo "Waiting for Kafka Connect to start listening on kafka-connect ⏳"
while [ $$(curl -s -o /dev/null -w %{http_code} http://kafka-connect:8083/connectors) -eq 000 ] ; do
echo -e $$(date) " Kafka Connect listener HTTP state: " $$(curl -s -o /dev/null -w %{http_code} http://kafka-connect:8083/connectors) " (waiting for 200)"
sleep 5
done
nc -vz kafka-connect 8083
echo -e "\n--\n+> Creating Kafka Connect Elasticsearch sink"
/scripts/create-es-sink.sh
sleep infinity
This calls a connector script, but if you want to embed it directly you can do it like this.

Couchbase in docker for integration tests: Make the ports 8092, 8093, 8094 and 8095 configurable to be able to use docker’s random ports

I am using Couchbase java client SDK 2.7.9 and am running into a problem while trying to run automated integration tests. In such test we usually use random ports to be able to run the same thing on the same Jenkins slave (using docker for example).
But, with the client, we can specify many custom ports but not the 8092, 8093, 8094 and 8095.
The popular TestContainers modules mention as well that those port have to remain static in their Couchbase module: https://www.testcontainers.org/modules/databases/couchbase/ 1
Apparently it is also possible to change those ports at the server level.
Example:
Docker-compose.yml
version: '3.0'
services:
rapid_test_cb:
build:
context: ""
dockerfile: cb.docker
ports:
- "8091"
- "8092"
- "8093"
- "11210"
The docker image is ‘couchbase:community-5.1.1’
Internally the ports are the ones written above but externally they are random. At the client level you can set up bootstrapHttpDirectPort and bootstrapCarrierDirectPort but apparently the 8092 and 8093 ports are taken from the server-side (who does not know which port was assigned to him).
I would like to ask you whether it is possible to change those ports at the client level and, if not, to seriously consider adding that feature.
So, as discussed with the Couchbase team here,
it is not really possible. So we found a way to make it work using Gradle's docker compose plugin but I imagine it would work in different situations (TestContainer could use a similar system).
docker-compose.yml:
version: '3.0'
services:
rapid_test_cb:
build:
context: ""
dockerfile: cb.docker
ports:
- "${COUCHBASE_RANDOM_PORT_8091}:${COUCHBASE_RANDOM_PORT_8091}"
- "${COUCHBASE_RANDOM_PORT_8092}:${COUCHBASE_RANDOM_PORT_8092}"
- "${COUCHBASE_RANDOM_PORT_8093}:${COUCHBASE_RANDOM_PORT_8093}"
- "${COUCHBASE_RANDOM_PORT_11210}:${COUCHBASE_RANDOM_PORT_11210}"
environment:
COUCHBASE_RANDOM_PORT_8091: ${COUCHBASE_RANDOM_PORT_8091}
COUCHBASE_RANDOM_PORT_8092: ${COUCHBASE_RANDOM_PORT_8092}
COUCHBASE_RANDOM_PORT_8093: ${COUCHBASE_RANDOM_PORT_8093}
COUCHBASE_RANDOM_PORT_11210: ${COUCHBASE_RANDOM_PORT_11210}
cb.docker:
FROM couchbase:community-5.1.1
COPY configure-node.sh /opt/couchbase
#HEALTHCHECK --interval=5s --timeout=3s CMD curl --fail http://localhost:8091/pools || exit 1
RUN chmod u+x /opt/couchbase/configure-node.sh
RUN echo "{rest_port, 8091}.\n{query_port, 8093}.\n{memcached_port, 11210}." >> /opt/couchbase/etc/couchbase/static_config
CMD ["/opt/couchbase/configure-node.sh"]
configure-node.sh:
#!/bin/bash
poll() {
# The argument supplied to the function is invoked using "$#", we check the return value with $?
"$#"
while [ $? -ne 0 ]
do
echo 'waiting for couchbase to start'
sleep 1
"$#"
done
}
set -x
set -m
if [[ -n "${COUCHBASE_RANDOM_PORT_8092}" ]]; then
sed -i "s|8092|${COUCHBASE_RANDOM_PORT_8092}|g" /opt/couchbase/etc/couchdb/default.d/capi.ini
fi
if [[ -n "${COUCHBASE_RANDOM_PORT_8091}" ]]; then
sed -i "s|8091|${COUCHBASE_RANDOM_PORT_8091}|g" /opt/couchbase/etc/couchbase/static_config
fi
if [[ -n "${COUCHBASE_RANDOM_PORT_8093}" ]]; then
sed -i "s|8093|${COUCHBASE_RANDOM_PORT_8093}|g" /opt/couchbase/etc/couchbase/static_config
fi
if [[ -n "${COUCHBASE_RANDOM_PORT_11210}" ]]; then
sed -i "s|11210|${COUCHBASE_RANDOM_PORT_11210}|g" /opt/couchbase/etc/couchbase/static_config
fi
/entrypoint.sh couchbase-server &
poll curl -s localhost:${COUCHBASE_RANDOM_PORT_8091:-8091}
# Setup index and memory quota
curl -v -X POST http://127.0.0.1:${COUCHBASE_RANDOM_PORT_8091:-8091}/pools/default --noproxy '127.0.0.1' -d memoryQuota=300 -d indexMemoryQuota=300
# Setup services
curl -v http://127.0.0.1:${COUCHBASE_RANDOM_PORT_8091:-8091}/node/controller/setupServices --noproxy '127.0.0.1' -d services=kv%2Cn1ql%2Cindex
# Setup credentials
curl -v http://127.0.0.1:${COUCHBASE_RANDOM_PORT_8091:-8091}/settings/web --noproxy '127.0.0.1' -d port=${couchbase_random_port_8091:-8091} -d username=Administrator -d password=password
# Load the rapid_test bucket
curl -X POST -u Administrator:password -d name=rapid_test -d ramQuotaMB=128 --noproxy '127.0.0.1' -d authType=sasl -d saslPassword=password -d replicaNumber=0 -d flushEnabled=1 -v http://127.0.0.1:${COUCHBASE_RANDOM_PORT_8091:-8091}/pools/default/buckets
fg 1
Gradle's docker compose configuration:
def findRandomOpenPortOnAllLocalInterfaces = {
new ServerSocket(0).withCloseable { socket ->
return socket.getLocalPort().intValue()
}
}
dockerCompose {
environment.put 'COUCHBASE_RANDOM_PORT_8091', findRandomOpenPortOnAllLocalInterfaces()
environment.put 'COUCHBASE_RANDOM_PORT_8092', findRandomOpenPortOnAllLocalInterfaces()
environment.put 'COUCHBASE_RANDOM_PORT_8093', findRandomOpenPortOnAllLocalInterfaces()
environment.put 'COUCHBASE_RANDOM_PORT_11210', findRandomOpenPortOnAllLocalInterfaces()
}
integTest.doFirst {
systemProperty 'com.couchbase.bootstrapHttpDirectPort', couchbase_random_port_8091
systemProperty 'com.couchbase.bootstrapCarrierDirectPort', couchbase_random_port_11210
}

Resources