Elasticsearch can't find hunspell - docker

I'm running elasticsearch from a docker-compose.yml file:
version: '3'
services:
mongo:
image: mongo
container_name: mongo-cust-mycom
ports:
- 27017:27017
volumes:
- cust-mycom-mongo:/data/db
networks:
- cust-mycom
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
container_name: elasticsearch-cust-mycom
ports:
- 9200:9200
- 9300:9300
volumes:
- cust-mycom-elastic:/usr/share/elasticsearch/data
- ./cust/config/elasticsearch/config/hunspell:/usr/share/elasticsearch/config/hunspell
networks:
- cust-mycom
environment:
- cluster.name=i3-elasticsearch
- xpack.security.enabled=false
- xpack.monitoring.enabled=false
- xpack.ml.enabled=false
- xpack.graph.enabled=false
- xpack.watcher.enabled=false
restart: unless-stopped
kibana:
image: docker.elastic.co/kibana/kibana:6.5.4
container_name: kibana-cust-mycom
ports:
- 5601:5601
networks:
- cust-mycom
depends_on:
- elasticsearch
restart: unless-stopped
networks:
cust-mycom:
driver: bridge
volumes:
cust-mycom-mongo:
cust-mycom-elastic:
using docker-compose up -d.
When I try to create my desired index, using the following json:
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 2,
"analysis": {
"filter": {
"swedish_stemmer": {
"type": "hunspell",
"language": "sv_SE"
},
"ins_pattern": {
"type": "pattern_capture",
"patterns": [
"([a-zåäö]*)(prod)"
]
},
"cust_stopwords": {
"type": "stop",
"stopwords": [ "en", "ett", "det", "den" ]
}
},
"analyzer": {
"swedish_index": {
"tokenizer": "standard",
"filter": [
"lowercase",
"ins_pattern",
"swedish_stemmer"
]
},
"swedish_query": {
"tokenizer": "standard",
"filter": [
"lowercase",
"swedish_stemmer",
"cust_stopwords"
]
}
}
}
},
"mappings": {
"default": {
"properties": {
"keywords": {
"type": "text",
"store": true,
"norms": false,
"analyzer": "swedish_index",
"search_analyzer": "swedish_query"
},
"audience": {
"type": "keyword"
},
"contentExcerpt": {
"type": "text"
},
"date": {
"type": "date",
"store": true,
"format": "dateOptionalTime"
},
"validUntil": {
"type": "date",
"store": true,
"format": "dateOptionalTime"
},
"informationType": {
"type": "text",
"store": true,
"norms": false,
"analyzer": "swedish_index",
"search_analyzer": "swedish_query"
},
"mainContentOfPage": {
"type": "text",
"store": true,
"norms": false,
"analyzer": "swedish_index",
"search_analyzer": "swedish_query",
"term_vector": "with_positions_offsets"
},
"thumbnailUrl": {
"type": "keyword",
"store": true,
"norms": false
},
"title": {
"type": "text",
"store": true,
"norms": false,
"analyzer": "swedish_index",
"search_analyzer": "swedish_query"
},
"url": {
"type": "keyword",
"store": true,
"norms": false
},
"tags": {
"type": "text",
"store": true,
"norms": false,
"analyzer": "swedish_index",
"search_analyzer": "swedish_query"
}
}
}
}
}
and the following script:
#!/bin/bash
curl -XDELETE http://localhost:9200/main
curl -XPUT -H "Content-type: application/json" -d #json/custse.index.json http://localhost:9200/main
curl -XPUT http://localhost:9200/main/_settings -H "Content-Type: application/json" -d "{
\"index\" : {
\"number_of_replicas\" : 0
}
}"
I get the following error message:
{"error":{"root_cause":[{"type":"illegal_state_exception","reason":"failed to load hunspell dictionary for locale: sv_SE"}]
I've tried putting my hunspell dictionaries in /usr/share/elasticsearch/config/hunspell/, /usr/share/elasticsearch/hunspell, /etc/elasticsearch/hunspell and /etc/elasticsearch/config/hunspell. It can't find any of them.
Here are the contents of the hunspell file:
/etc/elasticsearch$ ls hunspell
sv_SE
/etc/elasticsearch$ ls hunspell/sv_SE/
cust.dic README_sv_SE.txt sv_SE.aff sv_SE.dic
How can I make elasticsearch find the hunspell dictionaries?

I confirm that this configuration works:
version: '3.4'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.13
container_name: elasticsearch6
ports:
- "127.0.0.1:9200:9200"
environment:
- "ES_JAVA_OPTS=-Xms256m -Xmx256m"
restart: always
volumes:
- "es_data:/usr/share/elasticsearch/data"
- ./elasticsearch/hunspell:/usr/share/elasticsearch/config/hunspell
volumes:
es_data:

Related

Filebeat 7.10.1 add_docker_metadata adds only container.id

I'm using filebeat 7.10.1 installed on host system (not docker container), running as service by root
according to https://www.elastic.co/guide/en/beats/filebeat/current/add-docker-metadata.html
and https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-container.html
filebeat config, filebeat.yml:
filebeat.inputs:
- type: container
enabled: true
paths:
- '/var/lib/docker/containers/*/*.log'
processors:
- add_docker_metadata: ~
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
setup.kibana:
output.logstash:
hosts: ["<logstash_host>:5044"]
started container:
docker run --rm -d -l my-label --label com.example.foo=bar -p 80:80 nginx
filebeat get logs and successfully send them to endpoint (in my case to logstash, which resend to elasticsearch), but generated json by filebeat contains only container.id without container.name, container.labels and container.image
it looks like (copy-paste from kibana):
{
"_index": "logstash-2021.02.10",
"_type": "_doc",
"_id": "s4a4i3cB8j0XLXFVuyMm",
"_version": 1,
"_score": null,
"_source": {
"#version": "1",
"ecs": {
"version": "1.6.0"
},
"#timestamp": "2021-02-10T11:33:54.000Z",
"host": {
"name": "<some_host>"
},
"input": {
"type": "container"
},
"tags": [
"beats_input_codec_plain_applied"
],
"log": {
.....
},
"stream": "stdout",
"container": {
"id": "15facae2115ea57c9c99c13df815427669e21053791c7ddd4cd0c8caf1fbdf8c-json.log"
},
"agent": {
"version": "7.10.1",
"ephemeral_id": "adebf164-0b0d-450f-9a50-11138e519a27",
"id": "0925282e-319e-49e0-952e-dc06ba2e0c43",
"name": "<some_host>",
"type": "filebeat",
"hostname": "<some_host>"
}
},
"fields": {
"log.timestamp": [
"2021-02-10T11:33:54.000Z"
],
"#timestamp": [
"2021-02-10T11:33:54.000Z"
]
},
"highlight": {
"log.logger_name": [
"#kibana-highlighted-field#gw_nginx#/kibana-highlighted-field#"
]
},
"sort": [
1612956834000
]
}
what am I doing wrong? How to configure filebeat for send container.name, container.labels, container.image?
So after looking on filebeat-debug and paths on filesystem - issue closed
Reason: symlink /var/lib/docker -> /data/docker produces unexpected behavior
Solution:
filebeat.inputs:
- type: container
enabled: true
paths:
- '/data/docker/containers/*/*.log' #use realpath
processors:
- add_docker_metadata:
match_source_index: 3 #subfolder for extract container id from path

How to set docker command --log.level=DEBUG in a marathon deployment file?

reading at this docker-compose file : (Source: https://docs.traefik.io/user-guides/docker-compose/acme-dns/#setup)
version: "3.3"
services:
traefik:
image: "traefik:v2.1.0"
container_name: "traefik"
command:
- "--log.level=DEBUG"
How can I set --log.level=DEBUG command in this marathon deployment file:
{
"id": "/whoami",
"cpus": 0.1,
"mem": 256.0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "traefik:v2.1.0",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 80,
"protocol": "tcp"
}
]
}
}
}
I think you just need to add "args": ["--log.level=DEBUG"] to your JSON.
{
"id": "/whoami",
"cpus": 0.1,
"mem": 256.0,
"instances": 1,
"args": ["--log.level=DEBUG"],
"container": {
"type": "DOCKER",
"docker": {
"image": "traefik:v2.1.0",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 80,
"protocol": "tcp"
}
]
}
}
}

Does ECS task definition support volume mapping syntax?

docker-compose spec support volume mapping syntax under services, for example:
version: '2'
volumes:
jenkins_home:
external: true
services:
jenkins:
build:
context: .
args:
DOCKER_GID: ${DOCKER_GID}
DOCKER_VERSION: ${DOCKER_VERSION}
DOCKER_COMPOSE: ${DOCKER_COMPOSE}
volumes:
- jenkins_home:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock
ports:
- "8080:8080"
Following "AWSTemplateFormatVersion": "2010-09-09", the corresponding ECS task definition has volume syntax un-readable(with MountPoints and Volumes), as shown below:
"EcsTaskDefinition": {
"Type": "AWS::ECS::TaskDefinition",
"Properties": {
"ContainerDefinitions": [
{
"Name": "jenkins",
"Image": "xyzaccount/jenkins:ecs",
"Memory": 995,
"PortMappings": [ { "ContainerPort": 8080, "HostPort": 8080 } ],
"MountPoints": [
{
"SourceVolume": "docker",
"ContainerPath": "/var/run/docker.sock"
},
{
"SourceVolume": "jenkins_home",
"ContainerPath": "/var/jenkins_home"
}
]
}
],
"Volumes": [
{
"Name": "jenkins_home",
"Host": { "SourcePath": "/ecs/jenkins_home" }
},
{
"Name": "docker",
"Host": { "SourcePath": "/var/run/docker.sock" }
}
]
}
}
Does ECS task definition syntax of CloudFormation (now) support volume mapping syntax? similar to docker-compose....
Yes, of course, ECS support docker socket mounting, but the syntax is bit different. Add DOCKER_HOST environment variable in the task definition and source path should start with //.
"volumes": [
{
"name": "docker",
"host": {
"sourcePath": "//var/run/docker.sock"
}
}
]
The // worked in case of AWS ecs.
Also, you need to add DOCKER_HOST environment variable in your task definition.
"environment": [
{
"name": "DOCKER_HOST",
"value": "unix:///var/run/docker.sock"
}
]

Docker swarm worker nodes bad gateway

I setup Traefik with docker swarm based on this guide. It only worked for single node for the SSL. But I add certificate to my other nodes on swarm.
Update cert all the nodes
docker node update --label-add traefik-public.traefik-public-certificates=true $NODE_ID
Start Traefik as Manager Role
docker service create \
--name traefik \
--constraint=node.labels.traefik-public.traefik-public-certificates==true \
--constraint=node.role==manager \
--publish 80:80 \
--publish 443:443 \
--mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
--mount type=volume,source=traefik-public-certificates,target=/certificates \
--network traefik-public \
--label "traefik.frontend.rule=Host:monitor.$DOMAINNAME" \
--label "traefik.enable=true" \
--label "traefik.port=8080" \
--label "traefik.tags=traefik-public" \
--label "traefik.docker.network=traefik-public" \
--label "traefik.redirectorservice.frontend.entryPoints=http" \
--label "traefik.redirectorservice.frontend.redirect.entryPoint=https" \
--label "traefik.webservice.frontend.entryPoints=https" \
--label "traefik.frontend.auth.basic.users=${HTTP_USERNAME}:${HASHED_PASSWORD}" \
traefik:v1.7 \
--docker \
--docker.swarmmode \
--docker.watch \
--docker.exposedbydefault=false \
--constraints=tag==traefik-public \
--entrypoints='Name:http Address::80' \
--entrypoints='Name:https Address::443 TLS' \
--acme \
--acme.email=$EMAIL \
--acme.storage=/certificates/acme.json \
--acme.entryPoint=https \
--acme.httpChallenge.entryPoint=http\
--acme.onhostrule=true \
--acme.acmelogging=true \
--logLevel=INFO \
--accessLog \
--api
Docker Node Inspect Worker Node
[
{
"ID": "nv15pwb6bie7nvz2yk9uiii10",
"Version": {
"Index": 39862
},
"CreatedAt": "2019-04-20T09:11:29.540161196Z",
"UpdatedAt": "2019-04-22T05:37:40.858645073Z",
"Spec": {
"Labels": {
"traefik-public.traefik-public-certificates": "true"
},
"Role": "worker",
"Availability": "active"
},
"Description": {
"Hostname": "node-1",
"Platform": {
"Architecture": "x86_64",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 1000000000,
"MemoryBytes": 1040351232
},
"Engine": {
"EngineVersion": "18.09.5",
"Labels": {
"provider": "digitalocean"
},
"Plugins": [
{
"Type": "Log",
"Name": "awslogs"
},
{
"Type": "Log",
"Name": "fluentd"
},
{
"Type": "Log",
"Name": "gcplogs"
},
{
"Type": "Log",
"Name": "gelf"
},
{
"Type": "Log",
"Name": "journald"
},
{
"Type": "Log",
"Name": "json-file"
},
{
"Type": "Log",
"Name": "local"
},
{
"Type": "Log",
"Name": "logentries"
},
{
"Type": "Log",
"Name": "splunk"
},
{
"Type": "Log",
"Name": "syslog"
},
{
"Type": "Network",
"Name": "bridge"
},
{
"Type": "Network",
"Name": "host"
},
{
"Type": "Network",
"Name": "macvlan"
},
{
"Type": "Network",
"Name": "null"
},
{
"Type": "Network",
"Name": "overlay"
},
{
"Type": "Volume",
"Name": "local"
}
]
},
"TLSInfo": {
"TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBazCCARCgAwIBAgIUJaJpxCmObYclnve1gMoDgqpbHpYwCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMTkwNDIwMDEwMDAwWhcNMzkwNDE1MDEw\nMDAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABKZ3tuUh1fbvsBrRxCr/2QpK42UXKH114Y5xUNjCdoVL7sDNJnPqHhGasbXZ\ncuYSf4oFPXau1Euqyo/lHFcn0TqjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBQLUasL0NneSvfzCIyelreL3Zl8GDAKBggqhkjO\nPQQDAgNJADBGAiEAuPDayJHm8fZEf1yYzS6CtrY/XuRYZK8NuNfG8Xsqs9YCIQDI\nPs6g4c65XPS7Gn931JEC/Qi7Zlu+ccMHy+Eup5SHsQ==\n-----END CERTIFICATE-----\n",
"CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
"CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEpne25SHV9u+wGtHEKv/ZCkrjZRcofXXhjnFQ2MJ2hUvuwM0mc+oeEZqxtdly5hJ/igU9dq7US6rKj+UcVyfROg=="
}
},
"Status": {
"State": "ready",
"Addr": "worker-machine-ip"
}
}
]
Docker Node Inspect Manager Node
{
"ID": "fw4k9vgz7y3929i731o7fll7d",
"Version": {
"Index": 39771
},
"CreatedAt": "2019-04-20T01:04:39.695763427Z",
"UpdatedAt": "2019-04-22T05:06:58.875733526Z",
"Spec": {
"Labels": {
"traefik-public.traefik-public-certificates": "true"
},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "shijie-master",
"Platform": {
"Architecture": "x86_64",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 1000000000,
"MemoryBytes": 2097283072
},
"Engine": {
"EngineVersion": "18.09.4",
"Plugins": [
{
"Type": "Log",
"Name": "awslogs"
},
{
"Type": "Log",
"Name": "fluentd"
},
{
"Type": "Log",
"Name": "gcplogs"
},
{
"Type": "Log",
"Name": "gelf"
},
{
"Type": "Log",
"Name": "journald"
},
{
"Type": "Log",
"Name": "json-file"
},
{
"Type": "Log",
"Name": "local"
},
{
"Type": "Log",
"Name": "logentries"
},
{
"Type": "Log",
"Name": "splunk"
},
{
"Type": "Log",
"Name": "syslog"
},
{
"Type": "Network",
"Name": "bridge"
},
{
"Type": "Network",
"Name": "host"
},
{
"Type": "Network",
"Name": "macvlan"
},
{
"Type": "Network",
"Name": "null"
},
{
"Type": "Network",
"Name": "overlay"
},
{
"Type": "Volume",
"Name": "local"
}
]
},
"TLSInfo": {
"TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBazCCARCgAwIBAgIUJaJpxCmObYclnve1gMoDgqpbHpYwCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMTkwNDIwMDEwMDAwWhcNMzkwNDE1MDEw\nMDAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABKZ3tuUh1fbvsBrRxCr/2QpK42UXKH114Y5xUNjCdoVL7sDNJnPqHhGasbXZ\ncuYSf4oFPXau1Euqyo/lHFcn0TqjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBQLUasL0NneSvfzCIyelreL3Zl8GDAKBggqhkjO\nPQQDAgNJADBGAiEAuPDayJHm8fZEf1yYzS6CtrY/XuRYZK8NuNfG8Xsqs9YCIQDI\nPs6g4c65XPS7Gn931JEC/Qi7Zlu+ccMHy+Eup5SHsQ==\n-----END CERTIFICATE-----\n",
"CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
"CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEpne25SHV9u+wGtHEKv/ZCkrjZRcofXXhjnFQ2MJ2hUvuwM0mc+oeEZqxtdly5hJ/igU9dq7US6rKj+UcVyfROg=="
}
},
"Status": {
"State": "ready",
"Addr": "manager-machine-ip"
},
"ManagerStatus": {
"Leader": true,
"Reachability": "reachable",
"Addr": "manager-machine-ip:2377"
}
}
]
Docker network inspect on traefik-public
[
{
"Name": "traefik-public",
"Id": "6655p8lsxjmhqhha3e3fbs5xz",
"Created": "2019-04-21T06:07:01.862111049Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.12.0/24",
"Gateway": "10.0.12.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"Containers": { .. },
"ConfigOnly": false,
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4109"
},
"Labels": {},
"Peers": [...]
}
]
Problem
Running services on manager node works fine, but worker nodes return Error 502 Bad Gateway

Docker-Compose Up Works but Eb Local Run does not

Attached is my docker-compose file. Its a very simple project with a database and phpmyadmin to access it.
web:
build: ./docker_web/
links:
- db
ports:
- "80:80"
volumes:
- "./docker_web/www/:/var/www/site"
db:
image: mysql:latest
restart: always
volumes:
- "./.data/db:/var/lib/mysql"
environment:
MYSQL_ROOT_PASSWORD: ^^^^
MYSQL_DATABASE: electionbattle
MYSQL_USER: admin
MYSQL_PASSWORD: ^^^^
phpmyadmin:
image: phpmyadmin/phpmyadmin
container_name: phpmyadmin
environment:
- PMA_ARBITRARY=1
restart: always
ports:
- 8081:80
volumes:
- /sessions
links:
- db
If I run this it works fine. I created the equivlent for Amazon Docker Run for Elastic Beanstalk and it starts up but for some reason it can't find the volume that is holding my persisted database data in the .data folder.
I've tried changing .data to just data no luck.
Also I get a weird error when trying to do eb deploy
2016-09-24 19:56:10 UTC-0700 ERROR ECS task stopped due to: Essential container in task exited. (db: CannotCreateContainerError: API error (500): create ./.data/db/: "./.data/db/" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed web: phpmyadmin: )
I have no idea how to fix this error or why its happening. Any ideas?
Oops forgot to add my amazon file :).
{
"AWSEBDockerrunVersion": 2,
"volumes": [
{
"name": "web",
"host": {
"sourcePath": "./docker_web/www/"
}
},
{
"name": "db",
"host": {
"sourcePath": "./data/db/"
}
}
],
"containerDefinitions": [
{
"name": "web",
"image": "197984628663.dkr.ecr.us-west-1.amazonaws.com/electionbattleonline",
"memory": 200,
"essential": true,
"mountPoints": [
{
"sourceVolume": "web",
"containerPath": "/var/www/site",
"readOnly": false
}
],
"links": [
"db"
],
"portMappings": [
{
"hostPort": 80,
"containerPort": 80
}
]
},
{
"name": "db",
"image": "mysql:latest",
"environment": [
{
"name": "MYSQL_ROOT_PASSWORD",
"value": "^^^^"
},
{
"name": "MYSQL_DATABASE",
"value": "electionbattleonline"
},
{
"name": "MYSQL_USER",
"value": "admin"
},
{
"name": "MYSQL_PASSWORD",
"value": "^^^^"
}
],
"portMappings": [
{
"hostPort": 3306,
"containerPort": 3306
}
],
"mountPoints": [
{
"sourceVolume": "db",
"containerPath": "/var/lib/mysql",
"readOnly": false
}
],
"essential": true,
"memory": 200
},
{
"name": "phpmyadmin",
"image": "phpmyadmin/phpmyadmin",
"environment": [
{
"name": "PMA_ARBITRARY",
"value": "1"
}
],
"essential": true,
"memory": 128,
"links": [
"db"
],
"portMappings": [
{
"hostPort": 8081,
"containerPort": 80
}
]
}
]
}
Don't use relative paths.
Use eb local run to test before deploying, it will help you solve deployment issues. Your Dockerrun.aws.json file will be converted into a docker-compose.yml file and started using a local copy of docker.
You can find the generated docker-compose.yml in your project directory at the path .elasticbeanstalk/docker-compose.yml. You will notice that your volumes are missing from the docker-compose config file.
To fix this change your volumes to:
"volumes": [
{
"name": "web",
"host": {
"sourcePath": "/var/app/current/docker_web/www/"
}
},
{
"name": "db",
"host": {
"sourcePath": "/var/app/current/data/db/"
}
}
],
and create the directories in your app, then "eb local run" will correctly convert them.
eb deploy should now work correctly.

Resources