I have 2 working containers for Hadoop HDFS (I have built a Dockerfile for them starting from the one here) and need to add a HIVE instance to the set.
I cannot find a good working image to use (it would be nice to have version 2.3.4).
Is there anything you suggest, easy to add, that I could use?
Thank you!
Edit:
Here is a try I did:
hive-server:
container_name: hive-server
image: bde2020/hive:2.3.2-postgresql-metastore
depends_on:
- hadoop-namenode
env_file:
- ./hive_build/hadoop-hive.env
environment:
HIVE_CORE_CONF_javax_jdo_option_ConnectionURL: "jdbc:postgresql://hive-metastore/metastore"
SERVICE_PRECONDITION: "hive-metastore:9083"
ports:
- "10000:10000"
restart: unless-stopped
hive-metastore:
container_name: hive-metastore
image: bde2020/hive:2.3.2-postgresql-metastore
depends_on:
- hive-server
- hive-metastore-postgresql
env_file:
- ./hive_build/hadoop-hive.env
command: /opt/hive/bin/hive --service metastore
environment:
SERVICE_PRECONDITION: "hadoop-namenode:50070 hadoop-datanode1:50075 hive-metastore-postgresql:5432"
ports:
- "9083:9083"
restart: unless-stopped
hive-metastore-postgresql:
container_name: hive-metastore-postgresql
image: bde2020/hive-metastore-postgresql:2.3.0
ports:
- "5433:5432"
restart: unless-stopped
but when I enter it and try to connect I get an error:
docker exec -it hive-server bash
/opt/hive/bin/beeline -u jdbc:hive2://localhost:10000
error:
19/05/03 09:13:46 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Related
How to take place name resolution. net::ERR_NAME_NOT_RESOLVED error is occurred, when using docker compose and docker network
Hello.
When I use docker compose and docker network to create frontend server and backend server and db server, net::ERR_NAME_NOT_RESOLVED is occurred in google chrome console.
Below file is my docker-compose.yml.
I assured that the communication of backend-container to frontend-container is no problem, because I attached to frontend-container ( using command of docker exec -it frontend-container /bin/bash) and confirmed communication ( using command of curl backend-container:4000).
However, when I attempted to communicate backend-container from web browser (frontend application), net::ERR_NAME_NOT_RESOLVED is occurred in chrome developer console.
I think that name resolution is not taking place from web browser.
However I have no idea how to name resolution in this case.
Please help me and teach me how to resolve name in this case.
Thank you.
version: "3.5"
services:
frontend:
build:
context: "./project01_frontend"
dockerfile: "Dockerfile.prod"
container_name: frontend-container
restart: always
tty: true
ports:
- "80:80"
env_file:
- "./project01_frontend/.env"
depends_on:
- "db"
- "backend"
backend:
build:
context: "./project01_backend"
dockerfile: "Dockerfile.dev"
container_name: backend-container
restart: always
tty: true
expose:
- "4000"
env_file:
- "./project01_backend/.env"
depends_on:
- "db"
command: ["./wait-for-db-container.sh", "yarn", "start"]
db:
image: mysql:5.7.34
command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
container_name: db-container
volumes:
- ./project01_backend/initdb.d:/docker-entrypoint-initdb.d
environment:
MYSQL_DATABASE: "typeorm_db"
MYSQL_ROOT_PASSWORD: "password"
TZ: "Asia/Tokyo"
restart: "always"
expose:
- "3306"
I have laravel app that lives in docker, and I want to integrate elasticsearch to my app
That is how my docker-compose.yaml looks
version: '3'
services:
laravel:
build: ./docker/build
container_name: laravel
restart: unless-stopped
privileged: true
ports:
- 8084:80
- "22:22"
volumes:
- ./docker/settings:/settings
- ../2agsapp:/var/www/html
# - vendor:/var/www/html/vendor
- ./docker/temp:/backup
- composer_cache:/root/.composer/cache
environment:
- ENABLE_XDEBUG=true
links:
- mysql
mysql:
image: mariadb:10.2
container_name: mysql
volumes:
- ./docker/db_config:/etc/mysql/conf.d
- ./db:/var/lib/mysql
ports:
- "8989:3306"
environment:
- MYSQL_USER=dev
- MYSQL_PASSWORD=dev
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=laravel
command: --innodb_use_native_aio=0
phpmyadmin:
container_name: pma_laravel
image: phpmyadmin/phpmyadmin:latest
environment:
- MYSQL_USER=dev
- MYSQL_ROOT_PASSWORD=root
- MYSQL_PASSWORD=dev
- MYSQL_DATABASE=laravel
- PMA_HOST=mysql
ports:
- 8083:80
links:
- mysql
es:
image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
ports:
- "9200:9200"
- "9300:9300"
environment:
- discovery.type=single-node
volumes:
storage:
composer_cache:
I run docker-compose up -d and then got really strange issue
If I execute curl localhost:9200 inside laravel container it returns this message Failed to connect to localhost port 9200: Connection refused
But if I wull run curl localhost:9200 out of the docker it returns expected response
Maybe I don't understand how it works, hope someone will help me
when you want to access another container within some container you should use the container name, not localhost.
If you are inside laravel and want to access Elasticsearch you should:
curl es:9200
Since you mapped the 9200 port to localhost (ports section in docker-compose) this port is available from your local machine as well, that's why curling from local machine to 9200 works.
I try to setup zipkin, elasticsearch, prometheus and grafana with docker-compose.yml
When I run dockers, see in the log:
dependencies_zipkin | 19/09/30 14:37:09 ERROR NetworkClient: Node [172.28.0.2:9200] failed (java.net.ConnectException: Connection refused (Connection refused)); no other nodes left - aborting...
I'm on MacOS X with docker 2.1.0.3
the content of my docker-compose.yml is this one:
version: '3.7'
services:
storage:
image: openzipkin/zipkin-elasticsearch7
container_name: elasticsearch
ports:
- "9200:9200"
environment:
- "xpack.security.enabled=false"
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
restart: unless-stopped
prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- $PWD/prometheus:/etc/prometheus/
- /tmp/prometheus:/prometheus/data:rw
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/usr/share/prometheus/console_libraries'
- '--web.console.templates=/usr/share/prometheus/consoles'
ports:
- "9090:9090"
restart: unless-stopped
zipkin:
image: openzipkin/zipkin
container_name: zipkin
depends_on:
- dependencies
- storage
environment:
- "STORAGE_TYPE=elasticsearch"
- "ES_HOSTS=storage"
ports:
- "9411:9411"
restart: unless-stopped
grafana:
image: grafana/grafana
container_name: grafana
ports:
- "3000:3000"
restart: unless-stopped
dependencies:
image: openzipkin/zipkin-dependencies
container_name: dependencies_zipkin
depends_on:
- storage
environment:
- "STORAGE_TYPE=elasticsearch"
- "ES_HOSTS=storage"
When I connect to localhost:9200, I see that elasticsearch is working fine and on port 9411, zipkin is deployed but I have the error:
ERROR: cannot load service names: server error (Service Unavailable)(due to the network error
In the log, I have this information:
105 ^[[35mdependencies_zipkin |^[[0m 19/09/30 14:45:20 ERROR NetworkClient: Node [172.28.0.2:9200] failed (java.net.ConnectException: Connection refused (Connection refused)); no other nodes left - aborting...
and this one
^[[31mzipkin |^[[0m java.lang.IllegalStateException: couldn't connect any of [Endpoint{storage:80, ipAddr=172.28.0.2, weight=1000}]
Any idea?
UPDATE
by using mysql it is working fine, so the problem is at the level of elastic search.
I tried alsoo by using
"STORAGE_PORT_9200_TCP_ADDR=127.0.0.1"
but the issue still occurs.
UPDATE
As mention is the solution gave by Brian, I have to use:
ES_HOSTS=http://storage:9300
the key is on port, I was using the port 9200
The error disappear between zipkin and es but still occurs between es and zipkin-dependencies.
The problem lies in your ES_HOSTS variable, from the docs here:
ES_HOSTS: A comma separated list of elasticsearch base urls to connect to ex. http://host:9200.
Defaults to "http://localhost:9200".
So you will need: ES_HOSTS=http://storage:9200
Finally I have this file:
version: '3.7'
services:
storage:
image: openzipkin/zipkin-elasticsearch7
container_name: elasticsearch
ports:
- 9200:9200
zipkin:
image: openzipkin/zipkin
container_name: zipkin
environment:
- STORAGE_TYPE=elasticsearch
- "ES_HOSTS=elasticsearch:9300"
ports:
- 9411:9411
depends_on:
- storage
dependencies:
image: openzipkin/zipkin-dependencies
container_name: dependencies
entrypoint: crond -f
depends_on:
- storage
environment:
- STORAGE_TYPE=elasticsearch
- "ES_HOSTS=elasticsearch:9300"
- "ES_NODES_WAN_ONLY=true"
prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- $PWD/prometheus:/etc/prometheus/
- /tmp/prometheus:/prometheus/data:rw
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/usr/share/prometheus/console_libraries'
- '--web.console.templates=/usr/share/prometheus/consoles'
ports:
- "9090:9090"
grafana:
image: grafana/grafana
container_name: grafana
depends_on:
- prometheus
ports:
- "3000:3000"
Main differences are the usage of
"ES_HOSTS=elasticsearch:9300"
instead of
"ES_HOSTS=storage:9300"
and in the dependencies configuration I add the entrypoint in dependencies:
entrypoint: crond -f
This one is really the key to not have the exception when I start docker-compose.
To solve this issue, I check the this project: https://github.com/openzipkin/docker-zipkin
The remaining question is: why do I need to use entrypoint: crond -f
I am new to the Docker World.I have got my db and server connected. Now i want to connect my frontend with server. I an unable to use network mode with link. Without using network mode, i am unable to view my frontend site on localhost:8080.
How can i connect my front and server?
version: '2.2'
services:
front:
build: ../Devops/operations-portal
ports:
- "127.0.0.1:8080:8080"
network_mode: "host"
volumes:
- ../Devops/operations-portal
links:
- server
server:
build: ../Devops/data-mining-backend
ports:
- "5000:5000"
volumes:
- ../Devops/data-mining-backend/:/app
links:
- db
db:
image: mongo:3.6.5
ports:
- "27017:27017"
volumes:
- '../database:/data/db'
command: mongod --smallfiles --logpath=/dev/null # --quiet
I am trying to build my airflow using docker and rabbitMQ. I am using rabbitmq:3-management image. And I am able to access rabbitMQ UI, and API.
In airflow I am building airflow webserver, airflow scheduler, airflow worker and airflow flower. Airflow.cfg file is used to config airflow.
Where I am using broker_url = amqp://user:password#127.0.0.1:5672/ and celery_result_backend = amqp://user:password#127.0.0.1:5672/
My docker compose file is as follows
version: '3'
services:
rabbit1:
image: "rabbitmq:3-management"
hostname: "rabbit1"
environment:
RABBITMQ_ERLANG_COOKIE: "SWQOKODSQALRPCLNMEQG"
RABBITMQ_DEFAULT_USER: "user"
RABBITMQ_DEFAULT_PASS: "password"
RABBITMQ_DEFAULT_VHOST: "/"
ports:
- "5672:5672"
- "15672:15672"
labels:
NAME: "rabbitmq1"
webserver:
build: "airflow/"
hostname: "webserver"
restart: always
environment:
- EXECUTOR=Celery
ports:
- "8080:8080"
depends_on:
- rabbit1
command: webserver
scheduler:
build: "airflow/"
hostname: "scheduler"
restart: always
environment:
- EXECUTOR=Celery
depends_on:
- webserver
- flower
- worker
command: scheduler
worker:
build: "airflow/"
hostname: "worker"
restart: always
depends_on:
- webserver
environment:
- EXECUTOR=Celery
command: worker
flower:
build: "airflow/"
hostname: "flower"
restart: always
environment:
- EXECUTOR=Celery
ports:
- "5555:5555"
depends_on:
- rabbit1
- webserver
- worker
command: flower
I am able to build images using docker compose. However, I am not able to connect my airflow scheduler to rabbitMQ. I am getting following error:
consumer: Cannot connect to amqp://user:**#localhost:5672//: [Errno
111] Connection refused.
I have tried using 127.0.0.1 and localhost both.
What I am doing wrong ?
From within your airflow containers, you should be able to connect to the service rabbit1. So all you need to do is to change amqp://user:**#localhost:5672//: to amqp://user:**#rabbit1:5672//: and it should work.
Docker compose creates a default network and attaches services that do not explicitly define a network to it.
You do not need to expose the 5672 & 15672 ports on rabbit1 unless you want to be able to access it from outside the application.
Also, generally it is not recommended to build images inside docker-compose.
I solved this issue by installing rabbitMQ server into my system with command sudo apt install rabbitmq-server.