Docker & Hive - Port 50070 ports on Windows permission denied - docker

Question

I want to setup a local hive server and found this repo:
https://github.com/big-data-europe/docker-hive
This is the yaml file I use.
version: "3"
services:
namenode:
image: bde2020/hadoop-namenode:2.0.0-hadoop2.7.4-java8
volumes:
- namenode:/hadoop/dfs/name
environment:
- CLUSTER_NAME=test
env_file:
- ./hadoop-hive.env
ports:
- "50070:50070"
datanode:
image: bde2020/hadoop-datanode:2.0.0-hadoop2.7.4-java8
volumes:
- datanode:/hadoop/dfs/data
env_file:
- ./hadoop-hive.env
environment:
SERVICE_PRECONDITION: "namenode:50070"
ports:
- "50075:50075"
hive-server:
image: bde2020/hive:2.3.2-postgresql-metastore
env_file:
- ./hadoop-hive.env
environment:
HIVE_CORE_CONF_javax_jdo_option_ConnectionURL: "jdbc:postgresql://hive-metastore/metastore"
SERVICE_PRECONDITION: "hive-metastore:9083"
ports:
- "10000:10000"
hive-metastore:
image: bde2020/hive:2.3.2-postgresql-metastore
env_file:
- ./hadoop-hive.env
command: /opt/hive/bin/hive --service metastore
environment:
SERVICE_PRECONDITION: "namenode:50070 datanode:50075 hive-metastore-postgresql:5432"
ports:
- "9083:9083"
hive-metastore-postgresql:
image: bde2020/hive-metastore-postgresql:2.3.0
presto-coordinator:
image: shawnzhu/prestodb:0.181
ports:
- "8080:8080"
volumes:
namenode:
datanode:
Error:
Error starting userland proxy: Bind for 0.0.0.0:50075: unexpected error Permission denied
The ports >50000 are blocked on windows, I don´t have admin rights on my company pc, so I tried to map the ports like this:
ports:
- "40070:50070"
environment:
SERVICE_PRECONDITION: "namenode:40070 datanode:40075 hive-metastore-postgresql:5432"
This will let me get the Container started, but the container seem not to be able to communicate.
hive-metastore_1 | [1/100] check for namenode:40070...
hive-metastore_1 | [1/100] namenode:40070 is not available yet
hive-metastore_1 | [1/100] try in 5s once again ...
956a5237dbe2_docker-hive_datanode_1 | [4/100] check for namenode:40070...
956a5237dbe2_docker-hive_datanode_1 | [4/100] namenode:40070 is not available yet
I tried to change both ports:
ports:
- "40070:40070"
This will not work, because some IPs seem to be hardcoded:
ded7410db1b9_docker-hive_namenode_1 | 21/10/08 12:39:05 INFO hdfs.DFSUtil: Starting Web-server for hdfs at: http://0.0.0.0:50070
ded7410db1b9_docker-hive_namenode_1 | 21/10/08 12:39:05 INFO http.HttpServer2: Jetty bound to port 50070
Does anyone know how to get this running?

Answer 1

With the following:
ports:
- "40070:50070"
all you are doing is directing traffic from host port 40070 to container port 50070.
So to access "namenode" from the host machine for example:
localhost:40070
And to access "namenode" inside the compose network:
namenode:50070
Service precondition with BDE checks the container and the port repeatedly to see if the service is running before setting up its own services to ensure things are ready first. You have not changed the port running on the container, so your containers should still communicate via port 50070.
You have incorrectly changed the precondition to scan instead for your host port 40070, whereas it should look for the internal network container port 50070 regardless of host port.
Change it to the following:
ports:
- "40070:50070"
environment:
SERVICE_PRECONDITION: "namenode:50070 datanode:50075 hive-metastore-postgresql:5432"
You can change the operating ports on Hive etc. with the environmental variable file provided, but you shouldn't need to. Exposing host port 40070 to container port 50070 has no impact on the operation of the docker services.