For the life of me I can't get to the NiFi Web UI. It makes me hate security.
TLDR; I can't find the right way to start NiFi in a docker container and still access the UI. Here's what I've tried (for 8 hours):
docker run --name nifi \
-p 8080:8080 \
-d \
apache/nifi:latest
I go to localhost:8080/nifi - timeout. Same on 127.0.0.1.
docker inspect nifi - IP Gateway is 172.20.0.1 with actual IP of 172.0.0.2. Invalid Host Header and timeout, respectively.
Start randomly trying stuff:
# I tried localhost, 0.0.0.0, various IP addresses
docker run --name nifi \
-p 8080:8080 \
-e NIFI_WEB_HTTP_HOST=${hostname-here}
-d \
apache/nifi:latest
I also built a full docker-compose.yml for my diminishingly-possible stack. Everything thing works except for:
nifi:
image: apache/nifi:latest
hostname: nifi
depends_on:
- zookeeper
- broker
- schema_registry
- nifi-registry
ports:
- "8080:8080"
No changes. Can you help me?
Updates 1
I used the docker-compose.yml file from the repo linked in comments below; thank you #Chaffelson. Still dealing with timeout on localhost. So I spun up a droplet with docker-machine.
The services start fine, and logs indicate Jetty server is up for both NiFi Registry and NiFi. I can access NiFi registry # <host ip>:18080/nifi-registry exactly like I can on my local machine.
I can't access <host ip>8080/nifi - I get invalid host header response.
So I added to docker-compose.yml:
environment:
# Tried both with and without quotes
NIFI_WEB_HTTP_HOST: "<host-ip>"
Jetty server fails to start. Insight?
Updates 2
From the logs, using just docker run --name nifi -p 8080:8080 -d apache/nifi:1.5.0:
[NiFi Web Server-24] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [45.55.36.15:8080] different from web hostname [348146fc286f(:8080)]. Overriding to [348146fc286f:8080/nifi] where 45.55.36.15 is the host ip.
This is the crux of my problem.
Updates 3
I disabled ufw (firewall) on my local machine. I can now access nifi via localhost:8080. No progress on actually accessing on a remote host (which is the point of all this).
Sorry to hear you are having trouble with this. In Apache NiFi 1.5.0, the stricter host header protection was enabled to prevent host header poisoning attacks. Unfortunately, we have seen that this was not documented sufficiently for users who were not familiar with the configuration. In response, we have made changes that are currently in master and will be included in the upcoming 1.6.0 release:
a new property nifi.web.proxy.host was added to nifi.properties which accepts a comma-separated list of valid host headers independent of the Jetty hostname
the Docker configuration has been updated to allow proxy whitelisting from the run command
the host header protection is only enforced on "secured" NiFi instances. This should make it much easier for users to quickly deploy sandbox environments like you are doing in this case
For an immediate fix, this command should work:
docker run --name nifi \
-p 8080:8080 \
-e NIFI_WEB_HTTP_HOST=172.20.0.1
-d \
apache/nifi:latest
You can also intercept the requests using a Chrome extension like ModHeader to override the Host header and verify that it works when it matches the expected host. Along with Daniel's excellent additions, this should help you until the next version is released.
I use this and similar docker compose files for my automated NiFi Python client testing. It looks superficially similar to yours, and works perfectly well on both Ubuntu (Travis-CI) and my local MacBook pro for myself.
I suggest you try running this file as a known-good configuration, and also examine 'docker logs -f nifi' for the above to see if your environment is throwing errors on startup.
The environment variables for NIFI_WEB_HTTP_HOST and NIFI_WEB_HTTP_PORT are for when you are accessing Docker nifi on a port other than 8080, so that you don't get the host-headers blocker. I contributed these modifications to the project recently, so if you are having trouble with them I would like to know so I can fix it.
I had the same issue, I was not able to access the web ui remotely. Turns out the firewall issue. Disabling the firewalld & adding a custom firewall rule to allow docker network with port should solve the issue.
Here is my docker-compose.yml:
in docker use this. It fixed my problem.
--net=host
so that docker can reduce the internal port forwarding path.
Related
I'm a beginner in this docker world and as it is very suffering to set all these 'localhost' thingy with apache and stuff, it's the same with docker.
I don't know if it's me because but i tried with the help of other post to solve my problem but after several hour i give up and i ask for your help, because some post are just not comprehensible for me ( post that includes bridges stuff NAT iptables docker-machine, etc )
After several hour i'm just simply trying to access apache website on localhost:5000 on windows who is launched with service apache2 start within a docker, and if i do w3m localhost in this docker i can see it running.
But when i'm trying to access it with a browser no response.
I also tried this command :
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' bce97a49b68c
172.17.0.2
The adress with :5000 don't have an access, i even it put in the hosts file. No success.
If someone has the last solution for this problem, it's seems there are plenty and everything seems to be so simple in blog of article (i even tried something with docker-composer, it deleted docker i had to reinstall the whole thing)
I'm a little unsure what you're asking, but it seems like you may need to expose your ports. When running something in Docker, it runs in its own little box unconnected to the outside world - the rest of your machine. If you want to connect ports - say to access a web server running inside a Docker container, you need to use the -p or --publish option when running your Docker container. There are similar commands for mounting drives and such.
Here's an example from the database I run locally in Docker:
docker run \
--publish=7474:7474 \
--volume=/home/me/logs:/logs \
--env=NEO4J_AUTH=none \
neo4j:4.2.
This says:
Allow the outside system to access port 7474 inside the Docker container from the port 7474 outside the docker container
Mount the outside system's /home/me/logs folder as /logs inside the Docker container
Set the environment variable NEO4J_AUTH inside the Docker container to the value none
I'm trying to stand up a temporary NiFi server to support a proof of concept demo for a customer. For these types of short lived servers I like to use Docker when possible. I'm able to get the NiFi container up and running with out any issues but I can't figure out how to access its UI from the browser on a remote host. I've tried the following docker run variations:
docker run --name nifi \
-p 8080:8080 \
-d \
apache/nifi:latest
docker run --name nifi \
-p 8080:8080 \
-e NIFI_WEB_HTTP_PORT='8080' \
-d \
apache/nifi:latest
docker run --name nifi \
-p 8080:8080 \
-e NIFI_WEB_HTTP_HOST=${hostname-here} \
-e NIFI_WEB_HTTP_PORT='8080' \
-d \
apache/nifi:latest
My NiFi version is 1.8.0. I'm fairly certain that my problems are related to the host-headers blocker feature added to version 1.5.0. I've seen a few questions similar to mine but no solutions.
Is it possible to access the NiFi UI from a remote host after version 1.5.0?
Can host-headers blocker be disabled for a non-prod demo?
Would a non-Docker install on my server present the same host-headers blocker issues?
Should a use 1.4 for my demo and save myself a headache?
While there was a bug around 1.5.0 surrounding the host headers in Docker that issue was resolved and, additionally, the host header check now is only enforced for secured environments (you will see a note about this in the logs on container startup).
The commands you provide in your question are all workable for accessing NiFi on the associated mapped port in each example and I have verified this in 1.6.0, 1.7.0, and 1.8.0. You may want to evaluate the network security settings of your remote machine in question (cloud provided instances, for example. will typically require explicit security groups exposing ports).
I had the same issue, I was not able to access the web ui remotely. Turns out the firewall issue. Disabling the firewalld & adding a custom firewall rule to allow docker network with port should solve the issue.
The docker-compose.yml is shared here
In short - Can I run Elasticsearch & Dropwizard app in separate docker containers and allow them to see each other?
I am running Elasticsearch 6.2.2 from Docker (on mac). using the command..
docker run -p 9200:9200 -p 9300:9300 -e "network.host=0.0.0.0" \
-e "http.port=9200" -e "discovery.type=single-node" \
docker.elastic.co/elasticsearch/elasticsearch:6.2.2
I can access Elasticsearch (POST & GET) fine using Postman directly on mac e.g
localhost:9200/testindex/_search
However when running a Dropwizard application from a different docker image which accesses the docker Elasticsearch instance, I get connection refused using same host and port (localhost 9200).
I have no problems at all when running the Dropwizard app direct from an IDE, only when its running from a docker image and accessing ES from a different image.
docker image -p 8080:8080 -p 8081:8081 testapp
Has anyone else had similar issues or solved this in the past?
I'm assuming it 'network' related and that connecting to localhost from one docker image will not be map to the other docker image
The issue you are facing is in the url you pass to the dropwizard container. As a container by default has its own networking, a value of localhost means the dropwizard container itself, not what you see as your local host from outside the container.
Please have a look at docker networking, how you can link two containers by name. I would suggest to check out docker-compose for multi-container setups on a local machine.
What would also work (but is not good practice) is to pass the dropwizard container the ip of your machine as elasticsearch host because you created a port mapping from your host into the elasticsearch container. But better have a look at compose to do it as it is supposed to be done.
For details how to use compose please have a look at this answer with a similar example.
For using traefik as a reverse-proxy in front of a Docker container whose dynamic IP address might change over time, traefik comes with a docker backend. All examples that I could find for setting this up follow the same pattern:
First, start traefik in docker mode without an extra configuration file, activate host network mode (optional, so that traefik can see all Docker networks on the host if required) and mount the Docker unix socket so that traefik can listen to container starts and stops.
docker run --rm -p 80:80 --net=host --name traefik-reverse-proxy -v /dev/null/traefik.toml:/etc/traefik/traefik.toml -v /var/run/docker.sock:/var/run/docker.sock traefik --docker --loglevel debug
Then, start another container and set at least the following labels:
traefik.backend: "some-backend-name"
traefik.frontend.rule: "Host: localhost; Method: GET" (or whatever your rules are)
traefik.port: 80 (or whatever port your container exposes internally)
Example:
docker run --rm --name nginx -l traefik.backend="some-backend-name" -l traefik.frontend.rule="Host: localhost; Method: GET" -l traefik.port="80 nginx
Then, doing a curl localhost, one can see in the logs of the traefik container that it took the request and routed it to the NGINX container.
So far, so good... however, I do not like the fact that I have to configure my reverse-proxy forwarding rules (e.g. forward Host: some.host.name to container xxx) within the application itself (where my docker-compose files setting up the containers, labels etc. are usually located). Rather, I would like to separate this from the application and configure it as part of traefik's configuration instead.
Is this possible somehow? What I tried is leaving out the traefik.frontend.rule label from the example nginx container and instead mount the following configuration file for traefik:
[frontends]
[frontends.frontend1]
backend = "some-backend-name"
[frontends.frontend1.routes.test_1]
rule = "Host: localhost; Method: GET"
The startup command for traefik thus becomes:
docker run --rm -p 80:80 --net=host --name traefik-reverse-proxy -v $PWD/traefik.toml:/etc/traefik/traefik.toml -v /var/run/docker.sock:/var/run/docker.sock traefik --docker --loglevel debug
However, this does not seem to attach the frontend rule from the config file with the backend label from the nginx container. curl localhost now returns a 404 / Not found error.
the watch flag seems only works under the condition of rule.toml changed first time.
In your case, i suggest you write a service to update your rule in etcd or zookeeper. the service read etcd changes and update traefik configure in etcd.
This is likely an order of operations issue. Enabling debug logging in config (debug = true) shows that traefik is parsing the config file frontend rules first, and only later generating frontends and backends based on what's running in docker.
This means that the docker backends don't exist when the frontends from config are created, and it throws and error.
One solution is to put your rules config in a seperate file (e.g. rules.toml as shown in the docs) and add the watch = true directive to your config. This means that the frontend rules you define there will be updated after the backends from docker are generated.
We should probably submit a bug for this, because it's not exactly desirable functionality.
I have multiple docker containers on a single machine. On each container is running a process and a web server that provides an API for the process.
My question is, how can I access the API from my browser when the default port is 80? To be able to access the web server inside docker container I do the following:
sudo docker run -p 80:80 -t -i <yourname>/<imagename>
This way I can do from my computers terminal:
curl http://hostIP:80/foobar
But how to handle this with multiple containers and multiple web servers?
You can either expose multiple ports, e.g.
docker run -p 8080:80 -t -i <yourname>/<imagename>
docker run -p 8081:80 -t -i <yourname1>/<imagename1>
or put a proxy ( nginx, apache, varnish, etc.) in front of your API containers.
Update:
The easiest way to do a proxy would be to link it to the API containers, e.g. having apache config
RewriteRule ^api1/(.*)$ http://api1/$1 [proxy]
RewriteRule ^api2/(.*)$ http://api2/$1 [proxy]
you may run your containers like this:
docker run --name api1 <yourname>/<imagename>
docker run --name api2 <yourname1>/<imagename1>
docker run --link api1:api1 --link api2:api2 -p 80:80 <my_proxy_container>
This might be somewhat cumbersome though if you need to restart the api containers as the proxy container would have to be restarted either (links are fairly static at docker yet). If this becomes a problem, you might look at approaches like fig or autoupdated proxy configuration : http://jasonwilder.com/blog/2014/03/25/automated-nginx-reverse-proxy-for-docker/ . The later link also shows proxying with nginx.
Update II:
In a more modern versions of docker it is possible to use user defined network instead of the links shown above to overcome some of the inconveniences of the deprecated link mechanism.
Only a single process is allowed to be bound to a port at a time. So running multiple containers means each will be exposed on a different port number. Docker can do this automatically for you by using the "-P" attribute.
sudo docker run -P -t -i <yourname>/<imagename>
You can use the "docker port" and "docker inspect" commands to see the actual port number allocated to each container.