I am not running kafka using confluent platform packages
I am running kafka inside docker.
I am trying to capture real time twitter data.
My Docker file looks like
version: '2'
services:
# this is our kafka cluster.
kafka-cluster:
image: landoop/fast-data-dev:cp3.3.0
environment:
ADV_HOST: 127.0.0.1 # Change to 192.168.99.100 if using Docker Toolbox
RUNTESTS: 0 # Disable Running tests so the cluster starts faster
CONNECT_PLUGIN_PATH: '/data/connectors'
ports:
- 2181:2181 # Zookeeper
- 3030:3030 # Landoop UI
- 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka Connect ports
- 9581-9585:9581-9585 # JMX Ports
- 9092:9092 # Kafka Broker
volumes:
- ~/Documents/Docker/Kafka/code/jcustenborder-kafka-connect-twitter-0.3.34:/data/connectors
Twitter connector I am using here is jcustenborder-kafka-connect-twitter-0.3.34
it is not working to me I dont see twitter connector in kafka
# Basic configuration for our connector
name=source-twitter-distributed
connector.class=com.github.jcustenborder.kafka.connect.twitter.TwitterSourceConnector
tasks.max=1
topic=demo-4-twitter
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
# Twitter connector specific configuration
twitter.consumerkey=xxx
twitter.consumersecret=xxx
twitter.token=xxx
twitter.secret=xxx
track.terms=programming,java,kafka,scala
language=en
I get error connector class com.github.jcustenborder.kafka.connect.twitter.TwitterSourceConnector
not found
I downloaded the plugin to path /Documents/Docker/Kafka/code/jcustenborder-kafka-connect-twitter-0.3.34
docker-compose yml exist in /Documents/Docker/Kafka/code/
Related
I am using a kafka cluster of three kafka nodes, and it is running on zookeeper cluster of nodes which is already up and running.
Docker-compose is used with kafka docker image version 2.2.0 and zookeeper docker image version 3.5.
Now my plan is to use the upgraded version of kafka 3.x, where it is claimed that zookeeper is not required (may be explicitly not required). I was under impression that zookeeper will be started automatically. But I saw that Zookeeper is embedded in kafka and so I needed to start it also explicitly.
For that, I have to repackage kafka 3.x into two separate docker images.
First image ZOOKEEPER_IMAGE, will call zookeeper’s zoo-start.sh through Docker file’s CMD command and it will set the zoo specific parameters.
Second image KAFKA_IMAGE will on contrary, will call kafka’s start.sh through Docker file’s CMD command and it will set the kafka specific parameters.
CMD ["/start.sh"]
Also I can see that many kafka parameters are changed and at present I am getting this exception.
KafkaException: Unable to parse PLAINTEXT://127.0.0.1: to a broker endpoint -
Please suggest me if I am follow the correct approach and what is the solution of this exception.
Below is the code with one zookeeper and one Kafka node. Similar way in the original docker compose, it contains 3 kafka, 3 zookeeper nodes
zookeeper:
hostname: ${HOST_IP}
image: ${ZOOKEEPER_IMAGE}
container_name: zookeeper
command: /bin/bash -c "/start.sh"
volumes:
- ${VOLUMES_FOLDER}/zk/data:/data
- ${VOLUMES_FOLDER}/zk/conf:/conf
network_mode: "host"
ports:
- 0.0.0.0:${ZOOKEEPER_EXPOSED_PORT}:${ZOOKEEPER_EXPOSED_PORT}
kafka1:
hostname: ${HOST_IP}
image: ${KAFKA_IMAGE}
depends_on:
- zookeeper
container_name: kafka1
command: /bin/bash -c "/start.sh"
volumes:
- ${VOLUMES_FOLDER}/kf/1/logs:/logs
- ${VOLUMES_FOLDER}/kf/1/data:/data
environment:
- KAFKA_ADVERTISED_HOST_NAME=${HOST_IP}
- KAFKA_DELETE_TOPIC_ENABLE=true
- KAFKA_BROKER_ID=10
- port = ${KAFKA1_EXPOSED_PORT}
- ZOOKEEPER_IP=${HOST_IP}
- JMX_PORT=7208
network_mode: "host"
ports:
- 0.0.0.0:${KAFKA1_EXPOSED_PORT}:${KAFKA1_EXPOSED_PORT}
under impression that zookeeper will be started automatically
It won't. Kafka KRaft uses Raft protocol. Zookeeper is not required, at all, therefore is not started.
Zookeeper is embedded in kafka
It is not. Zookeeper server scripts are available, but they are not used in any embedded way, or automatically.
repackage kafka 3.x into two separate docker images
bitnami/kafka images, for example, already offer KRaft mode. Unclear what images you're currently using.
Your error is likely because advertised.host.name and port were both removed as Kafka properties. You should be using listeners and advertised.listeners.
I am running automated UI tests using selenium grid 4 (doku)
I am trying to start my webdriver using a fake webcam (a .y4m File), in order to start the driver using the .y4m File, I need the file in my docker container.
My config.toml
[docker]
# Configs have a mapping between the Docker image to use and the capabilities that need to be matched to
# start a container with the given image.
configs = [
"selenium/standalone-firefox:4.1.4-20220427", "{\"browserName\": \"firefox\"}",
"selenium/standalone-chrome:4.1.4-20220427", "{\"browserName\": \"chrome\"}",
"selenium/standalone-edge:4.1.4-20220427", "{\"browserName\": \"MicrosoftEdge\"}"
]
# URL for connecting to the docker daemon
# Most simple approach, leave it as http://127.0.0.1:2375, and mount /var/run/docker.sock.
# 127.0.0.1 is used because interally the container uses socat when /var/run/docker.sock is mounted
# If var/run/docker.sock is not mounted:
# Windows: make sure Docker Desktop exposes the daemon via tcp, and use http://host.docker.internal:2375.
# macOS: install socat and run the following command, socat -4 TCP-LISTEN:2375,fork UNIX-CONNECT:/var/run/docker.sock,
# then use http://host.docker.internal:2375.
# Linux: varies from machine to machine, please mount /var/run/docker.sock. If this does not work, please create an issue.
url = "http://127.0.0.1:2375"
# Docker image used for video recording
video-image = "selenium/video:ffmpeg-4.3.1-20220427"
# Uncomment the following section if you are running the node on a separate VM
# Fill out the placeholders with appropriate values
#[server]
#host = <ip-from-node-machine>
#port = <port-from-node-machine>
And my docker-compose.yml
# To execute this docker-compose yml file use `docker-compose -f docker-compose-v3-dynamic-grid.yml up`
# Add the `-d` flag at the end for detached execution
# To stop the execution, hit Ctrl+C, and then `docker-compose -f docker-compose-v3-dynamic-grid.yml down`
version: "3"
services:
node-docker:
image: selenium/node-docker:4.1.4-20220427
volumes:
- ./assets:/opt/selenium/assets
- ./NodeDocker/config.toml:/opt/bin/config.toml
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- selenium-hub
environment:
- SE_EVENT_BUS_HOST=selenium-hub
- SE_EVENT_BUS_PUBLISH_PORT=4442
- SE_EVENT_BUS_SUBSCRIBE_PORT=4443
selenium-hub:
image: selenium/hub:4.1.4-20220427
container_name: selenium-hub
ports:
- "4442:4442"
- "4443:4443"
- "4444:4444"
those 2 Files are the exact same from the doku.
The grid works like this:
I am connecting a remoteDriver using C# to localhost:4444, the selenium-hub spawns a new selenium/standalone-chrome container using the config.toml.
My problem: I was not able to find any dokumentation on how to map a volume using the config.toml
I have trouble understanding how docker port mapping works. I have a docker-compose file with a couple of containers, one of them is a rabbitmq service.
The docker-compose file is:
version: "3.9"
volumes:
test:
external: true
services:
rabbitmq3:
container_name: "rabbitmq"
image: rabbitmq:3.8-management-alpine
environment:
- RABBITMQ_DEFAULT_USER=myuser
- RABBITMQ_DEFAULT_PASS=mypassword
ports:
# AMQP protocol port
- '5671:5672'
# HTTP management UI
- '15671:15672'
So the container runs using docker compose up, no problem. But when I access the rabbitmq management plugin using container_ip:15671 or container_ip:15671, I don't get anything. But when I access it using 127.0.0.1:15672, I can access the management plugin.
It probably is a stupid question but how can I access the container service using localhost?
The port sematic is as such <HOST_PORT>:<CONTAINER_PORT>. So -p 15671:15672 implies that the container port 15672 is mapped to the port 15671 on your machine.
Based on your docker compose file, the ports 5671 and 15671 are exposed on your machine.
The management portal can be accessed using http://localhost:15671 and the rabbitmq service can be used using the http://localhost:5671.
The IP 127.0.0.1 is localhost.
I am trying to set up a sample application with the Traefik reverse proxy in Docker.
I am using Traefik v2.2 for this project which has significant differences from Traefik.v1.0.
Here is my docker-compose.yml file:
version: '3'
services:
traefik:
# The official v2 Traefik docker image
image: traefik:v2.2
# Enables the web UI and tells Traefik to listen to docker
command:
- --api.insecure=true
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --entrypoints.web.address=:80
ports:
# The HTTP port
- "89:80"
# The Web UI (enabled by --api.insecure=true)
- "8089:8080"
volumes:
# So that Traefik can listen to the Docker events
- "/var/run/docker.sock:/var/run/docker.sock:ro"
whoami:
# A container that exposes an API to show its IP address
image: containous/whoami
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`whoami.localhost`)"
- "traefik.http.routers.whoami.entrypoints=web"
I can access Traefik's dashboard when I go to localhost:8089 on my web browser, but I cannot access the whoami application when I type in the whoami.localhost address on my web browser. I'm just wondering if there is anything I need to change before I can access it, or do I need to change the host from whoami.localhost to localhost:3000 since that's the port I want to access the application in.
One problem I am spotting is that you exposed container port 80 of the traefik container to the host port 89. If you type in whoami.localhost in your web browser, your browser is going to search for an application on host port 80 at that address (since localhost maps natively to port 80), but it is not going to find anything there, because it can only be found at port 89. From my understanding, you should be able to access the application via the command line with the command curl -H Host:whoami.localhost http://127.0.0.1:89. Unfortunately, I am unsure how the URL whoami.localhost:89 is handled by your browser respectively by your DNS.
You can to modify the docker-compose.yml file this way:
version: "3"
services:
traefik:
# The official v2 Traefik docker image
image: traefik:v2.2
# Enables the web UI and tells Traefik to listen to docker
command:
- --api.insecure=true
- --providers.docker=true
ports:
# The HTTP port
- "89:80"
# The Web UI (enabled by --api.insecure=true)
- "8089:8080"
volumes:
# So that Traefik can listen to the Docker events
- /var/run/docker.sock:/var/run/docker.sock
whoami:
# A container that exposes an API to show its IP address
image: containous/whoami
labels:
- traefik.http.routers.whoami.rule=Host(`whoami.localhost`)
And then you can access the application on your command terminal by typing in:
curl -H Host:whoami.localhost http://127.0.0.1:89
Note: whoami.localhost can be whoami.docker.localhost or app.localhost or whatever you want. The thing here is that you should localhost attached to the end, except if you're adding a Fully Qualifies Domain name (FQDN).
That's all.
I hope this helps
I try to run Traefik on docker on Windows native container but I don't find any exemple. I just want to run the Getting Started exemple with whoami.
I try many parameters without success. I have two question :
how to to pass a configuration file for traefik with an Windows Container ? (binding file don't work on Windows)
how to connect to docker host with named pipe ?
Exemple of docker compose I've tried :
version: '3.7'
services:
reverse-proxy:
image: traefik:v1.7.2-nanoserver # The official Traefik docker image
command: --api --docker --docker.endpoint=npipe:////./pipe/docker_engine # Enables the web UI and tells Træfik to listen to docker
ports:
- "80:80" # The HTTP port
- "8080:8080" # The Web UI (enabled by --api)
volumes:
- source: '\\.\pipe\docker_engine'
target: '\\.\pipe\docker_engine'
type: npipe
whoami:
image: emilevauge/whoami # A container that exposes an API to show its IP address
labels:
- "traefik.frontend.rule=Host:whoami.docker.localhost"
Traefik dashboard work fine on 8080 but no provider found and whoami container not found.
I'm with Windows 10 1803, Docker version 18.06.1-ce, build e68fc7a, docker-compose version 1.22.0, build f46880fe
Note that Traefik work fine if I launch it on my Windows (not in a container).
Thank you for help.
There is a working example of it here. If you would like to run it in swarm, try using docker network create and docker service create instead of docker stack deploy. See my question here for more details.