I am trying to mirror a kafka topic from a provider in a container in an ec2 instance to a consumer, and messages are not coming through. I suspect that I am messing things up with the .properties configs, as this is my first time using MirrorMaker. I may have also messed up with pointing to the wrong ports somewhere along the way.
The would-be provider broker is running in a centOS container in an ec2 instance. The provider is receiving data from a remote MySQL server through a custom-configured jdbc source connector to a topic called mysql-jdbc-events. The provider is successfully receiving messages.
The would-be consumer is currently on the host ec2 instance, although that will change once it's successfully been tested. I mapped port 12181 of the host to port 2181 of the container (where zookeeper is running). I am running the MirrorMaker command from the consumer.
I ran the command
./kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config /path/to/config/consumer.properties --producer.config /path/to/config/producer.properties --whitelist "mysql-jdbc-events"
consumer.properties:
# format: host1:port1,host2:port2 ...
zookeeper.connect=(host ip-address):12181
zookeeper.connection.timeout.ms=10000
bootstrap.servers=localhost:9092
# consumer group id
group.id=mirror_group
producer.properties:
# format: host1:port1,host2:port2 ...
zookeeper.connect=(host ip-address):2181
bootstrap.servers=localhost:9092
# specify the compression codec for all data generated: none, gzip, snappy, lz4, zstd
compression.type=none
I tried both with and without the zookeeper.connect parameter in the producer config because I found conflicting about it being necessary. I also got a warning to the effect of WARN The configuration 'zookeeper.connect' was supplied but isn't a known config., but I read elsewhere on SO that this could be ignored.
I did not get any messages populated to the topic at the consumer, but there are messages in the topic at the producer.
If any more information would be helpful, please let me know.
I am also not married to this configuration - if there is a simpler way to keep the jdbc in the container and forward the messages to a kafka instance outside the container, that's good too.
Related
I'm trying to run Kafka-Connect locally with docker-compose. Much as I like Confluent products and Kafka, it's sometimes a huge quest to pass some config or find one consistent example.
In my docker-compose file, I'm using 6.0.0 versions for broker, zk, sr and kafka-connect right now, but I've tried older versions as well.
The broker(confluentinc/cp-server:6.0.0) fails with:
INFO [Admin Manager on Broker 1]: Error processing create topic
request CreatableTopic(name='_confluent-license', numPartitions=1,
replicationFactor=3, assignments=[],
configs=[CreateableTopicConfig(name='cleanup.policy',
value='compact'), CreateableTopicConfig(name='min.insync.replicas',
value='2')], linkName=null, mirrorTopic=null)
(kafka.server.AdminManager)
And I simply don't know how to pass confluent.topic.replication.factor as env vars to my workers. I've added both:
CONNECT_CONFLUENT_TOPIC_REPLICATION_FACTOR: "1"
CONFLUENT_TOPIC_REPLICATION_FACTOR: "1"
...but they are both ignored.
What's more, I can't even find _confluent-license mentioned in docs anywhere, only _confluent-command.
How can I possibly make connect work locally inside docker-compose and without setting up 3 brokers?
The broker is failing, because that topic is created from it, so you want the variable on the broker
KAFKA_CONFLUENT_TOPIC_REPLICATION_FACTOR: 1
Otherwise, you want cp-kafka, not cp-server
As OneCricketeer pointed out, the value must be set in the broker container, and that also explains why it's broker container that fails, not connect workers. That definitely makes sense. Broker can't fail because of a misconfigured clients.
What probably made it a lot more confusing to find a solution is that the env var and configuration property for license topic replication factor do not follow a regular pattern where env var is an uppercases configuration property with dots replaced with underscores(and prefix where applicable). In reality we get:
confluent.topic.replication.factor -> KAFKA_CONFLUENT_TOPIC_REPLICATION_FACTOR
I have a server and I am using Ubuntu 20.04, nginx , mosquitto and node-red and docker , let's call the website http://mywebsite.com. The problem that I am facing that I have created a client lets call it client1 in docker so the URL will be http://mywebsite.com/client1
and I want to establish an MQTT connection via mosquitto and I'm sending the data on topic test
The problem that on node red node of MQTT when I write the IP address of my mosquitto container it works
But if I change the IP address 192.144.0.5 with mywebsite.com/client1 I can't connect to mosquitto and I can't send or receive any form of data
any idea on how to solve this problem
OK, you are going to have several problems here.
You can not do path based proxying with MQTT. If you want to have multiple MQTT brokers (1 per client) bound to a single public facing domain/IP address then they are all going to have to run on separate ports (other than the default 1883).
Nginx can do MQTT protocol proxying (e.g. like this), so you can use this to expose the different ports and forward them to the separate instances of mosquitto, but even if you had a different hostname (all pointing at the same IP address) nginx has no way to know which host name was used because there is no equivalent to the HOST HTTP header to direct it. If you were to use MQTT with TLS then you may be able to get it to work with SNI, but I've never seen anybody do that yet (possible docs for SNI based routing here) It works, explanation about how to do it here.
If you use MQTT over Websockets then you should be able to use hostname based routing.
Path based proxying for Node-RED currently doesn't work properly if you enable admin authentication, because the admin auth tokens are currently stored in browser local storage and only scoped to the hostname, not the hostname + path. This will mean that a client will only ever be able to log into one instance at a time.
You can work round this by using host based proxying, e.g. http://client1.mywebsite.com
A fix for this is on the backlog for Node-RED, probably (no promises) to be looked at after version 1.2.0 ships
I think to understood how fabric mainly works and how consens is reached. What I am still missing in the documentation is the part of what happens inside of a docker container of fabric to take part in communication process.
So, communication starting from a client (e.g. an app) takes place in using gRPC messages between peers and orderer.
But what happens inside of the containers?
I imagine it for myself as a process that is only receiving gRPC message and answering them in using functions in the background of a peer/orderer, to hands out its response for further processing in another unit like the client to collect the responses of multiple peers for a smart contract.
But what happens really inside a container? I mean, a container spawns, when the docker image file is loaded and launched by the yaml config file. But what is started there inside of it (is there only a single peer binary started, e.g. like the command "peer node start") - I mean the compiled go binary file "peer" only?? What is listening? What is responding there? I discovered only one port for every container that is exposed out. This seems for me to be the gate for gRPC (cause it is often used as Port ID: **51).
The same questions goes for the orderer, the chaincode and the cli. How are they talking to each other or is gRPC the only way of communication and processing (excluded of the discovery service and gossip, how is this started inside of the containers (in using the yaml files for lauchun only or is there further internal configuration or a startupscript in the image files (cause I cannot look inside the images, only login on running containers while runtime).
When your client sends request to one of the peers, peer instance checks if requested chaincode (CC) installed on it. If CC not installed: Obviously you'll get an error.
If CC is installed: Peer checks if a dedicated container is already started for the given CC and corresponding version. If container is started, peer sends transaction request to that CC instance and returns back the response to your client after signing the transaction. Signing guarantees that response is really sent by that peer.
If container not started:
It builds a docker image and starts that instance (docker container). New image would be based on one of the hyperledger images. i.e. if your CC is GO, then hyperledger/baseos, which is very basic linux os, will be used. This new image contains CC binary and META-DATA as well.
That peer instance is using underlying (your) machine's docker server to do all of those. That's the reason why we need to pass /var/run:/host/var/run into volume mapping and CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock into environment variables.
Once the CC container starts, it connects to its parent peer node which is defined with
CORE_PEER_CHAINCODEADDRESS attribute. Peer dictates to child (probably during image creation) to use this address, so they obey. Peer node defines its own listen URL with CORE_PEER_CHAINCODELISTENADDRESS attribute.
About your last question; communication is with gRPC in between nodes also with clients. If TLS is enabled, then it's for sure secure communication. Entry point for orderers to know about peers and peers know about other organizations' peers is the definition of anchor peers defined during channel creation. Discovery service is running in peer nodes, so they can hold a close to real-time network layout. Discovery service also provides peers' identity, that's how clients can detect other organizations' peers when endorsement policy requires multiple organizations' endorsement policy (i.e. if policy look like AND(Org1MSP.member, Org2MSP.member)).
We are using a python based solution which shall load and store files from S3. For developing and local testing we are using a vagrant environment with docker and docker-compose. We have two docker-compose defintions - one for the assisting backend services (mongo, restheart, redis and s3) and the other one containing the python based REST API exposing solution using the backend services.
When our "front-end" docker-compose group interacts with restheart this works fine (using the name of the restheart container as server host in http calls). When we are doing the same with scality/s3 server this does not work.
The interesting part is, that we have created a test suite for using the scality/s3 server from a python test suite running on the host (windows10) over the forwarded ports through vagrant to the docker container of scality/s3 server within the docker-compose group. We used the endpoint_url localhost and it works perfect.
In the error case (when frontend web service wants to write to S3) the "frontend" service always responds with:
botocore.exceptions.ClientError: An error occurred (InvalidURI) when calling the CreateBucket operation: Could not parse the specified URI. Check your restEndpoints configuration.
And the s3server always responds with http 400 and the message:
s3server | {"name":"S3","clientIP":"::ffff:172.20.0.7","clientPort":49404,"httpMethod":"PUT","httpURL":"/raw-data","time":1521306054614,"req_id":"e385aae3c04d99fc824d","level":"info","message":"received request","hostname":"cdc8a2f93d2f","pid":83}
s3server | {"name":"S3","bytesSent":233,"clientIP":"::ffff:172.20.0.7","clientPort":49404,"httpMethod":"PUT","httpURL":"/raw-data","httpCode":400,"time":1521306054639,"req_id":"e385aae3c04d99fc824d","elapsed_ms":25.907569,"level":"info","message":"responded with error XML","hostname":"cdc8a2f93d2f","pid":83}
We are calling the scality with this boto3 code:
s3 = boto3.resource('s3',
aws_access_key_id='accessKey1',
aws_secret_access_key='verySecretKey1',
endpoint_url='http://s3server:8000')
s3_client = boto3.client('s3',
aws_access_key_id='accessKey1',
aws_secret_access_key='verySecretKey1',
endpoint_url='http://s3server:8000')
s3.create_bucket(Bucket='raw-data') # here the exception comes
bucket = self.s3.Bucket('raw-data')
This issue is quite common. In your config.json file, which you mount in your Docker container, I assume, there is a restEndpoints section, where you must associate a domain name with a default region. What that means is your frontend domain name should be specified in there, matching a default region.
Do note that that default region does not prevent you from using other regions: it's just where your buckets will be created if you don't specify otherwise.
In the future, I'd recommend you open an issue directly on the Zenko Forum, as this is where most of the community and core developpers are.
Cheers,
Laure
I m trying setup a cassandra cluster as a test bed but gave the JMX remote connection error. I seem to found the answer for my error from cassandra FAQ page
Nodetool says "Connection refused to host: 127.0.1.1" for any remote host. What gives?
Nodetool relies on JMX, which in turn relies on RMI, which in turn sets up it's own listeners and connectors as needed on each end of the exchange. Normally all of this happens behind the scenes transparently, but incorrect name resolution for either the host connecting, or the one being connected to, can result in crossed wires and confusing exceptions.
If you are not using DNS, then make sure that your /etc/hosts files are accurate on both ends. If that fails try passing the -Djava.rmi.server.hostname=$IP option to the JVM at startup (where $IP is the address of the interface you can reach from the remote machine).
But can somebody help me on how to do -Djava.rmi.server.hostname=$IP
Or what to add is hosts file, i know that in hosts normally we add "IP Alias", but whose ip and alias.
I dont know much java or either linux
I m currently working on ubuntu v10.04 and cassandra v0.74
Sudesh
For JMX you need to enable JMX-remoting:
java -Dcom.sun.management.jmxremote
Depending on from where you want to access the jmx-server, you also need to specify a port:
-Dcom.sun.management.jmxremote.port=12345
and set or disable passwords.
Have a look at http://download.oracle.com/javase/1.5.0/docs/guide/management/agent.html for more details.