I am new to GitLab and Kafka and do not have more ideas on how to run the Kafka command using shell script through the GitLab CI/CD pipeline.
Following configuration I am using to achieve this but getting exceptions, is there anything I am missing or configured incorrectly? Please suggest
In .gitlab-ci.yml
run_kafka:
image: confluentinc/cp-kafka
script :
- . demo.sh
In demo.sh
kafka-consumer-groups --command-config ./config.properties --bootstrap-server "pkc-6ojv2.us-west4.gcp.confluent.cloud:9092" --group my-created-consumer-group --topic SampleTopic --reset-offsets --to-current
In config.properties
bootstrap.servers=pkc-confluent.cloud:9092
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";
exception getting when pipeline is running
Error: Executing consumer group command failed due to org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeGroups(api=FIND_COORDINATOR)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeGroups(api=FIND_COORDINATOR)
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$resetOffsets$1(ConsumerGroupCommand.scala:434)
at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:675)
at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:669)
at scala.collection.AbstractIterable.foldLeft(Iterable.scala:933)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:432)
at kafka.admin.ConsumerGroupCommand$.run(ConsumerGroupCommand.scala:76)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:59)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeGroups(api=FIND_COORDINATOR)
Related
NB: i've already installed logstash with docker and it successfully started at endpoint {:port=>9600, :ssl_enabled=>false}.
this is what i get as an error:
The stdin plugin is now waiting for input:
[2022-08-22T08:39:59,117][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-08-22T08:40:03,491][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://admin:xxxxxx#opensearch:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [https://admin:xxxxxx#opensearch:9200/][Manticore::ResolutionFailure] opensearch"}
I am trying to run/configure a Spring Data Cloud Data Flow (SCDF) to schedule a task for a Spring Batch Job.
I am running in a minikube that connects to a local postgresql(localhost:5432). The minikube runs in a virtualbox where I assigned a vnet thru the --cidr, so minikube can connect to the local postgres.
Here is the postgresql service yaml:
https://github.com/msuzuki23/SpringCloudDataFlowDemo/blob/main/postgres-service.yaml
Here is the SCDF config yaml:
https://github.com/msuzuki23/SpringCloudDataFlowDemo/blob/main/server-config.yaml
Here is the SCDF deployment yaml:
https://github.com/msuzuki23/SpringCloudDataFlowDemo/blob/main/server-deployment.yaml
Here is the SCDF server-svc.yaml:
https://github.com/msuzuki23/SpringCloudDataFlowDemo/blob/main/server-svc.yaml
To launch the SCDF server in minikube I do the following kubectl commands:
kubectl apply -f secret.yaml
kubectl apply -f configmap.yaml
kubectl apply -f postgres-service.yaml
kubectl create -f server-roles.yaml
kubectl create -f server-rolebinding.yaml
kubectl create -f service-account.yaml
kubectl apply -f server-config.yaml
kubectl apply -f server-svc.yaml
kubectl apply -f server-deployment.yaml
I am not running Prometeus, Grafana, Kafka/Rabbitmq as I want to test and make sure I can launch the Spring Batch Job from SCDF. I did not run the skipper deployment (Spring Cloud DataFlow server runnning locally pointing to Skipper in Kubernetes) it is not necessary if just running tasks.
This is the error I am getting when trying to add an application from a docker private repo:
And this is the full error stack from the pod:
https://github.com/msuzuki23/SpringCloudDataFlowDemo/blob/main/SCDF_Log_Error
Highlights from the error stack:
2021-07-08 13:04:13.753 WARN 1 --- [-nio-80-exec-10] o.s.c.d.s.controller.AboutController : Skipper Server is not accessible
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:7577/api/about": Connect to localhost:7577 [localhost/127.0.0.1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to localhost:7577 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
Postges Hibernate Error:
2021-07-08 13:05:22.142 WARN 1 --- [p-nio-80-exec-5] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 42P01
2021-07-08 13:05:22.200 ERROR 1 --- [p-nio-80-exec-5] o.h.engine.jdbc.spi.SqlExceptionHelper : ERROR: relation "hibernate_sequence" does not exist
Position: 17
2021-07-08 13:05:22.214 ERROR 1 --- [p-nio-80-exec-5] o.s.c.d.s.c.RestControllerAdvice : Caught exception while handling a request
org.springframework.dao.InvalidDataAccessResourceUsageException: could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could
The first couple of errors are from the SCDF trying to connect to the skipper, since it was not configured, that was expected.
The second error is the Postgres JDBC Hibernate. How do I solve that?
Is there a configuration I am missing when setting the SCDF to point into the local postgres?
Also, in my docker jar I have not added any annotation such as #EnableTask.
Any help is appreciated, thx! Markus.
I did a search on
Caused by: org.postgresql.util.PSQLException: ERROR: relation
"hibernate_sequence" does not exist Position: 17
And found this stackoverflow anwer:
Postgres error in batch insert : relation "hibernate_sequence" does not exist position 17
Went to the postgres and created the hibernate_sequence:
CREATE SEQUENCE my_seq_gen START 1;
Then, add application worked.
I want to connect a slave to Master-Jenkins, but when trying to connect i'm getting following Error:
[05/02/18 15:26:59] [SSH] Opening SSH connection to <IP>
Key exchange was not finished, connection is closed.
java.io.IOException: There was a problem while connecting to <IP>:22
at com.trilead.ssh2.Connection.connect(Connection.java:818)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1324)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:831)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:820)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:93)
at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:230)
at com.trilead.ssh2.Connection.connect(Connection.java:770)
... 7 more
Caused by: java.io.IOException: Cannot negotiate, proposals do not match.
at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:405)
at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
... 1 more
[05/02/18 15:26:59] Launch failed - cleaning up connection
[05/02/18 15:26:59] [SSH] Connection closed.
Configuration for Node:
- Start-Method: Start Slave over SSH
- Hostname: is the IP
- Access Data: user i created for SSH Access - > public key is in authorized keys on Slave Node
If i am on my Master as user "jenkins" and do a ssh jenkins#<IP> i can login wihtout problems (public key is on slave).
Why it doesn't work for "UI-Jenkins".
Jenkins-Version: 1.658
Node: Ubuntu 14.04
SSH-Slave Plugin: 1.26
That "solved" the issue:
"Workaround is by commenting out MACs and KexAlgorithm line in /etc/ssh/sshd_config of Jenkins Slave and restarting the sshd (service ssh restart on Ubuntu)
UPDATE: the issue has been resolved as of 2017-04-29 "
Jenkins master fails to connect to the slave over SSH
Thought I'd throw my experience in this thread: my environment had a Windows master and mixed Windows and Linux agents. One Windows agent refused to connect to master, even after Master pushed 'jenkins-agent' and the other supporting files to the agent.
This agent had 6 different versions of the JDK and JRE installed. I uninstalled all of them, reinstalled only the latest JDK we needed, and set JAVA_HOME. This fixed the connectivity issue.
Execute this command on destination node.
sudo -i su -c 'sed -i -e "s/MACs /MACs hmac-sha1,/g" /etc/ssh/sshd_config; service sshd restart'
Just recently run into this issue with docker
Find the Java Path
/home/jenkins # which java
/opt/java/openjdk/bin/java
Export the Java Path. In this case I am using the docker-compose
...
exp_agent:
image: jenkins/ssh-agent:alpine
restart: always
environment:
JENKINS_AGENT_SSH_PUBKEY: $ENV_JENKINS_AGENT_SSH_PUBKEY
JAVA_HOME: $ENV_JAVA_HOME
container_name: jenkins-ssh-agent
ports:
- 22:22
networks:
- jenkins
...
The master still complains about the path of Java as /opt/java/openjdk/bin/java is not among the expected paths
...
[12/04/21 11:44:07] [SSH] Checking java version of /usr/bin/java
...
Create a symbolic link between the java path and one of the expected paths in the docker container (This could be automated in a Dockerfile)
ln -s /opt/java/openjdk/bin/java /usr/bin/java
I have SonarQube and Jenkins dockerized(They're running on different containers); sonar is running on localhost:9000 and Jenkins is running on localhost:8080. I configured the Sonar credentials(with acces Token) on Jenkins, but i when run SonarScanner I'm getting the following error by SonarScaner on Jenkins:
ERROR: SonarQube server [http://localhost:9000] can not be reached
INFO: ----------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ----------------------------------------------------------------
INFO: Total time: 0.358s
INFO: Final Memory: 4M/119M
INFO: ----------------------------------------------------------------
ERROR: Error during SonarQube Scanner execution
ERROR: Unable to execute SonarQube
ERROR: Caused by: Fail to get bootstrap index from server
ERROR: Caused by: Failed to connect to localhost/127.0.0.1:9000
ERROR: Caused by: Connection refused (Connection refused)
Docker:
How could i solve this, or make that both containers communicate between each others?
You can't use 127.0.0.1 as the IP address because that's the IP of the local container. You need to use the IP of the host.
I wrote an article on how to setup a Docker container of Jenkins and SonarQube to talk to each other that explains the setup.
I'm trying to run official SonarQube Docker container locally. I'm using the command provided here:
https://hub.docker.com/_/sonarqube/
It exits about 1 minute after it was started. Logs are reporting Elasticsearch connectivity issue
2017.09.05 08:16:40 INFO web[][o.e.client.transport] [Edwin Jarvis] failed to connect to node [{#transport#-1}{127.0.0.1}{127.0.0.1:9001}], removed from nodes list
org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9001] connect_timeout[30s]
.....
Caused by: java.net.ConnectException: Connection refused: /127.0.0.1:9001
.....
... 3 common frames omitted
2017.09.05 08:17:10 INFO app[][o.s.a.SchedulerImpl] Process [web] is stopped
2017.09.05 08:17:10 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
Turns out SonarQube container didn't have enough resources. I shut down other docker containers and it works for me now.