running Kafka Connect 4.1.1 on DC/OS using the confluent community package. How can we upload or add our jdbc driver to the remote cluster?
Update: It's a package installed DC/OS catalog, which is a mesos framework, running docker images.
Update
Script borrowed from here (thanks to #rmoff)
It's an example of overriding the Docker CMD with a bash script to download and extract the REST API source connector.
bash -c 'echo Installing unzip… && \
curl -so unzip.deb http://ftp.br.debian.org/debian/pool/main/u/unzip/unzip_6.0-16+deb8u3_amd64.deb && \
dpkg -i unzip.deb && \
echo Downloading connector… && \
curl -so kafka-connect-rest.zip https://storage.googleapis.com/rmoff-connectors/kafka-connect-rest.zip && \
mkdir -p /u01/connectors/ && \
unzip -j kafka-connect-rest.zip -d /u01/connectors/kafka-connect-rest && \
echo Launching Connect… && \
/etc/confluent/docker/run'
You'll need to build your own Docker images and publish them to a resolvable Docker Registry for your Mesos cluster, and then edit the Mesos Service to pull these images instead of the Confluent one.
For example, in your Dockerfiles, you would have
ADD http://somepath.com/someJDBC-driver.jar /usr/share/java/kafka-connect-jdbc
Or curl rather than ADD, as shown in the Confluent docs (because it needs to extract that .tar.gz file).
FROM confluentinc/cp-kafka-connect
ENV MYSQL_DRIVER_VERSION 5.1.39
RUN curl -k -SL "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-${MYSQL_DRIVER_VERSION}.tar.gz" \
| tar -xzf - -C /usr/share/java/kafka-connect-jdbc/ --strip-components=1 mysql-connector-java-5.1.39/mysql-connector-java-${MYSQL_DRIVER_VERSION}-bin.jar
You can also use confluent-hub install to add other connectors that aren't JDBC JAR files
Related
I'm trying to run RStudio Server in a docker container. Users will connect to this docker container and use RStudio via the internet.
The built-in mechanism for uploading and downloading files in Rstudio is very slow so I'd also like to run an SFTP server in a separate container.
I'm trying to link the two containers using Docker Volumes but I'm having some trouble. Here's is how I'm trying to run the two images.
I'm running the FTP sever using:
docker run -p 2222:22 -v /home/rstudio --name ftpserver -d atmoz/sftp rstudio:foo:1001
Then I'm trying to connect to the same directory in RStudio by doing:
docker run -d -p 8787:8787 -e PASSWORD=foo --volumes-from ftpserver --name rstudio r-studio-bio:Dockerfile
This causes RStudio to give an error
RStudio Initialization Error. Unable to connect to service.
Likewise I'm unable to upload to the FTP server because it's saying I lack the proper permissions.
The FTP server image is here : https://hub.docker.com/r/atmoz/sftp/
The RStudio-Server Dockerfile is:
# See the following for more info:
# https://hub.docker.com/r/pgensler/sandboxr/
# https://www.rocker-project.org/images/
# https://hub.docker.com/r/rocker/rstudio
FROM rocker/tidyverse
LABEL maintainer="Alex"
#
RUN mkdir -p $HOME/.R
RUN mkdir $HOME/Rlibs
ENV R_LIBS $HOME/Rlibs
# COPY R/Makevars /root/.R/Makevars
RUN apt-get update -qq \
&& apt-get -y --no-install-recommends install \
curl \
clang \
ccache \
default-jdk \
default-jre \
wget \
systemd \
# openssh-server \
&& R CMD javareconf \
# && systemctl ssh start \
# && systemctl enable ssh \
&& rm -rf /var/lib/apt/lists/*
RUN wget \
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh
# # Install additional R packages
RUN Rscript -e "BiocManager::install()"
RUN Rscript -e "BiocManager::install('multtest')"
RUN Rscript -e "install.packages('Seurat')"
This question is somewhat related to one of my previous questions as in it gives a clearer idea on what I am trying to achieve.. This question is about an issue I ran into when trying to achieve the task in that previous question...
I am trying to test if my kubectl works from within the Jenkins container. When I start up my Jenkins container I use the following command:
docker run \
-v /home/student/Desktop/jenkins_home:/var/jenkins_home \
-v $(which kubectl):/usr/local/bin/kubectl \ #bind docker host binary to docker container binary
-v ~/.kube:/home/jenkins/.kube \ #docker host kube config file stored in /.kube directory. Binding this to $HOME/.kube in the docker container
-v /var/run/docker.sock:/var/run/docker.sock \
-v $(which docker):/usr/bin/docker -v ~/.kube:/home/root/.kube \
--group-add 998
-p 8080:8080 -p 50000:50000
-d --name jenkins jenkins/jenkins:lts
The container starts up and I can login/create jobs/run pipeline scripts all no issue.
I created a pipeline script just to check if I can access my cluster like this:
pipeline {
agent any
stages {
stage('Kubernetes test') {
steps {
sh "kubectl cluster-info"
}
}
}
}
When running this job, it fails with the following error:
+ kubectl cluster-info // this is the step
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
error: the server doesn't have a resource type "services"
Thanks!
I'm not getting why there is:
-v $(which kubectl):/usr/local/bin/kubectl -v ~/.kube:/home/jenkins/.kube
/usr/local/bin/kubectl is a kubectl binary and ~/.kube:/home/jenkins/.kube should be the location where the kubectl binary looks for the cluster context file i.e. kubeconfig. First, you should make sure that the kubeconfig is mounted to the container at /home/jenkins/.kube and is accessible to kubectl binary. After appropriate volume mounts, you can verify by creating a session in the jenkins container with docker container exec -it jenkins /bin/bash and test with kubectl get svc. Make sure you have KUBECONFIG env var set in the session with:
export KUBECONFIG=/home/jenkins/.kube/kubeconfig
Before you run the verification test and
withEnv(["KUBECONFIG=$HOME/.kube/kubeconfig"]) {
// Your stuff here
}
In your pipeline code. If it works with the session, it should work in the pipeline as well.
I would personally recommend to create a custom Docker image for Jenkins which will contain kubectl binary and other utilities necessary (such as aws-iam-authenticator for AWS EKS IAM-based authentication) for working with Kubernetes cluster. This creates isolation between your host system binaries and your Jenkins binaries.
Below is the Dockerfile I'm using which contains, helm, kubectl and aws-iam-authenticator.
# This Dockerfile contains Helm, Docker client-only, aws-iam-authenticator, kubectl with Jenkins LTS.
FROM jenkins/jenkins:lts
USER root
ENV VERSION v2.9.1
ENV FILENAME helm-${VERSION}-linux-amd64.tar.gz
ENV HELM_URL https://storage.googleapis.com/kubernetes-helm/${FILENAME}
ENV KUBE_LATEST_VERSION="v1.11.0"
# Install the latest Docker CE binaries
RUN apt-get update && \
apt-get -y install apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common && \
curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg > /tmp/dkey; apt-key add /tmp/dkey && \
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
$(lsb_release -cs) \
stable" && \
apt-get update && \
apt-get -y install docker-ce \
&& curl -o /tmp/$FILENAME ${HELM_URL} \
&& tar -zxvf /tmp/${FILENAME} -C /tmp \
&& mv /tmp/linux-amd64/helm /bin/helm \
&& rm -rf /tmp/linux-amd64/helm \
&& curl -L https://storage.googleapis.com/kubernetes-release/release/${KUBE_LATEST_VERSION}/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl \
&& chmod +x /usr/local/bin/kubectl \
&& curl -L https://amazon-eks.s3-us-west-2.amazonaws.com/1.11.5/2018-12-06/bin/linux/amd64/aws-iam-authenticator -o /usr/local/bin/aws-iam-authenticator \
&& chmod +x /usr/local/bin/aws-iam-authenticator
Kubernetes fails inside jenkins pipeline this was my solution for jenkins installed locally on a windows machine.
I am running windows 10, using docker for windows.
Here's the baseline:
docker pull nshou/elasticsearch-kibana:kibana3
docker image list
docker run -d -p 9200:9200 -p 5601:5601 {imageName}:kibana3
curl localhost:9200/_stats
Good response.
So I copied the Dockerfile from https://bitbucket.org/nshou/elasticsearch-kibana/src/kibana3/Dockerfile
FROM ubuntu:latest
RUN apt-get update -q
RUN apt-get install -yq wget default-jre-headless mini-httpd
ENV ES_VERSION 1.7.4
RUN cd /tmp && \
wget -nv https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz && \
tar zxf elasticsearch-${ES_VERSION}.tar.gz && \
rm -f elasticsearch-${ES_VERSION}.tar.gz && \
mv /tmp/elasticsearch-${ES_VERSION} /elasticsearch
ENV KIBANA_VERSION 3.1.3
RUN cd /tmp && \
wget -nv https://download.elastic.co/kibana/kibana/kibana-${KIBANA_VERSION}.tar.gz && \
tar zxf kibana-${KIBANA_VERSION}.tar.gz && \
rm -f kibana-${KIBANA_VERSION}.tar.gz && \
mv /tmp/kibana-${KIBANA_VERSION} /kibana
CMD /elasticsearch/bin/elasticsearch -Des.http.cors.enabled=true -Des.logger.level=OFF & mini_httpd -d /kibana -h `hostname` -r -D -p 5601
EXPOSE 9200 5601
and I build it with
docker build -t test/test .
Image builds successfully.
docker image list
docker run -d -p 9200:9200 -p 5601:5601 {imageName}:latest
curl localhost:9200/_stats
No response. Not a 404, but the server responds with a no response.
The problem seems to be that when I build the image myself it doesn't work. When I pull the same dockerfile image from the hub, it works.
Why and how do I fix it?
Figured it out.
When the locally built container is running, its actually crashing with this error
Unrecognized VM option 'UseParNewGC' , Error: Could not create the Java Virtual Machine
The default-jre-headless is using a version of Java that is incompatible with this older version of Elasticsearch.
Switching to openjdk-8-jre-headless solves the issue.
I guess the image on nshou is cached and so old that it's using an older version of the jre? I'm not sure why the baseline image would work when the latest default-jre-headless has this issue with the kibana3 tag of the repo.
Thankfully my problem is resolved.
I deployed my first scala project on docker but i have a problem, the problem is the docker says that the server has been started, but surprisingly it doesn't listen to any request, even i exposed the port to the host, when i tried to request a get, it says that the connection is refused, also i tried to telnet to the port and it seems that there are no listener on port 9000 neither 3200 an 3000, please find bellow what i have wrote in dockerFile
FROM jelastic/sbt
# Env variables
ENV SCALA_VERSION 2.12.4
ENV SBT_VERSION 1.1.0
# Scala expects this file
RUN touch /usr/lib/jvm/java-8-openjdk-amd64/release
# Install Scala
## Piping curl directly in tar
RUN \
curl -fsL https://downloads.typesafe.com/scala/$SCALA_VERSION/scala-$SCALA_VERSION.tgz | tar xfz - -C /root/ && \
echo >> /root/.bashrc && \
echo "export PATH=~/scala-$SCALA_VERSION/bin:$PATH" >> /root/.bashrc
# Install sbt
RUN \
curl -L -o sbt-$SBT_VERSION.deb https://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb && \
dpkg -i sbt-$SBT_VERSION.deb && \
rm sbt-$SBT_VERSION.deb && \
apt-get update && \
apt-get install sbt && \
sbt sbtVersion
WORKDIR /
ADD play /
RUN tree /
EXPOSE 9000
CMD sbt run
and my run command was
docker run -p 9000:9000 -t bee while bee is my image name
as you see the server is started properly.
please find bellow the attached picture to be more clearly
here is the docker ps
If you see your screenshot, it clear states the docker machine is located at 192.168.99.100. So that is the address you need to use.
Open http://192.168.99.100:9000 and it should work
Basic environment:
[root#0b3608de9c84 tmp]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
I want to install Kafka in Dockerfile,below is my Dockerfile instruction:
#choose scala and kafka version
ENV KAFKA_VERSION="0.8.2.1" SCALA_VERSION="2.11"
#install kafka
RUN cd /tmp && wget https://www.apache.org/dyn/closer.cgi?path=/kafka/$KAFKA_VERSION/kafka_$SCALA_VERSION-$KAFKA_VERSION.tgz && tar xf kafka_${SCALA_VERSIO N}-${KAFKA_VERSION}.tgz -C /opt && rm kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz
I know https://www.apache.org/dyn/closer.cgi?path=/kafka/$KAFKA_VERSION/kafka_$SCALA_VERSION-$KAFKA_VERSION.tgz(or https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.2.1/kafka_2.11-0.8.2.1.tgz) is not true resource address.Can you help me use wget and the URL on Kafka web page to install Kafka correctly?
As you’ve pointed out, https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.2.1/kafka_2.11-0.8.2.1.tgz is not the real address. Instead, it links to a mirror site that is close to your location, e.g., I live in Ireland so ftp.heanet.ie is the closest mirror to my location.
You should (manually) use the URL of the relevant mirror site in your wget command, e.g.
wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/kafka/0.8.2.1/kafka_2.11-0.8.2.1.tgz
The other issue is the spaces within the shell variable of your tar command:
tar xf kafka_${SCALA_VERSIO N}-${KAFKA_VERSION}.tgz
Combining the above fixes should give you something similar to:
RUN cd /tmp && wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/kafka/$KAFKA_VERSION/kafka_$SCALA_VERSION-$KAFKA_VERSION.tgz && tar xf kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt && rm kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz