Flink Docker Image - docker

I am new in Docker, I am trying to create Flink docker image. Docker image is showing created successfully. But I have a doubt, I can not see Flink binaries in /opt path as mentioned in Dockerfile.
How do I know my Flink Docker image is created successfully.
Screenshot from console, few set of commands are highlighted in red color, means is it OK ?
Status showing Flink Docker image is created:
Please help me. Thank you....
Full Log:
sudo /home/develk/cntx_eng/build.sh \
> --job-artifacts /home/develk/cntx_eng/FlinkContextEnginePoc-0.0.1-SNAPSHOT.jar \
> --from-archive /home/develk/cntx_eng/flink-1.4.0-bin-hadoop24-scala_2.11.tgz \
> --image-name contxeng-flink-poc:1.4.0
--job-artifacts
/home/develk/cntx_eng/FlinkContextEnginePoc-0.0.1-SNAPSHOT.jar
--from-archive
JOB_ARTIFACTS_PATH : /home/develk/cntx_eng/FlinkContextEnginePoc-0.0.1-SNAPSHOT.jar
FROM_ARCHIVE : /home/develk/cntx_eng/flink-1.4.0-bin-hadoop24-scala_2.11.tgz
HADOOP_VERSION :
FLINK_VERSION :
-------------------------Arg Values---------------------------
FLINK_DIST : _TMP_/flink.tgz
JOB_ARTIFACTS_TARGET : _TMP_/artifacts
SHADED_HADOOP :
IMAGE_NAME : contxeng-flink-poc:1.4.0
--------------------------------------------------------------
Sending build context to Docker daemon 606.6MB
Step 1/23 : FROM openjdk:8-jre-alpine
---> f7a292bbb70c
Step 2/23 : RUN apk add --no-cache bash snappy libc6-compat
---> Using cache
---> 9e84497f3616
Step 3/23 : ENV FLINK_INSTALL_PATH=/opt
---> Using cache
---> 87bc358ccf00
Step 4/23 : ENV FLINK_HOME $FLINK_INSTALL_PATH/flink
---> Using cache
---> 712ba8d54555
Step 5/23 : ENV FLINK_LIB_DIR $FLINK_HOME/lib
---> Using cache
---> 80e7b085252e
Step 6/23 : ENV FLINK_PLUGINS_DIR $FLINK_HOME/plugins
---> Using cache
---> 7d39101e47d3
Step 7/23 : ENV FLINK_OPT_DIR $FLINK_HOME/opt
---> Using cache
---> 9bff7fc7145d
Step 8/23 : ENV FLINK_JOB_ARTIFACTS_DIR $FLINK_INSTALL_PATH/artifacts
---> Using cache
---> b0c01f3aab84
Step 9/23 : ENV FLINK_USR_LIB_DIR $FLINK_HOME/usrlib
---> Using cache
---> f4236bc26cab
Step 10/23 : ENV PATH $PATH:$FLINK_HOME/bin
---> Using cache
---> 2cb7cd442b6f
Step 11/23 : ARG flink_dist=NOT_SET
---> Using cache
---> 1a6fc691baa2
Step 12/23 : ARG job_artifacts=NOT_SET
---> Using cache
---> e11400e03120
Step 13/23 : ARG python_version=NOT_SET
---> Using cache
---> 313089fd991e
Step 14/23 : ARG hadoop_jar=NOT_SET*
---> Using cache
---> ccbef4dfa806
Step 15/23 : RUN if [ "$python_version" = "2" ]; then apk add --no-cache python; elif [ "$python_version" = "3" ]; then apk add --no-cache python3 && ln -s /usr/bin/python3 /usr/bin/python; fi
---> Using cache
---> 7e6dca36dad4
Step 16/23 : ADD $flink_dist $hadoop_jar $FLINK_INSTALL_PATH/
---> 5afb7a8e5414
Step 17/23 : ADD $job_artifacts/* $FLINK_JOB_ARTIFACTS_DIR/
---> c2789d3d80b3
Step 18/23 : RUN set -x && ln -s $FLINK_INSTALL_PATH/flink-[0-9]* $FLINK_HOME && ln -s $FLINK_JOB_ARTIFACTS_DIR $FLINK_USR_LIB_DIR && if [ -n "$python_version" ]; then ln -s $FLINK_OPT_DIR/flink-python*.jar $FLINK_LIB_DIR; fi && if [ -f ${FLINK_INSTALL_PATH}/flink-shaded-hadoop* ]; then ln -s ${FLINK_INSTALL_PATH}/flink-shaded-hadoop* $FLINK_LIB_DIR; fi && addgroup -S flink && adduser -D -S -H -G flink -h $FLINK_HOME flink && chown -R flink:flink ${FLINK_INSTALL_PATH}/flink-* && chown -R flink:flink ${FLINK_JOB_ARTIFACTS_DIR}/ && chown -h flink:flink $FLINK_HOME
---> Running in c4cb70216f08
+ ln -s /opt/flink-1.4.0 /opt/flink-1.4.0-bin-hadoop24-scala_2.11.tgz /opt/flink
+ ln -s /opt/artifacts /opt/flink/usrlib
+ '[' -n ]
+ '[' -f /opt/flink-shaded-hadoop-2-uber-2.4.1-8.0.jar ]
+ ln -s /opt/flink-shaded-hadoop-2-uber-2.4.1-8.0.jar /opt/flink/lib
+ addgroup -S flink
+ adduser -D -S -H -G flink -h /opt/flink flink
+ chown -R flink:flink /opt/flink-1.4.0 /opt/flink-1.4.0-bin-hadoop24-scala_2.11.tgz /opt/flink-shaded-hadoop-2-uber-2.4.1-8.0.jar
+ chown -R flink:flink /opt/artifacts/
+ chown -h flink:flink /opt/flink
Removing intermediate container c4cb70216f08
---> 459b1156294b
Step 19/23 : COPY docker-entrypoint.sh /
---> d4ae4be34415
Step 20/23 : USER flink
---> Running in 95a2c9234cd5
Removing intermediate container 95a2c9234cd5
---> ebdc913c7dd9
Step 21/23 : EXPOSE 8081 6123
---> Running in f6fad553a1d7
Removing intermediate container f6fad553a1d7
---> 51e6c57d2bde
Step 22/23 : ENTRYPOINT ["/docker-entrypoint.sh"]
---> Running in 09e7c0759fb6
Removing intermediate container 09e7c0759fb6
---> 99cdeb095b8f
Step 23/23 : CMD ["--help"]
---> Running in f985bd546dcf
Removing intermediate container f985bd546dcf
---> 039086df61e6
Successfully built 039086df61e6
Successfully tagged contxeng-flink-poc:1.4.0

Seems like everything went well.
To see which containers are running:
docker ps
contxeng-flink-poc:1.4.0, should be in there.
If you actually want to interact with Flink you should expose some ports.
The Flink dashboard for example runs on port 8081 of your container.
You can also get an interactive bash shell into your container by running the following command (if your container has bash installed):
docker exec -it contxeng-flink-poc:1.4.0 bash
That's where you will find your Flink binaries.
If the container is not running, check if it was built:
docker images
If that's the case, run it:
docker run -d contxeng-flink-poc:1.4.0

Related

Can not connect redis-vm server using docker

I have use the Ubuntu Virtualbox.
Docker version : Docker version 20.10.12, build e91ed57
redis version: 3.0
From Ubuntu, I'm trying to initialize and connect to redis-vm server.
I have use following docker commands to build, create container and connect with the redis-server.
(base) root#adminsaturn-virtual-machine:/media/sf_scefwp01/redis# docker build --rm -f ./Dockerfile --tag redis:3.0 .
Sending build context to Docker daemon 1.571MB
Step 1/26 : FROM alpine:20210804 as redis-dev
---> 4e873038b87b
Step 2/26 : ENV REDIS_VERS "6.2.6"
---> Using cache
---> bd2bd3468229
Step 3/26 : ENV REDIS_REPO "https://github.com/redis/redis.git"
---> Using cache
---> 51831b501b6d
Step 4/26 : ENV REDISJSON_VERS "2.0.6"
---> Using cache
---> c9fbf3a205bc
Step 5/26 : ENV REDISJSON_REPO "https://github.com/RedisJSON/RedisJSON.git"
---> Using cache
---> a4d2767081c9
Step 6/26 : RUN apk add --no-cache coreutils gcc g++ cargo linux-headers make musl-dev git rust cargo clang-libs pkgconf
---> Using cache
---> a603a024df55
Step 7/26 : RUN git clone --depth 1 --branch ${REDIS_VERS} ${REDIS_REPO} redis && cd /redis && make -j "$(nproc)" && make install
---> Using cache
---> b784b1ea27e4
Step 8/26 : FROM alpine:3.14
---> dd53f409bf0b
Step 9/26 : RUN sed -i 's/https/http/' /etc/apk/repositories
---> Using cache
---> d5f72fd3c94e
Step 10/26 : RUN apk update && apk add bash
---> Using cache
---> fe89e27870d6
Step 11/26 : RUN apk add libgcc
---> Using cache
---> d4f8dec09ec0
Step 12/26 : RUN apk add --no-cache python3
---> Using cache
---> 558f61bd0567
Step 13/26 : RUN apk add --no-cache py3-pip
---> Using cache
---> f21fc80a1225
Step 14/26 : RUN pip3 install redis
---> Using cache
---> 634aae092977
Step 15/26 : RUN pip3 install pytest
---> Using cache
---> f03644be4ca8
Step 16/26 : COPY sysvar-py/ /sysvar-py/
---> Using cache
---> ca867b2bbc2e
Step 17/26 : ENV REDIS_URL=redis://localhost:6379
---> Using cache
---> 8e4ece4479d3
Step 18/26 : COPY startup/ /startup/
---> Using cache
---> fefed1c53dee
Step 19/26 : COPY --from=redis-dev /usr/local/bin/redis-* /usr/local/bin/
---> Using cache
---> 110bb43f3cfc
Step 20/26 : RUN mkdir /etc/redis
---> Using cache
---> 80f3fc513310
Step 21/26 : COPY --from=redis-dev /redis/redis.conf /etc/redis/
---> Using cache
---> 0fc7280f6738
Step 22/26 : RUN mkdir -p /usr/local/etc/redis
---> Using cache
---> 362645872a09
Step 23/26 : COPY redis.conf /usr/local/etc/redis/
---> Using cache
---> 9c545d22358f
Step 24/26 : EXPOSE 6379/tcp
---> Using cache
---> d41a476122fd
Step 25/26 : WORKDIR /sysvar-py
---> Using cache
---> 0cd5f4246142
Step 26/26 : ENTRYPOINT ["/startup/startup.sh"]
---> Using cache
---> 20ee87c5f5f4
Successfully built 20ee87c5f5f4
Successfully tagged redis:3.0
(base) root#adminsaturn-virtual-machine:/media/sf_scefwp01/redis# docker run --name=redis --publish=6379:6379 --hostname=redis --mount type=bind,source=/run,target=/run --detach redis:3.0
1fe16baec197920588783968fea43ceaba8286117d95b49e4d59b5a840d04241
(base) root#adminsaturn-virtual-machine:/media/sf_scefwp01/redis# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1fe16baec197 redis:3.0 "/startup/startup.sh" 5 seconds ago Exited (1) 4 seconds ago redis
c1aa99b92eca redis "docker-entrypoint.s…" 25 hours ago Exited (0) 25 hours ago funny_nobel
e236678456e2 redis "docker-entrypoint.s…" 25 hours ago Exited (0) 25 hours ago quirky_lovelace
2772ad6a9c9b 4e873038b87b "/bin/sh -c 'apk upd…" 28 hours ago Exited (2) 28 hours ago crazy_cohen
f34c6d1fde05 4e873038b87b "/bin/sh -c 'apk upd…" 28 hours ago Exited (2) 28 hours ago ecstatic_taussig
(base) root#adminsaturn-virtual-machine:/media/sf_scefwp01/redis# docker exec -it redis bash
Error response from daemon: Container 1fe16baec197920588783968fea43ceaba8286117d95b49e4d59b5a840d04241 is not running
Here you can see build and container creating successfully but redis-server exited immediately.
if we apply the docker exec -it redis bash command then getting Error response:Container is not running.
how can I fix this issue?
Kindly provide me suggestion.

volume already exists but was not created by Docker Compose

This is what i get after i use ddev start on new or any other projects.
I clear all docker images, volumes, etc... and problem repeats again...
Does anybody has the same problem, does it have any connection with versions:
Docker version 20.10.11
Docker Compose version 2.2.0
ddev version v1.18.0
Starting lmp...
Pushed mkcert rootca certs to ddev-global-cache/mkcert
Running Command=ip address show dev docker0
volume "lmp-mariadb" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume
Failed to start lmp: ComposeCmd failed to run 'COMPOSE_PROJECT_NAME=ddev-lmp docker-compose -f /home/loganmm/www/ddev/lmp/.ddev/.ddev-docker-compose-full.yaml up --build -d', action='[up --build -d]', err='exit status 1', stdout='Sending build context to Docker daemon 489B
Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM $BASE_IMAGE
---> 87d5f9bd7e6e
Step 3/6 : ARG username
---> Using cache
---> f60cc0e22e2e
Step 4/6 : ARG uid
---> Using cache
---> 01e2924ff925
Step 5/6 : ARG gid
---> Using cache
---> f76f3c567045
Step 6/6 : RUN (groupadd --gid $gid "$username" || groupadd "$username" || true) && (useradd -l -m -s "/bin/bash" --gid "$username" --comment '' --uid $uid "$username" || useradd -l -m -s "/bin/bash" --gid "$username" --comment '' "$username" || useradd -l -m -s "/bin/bash" --gid "$gid" --comment '' "$username")
---> Using cache
---> 74726db594e2
Successfully built 74726db594e2
Successfully tagged drud/ddev-dbserver-mariadb-10.3:v1.18.0-lmp-built
Sending build context to Docker daemon 448B
Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM $BASE_IMAGE
---> 1e45807db7e2
Step 3/6 : ARG username
---> Using cache
---> 99d798f1aa8c
Step 4/6 : ARG uid
---> Using cache
---> 30f01bb39027
Step 5/6 : ARG gid
---> Using cache
---> db8269893144
Step 6/6 : RUN (groupadd --gid $gid "$username" || groupadd "$username" || true) && (useradd -l -m -s "/bin/bash" --gid "$username" --comment '' --uid $uid "$username" || useradd -l -m -s "/bin/bash" --gid "$username" --comment '' "$username" || useradd -l -m -s "/bin/bash" --gid "$gid" --comment '' "$username")
---> Using cache
---> 059097a5008c
Successfully built 059097a5008c
Successfully tagged drud/ddev-webserver:v1.18.0-lmp-built
', stderr='volume "lmp-mariadb" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume'
If you use Docker Desktop - go to the preferences and under General uncheck Use Docker Compose v2, then click "Apply & Restart" - as found on https://github.com/drud/ddev/issues/3404#issuecomment-986990802
This is worked around in DDEV v1.18.2+ (and v1.19+), please upgrade. It was a bug in docker-compose 2.2.0+ - please see https://github.com/drud/ddev/issues/3404 for context.

Docker container not showing site based on newly copied site configuration

This is my Dockerfile.
FROM ubuntu:devel
MAINTAINER ciasto
RUN apt-get update && apt-get install -y apache2 && apt-get clean && rm -rf /var/lib/apt/lists/\*
RUN mkdir /var/run/apache2
COPY mysite.conf /etc/apache2/sites-available/000-default.conf
ENV APACHE\_RUN\_USER www-data
ENV APACHE\_RUN\_GROUP www-data
ENV APACHE\_LOG\_DIR /var/log/apache2
ENV APACHE\_RUN\_DIR /var/run/apache2
EXPOSE 80
CMD \["/usr/sbin/apache2", "-D", "FOREGROUND"\]
I am trying to load my site instead of default 000-default.conf but even though the mysite.conf overwrites 000-default.conf I still keep getting the default homepage of Apache instead.
It doesn't have any affect of a2dissite and then running a2ensite:
# docker build -t myapache .
Sending build context to Docker daemon 3.072kB
Step 1/13 : FROM ubuntu:devel
---> 3dd27c0da934
Step 2/13 : MAINTAINER ciasto
---> Using cache
---> ca48c7010bf5
Step 3/13 : RUN apt-get update && apt-get install -y apache2 && apt-get clean && rm -rf /var/lib/apt/lists/*
---> Using cache
---> 3893438f7d96
Step 4/13 : RUN mkdir /var/run/apache2
---> Using cache
---> 6cc65fe8d352
Step 5/13 : COPY mysite.conf /etc/apache2/sites-available/000-default.conf
---> Using cache
---> 469c402f7c57
Step 6/13 : RUN a2dissite 000-default
---> Running in 2de611818fc0
Site 000-default disabled.
To activate the new configuration, you need to run:
service apache2 reload
Removing intermediate container 2de611818fc0
---> 64d8deed8833
Step 7/13 : RUN a2ensite 000-default
---> Running in cc8a0374ab0d
Enabling site 000-default.
To activate the new configuration, you need to run:
service apache2 reload
Removing intermediate container cc8a0374ab0d
---> 3af0de88b74b
Step 8/13 : ENV APACHE_RUN_USER www-data
---> Running in fa2146657f5f
Removing intermediate container fa2146657f5f
---> dac302f42282
Step 9/13 : ENV APACHE_RUN_GROUP www-data
---> Running in 7a3035b5866d
Removing intermediate container 7a3035b5866d
---> fdfc7a1d86e4
Step 10/13 : ENV APACHE_LOG_DIR /var/log/apache2
---> Running in 93cc22c81f55
Removing intermediate container 93cc22c81f55
---> d8e073243eed
Step 11/13 : ENV APACHE_RUN_DIR /var/run/apache2
---> Running in ca2b2829d60c
Removing intermediate container ca2b2829d60c
---> c7f535e2c031
Step 12/13 : EXPOSE 80
---> Running in 613564334e15
Removing intermediate container 613564334e15
---> adae0f0e2aed
Step 13/13 : CMD ["apachectl", "-D", "FOREGROUND"]
---> Running in 97c845feed9f
Removing intermediate container 97c845feed9f
---> 67f7823b995b
Successfully built 67f7823b995b
Successfully tagged myapache:latest
then I still get default page of Apache and checked by entering running container:
# apachectl -S
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex watchdog-callback: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
This should fix your problem here. To be add after the COPY:
RUN a2dissite 000-default
RUN a2ensite 000-default

How to include Yourkit inside a neo4j docker image?

while trying to use the 3.5.5 enterprise Dockerfile (https://github.com/neo4j/docker-neo4j-publish/blob/master/3.5.5/community/Dockerfile) and to add YourKit's Docker related commands (https://www.yourkit.com/docs/java/help/docker.jsp)
so first I tried to docker build the Dockerfile without the Yourkit changes but I get this error:
Sending build context to Docker daemon 3.072kB │·······················
Step 1/13 : FROM openjdk:8-jre-alpine │·······················
---> f7a292bbb70c │·······················
Step 2/13 : RUN addgroup -S neo4j && adduser -S -H -h /var/lib/neo4j -G neo4j neo4j │·······················
---> Using cache │·······················
---> e0b1355abc58 │·······················
Step 3/13 : ENV NEO4J_SHA256=0421bb95fd377a1a45cd0a22f3f4163d8b2b07d1f9d8979e42c7f1952961ab92 NEO4J_TARBALL=neo4j-enterprise-3.5.5-unix.tar.gz NEO4J_EDITION=enterprise │·······················
NEO4J_HOME="/var/lib/neo4j" │·······················
---> Running in f33c9400f05b │·······················
Removing intermediate container f33c9400f05b │·······················
---> b5db09236a0f │·······················
Step 4/13 : ARG NEO4J_URI=http://dist.neo4j.org/neo4j-enterprise-3.5.5-unix.tar.gz │·······················
---> Running in 722c0d5ef000 │·······················
Removing intermediate container 722c0d5ef000 │·······················
---> 05ff355a8396 │·······················
Step 5/13 : COPY ./local-package/* /tmp/ │·······················
COPY failed: no source files were specified```
(edited)
Once this works I will add the yourkit specific RUN commands.
One could simply do the following instead:
FROM neo4j:3.5.5
RUN wget https://www.yourkit.com/download/docker/YourKit-JavaProfiler-2019.1-docker.zip -P /tmp/ && \
unzip /tmp/YourKit-JavaProfiler-2019.1-docker.zip -d /usr/local && \
rm /tmp/YourKit-JavaProfiler-2019.1-docker.zip
RUN apk add --no-cache --quiet libc6-compat
ENV PATH "${NEO4J_HOME}"/bin:/lib64:$PATH
ENV LD_LIBRARY_PATH /lib64:$LD_LIBRARY_PATH
VOLUME /data /logs
EXPOSE 7474 7473 7687 10001
ENTRYPOINT ["/sbin/tini", "-g", "--", "/docker-entrypoint.sh"]
CMD ["neo4j"]

dockerfile Add file can not be found

Server Version: 18.03.1-ce ,RHEL 7.2 .Here is my dockerfile:
FROM openjdk:8-jdk-alpine
ENV http_proxy http://192.168.156.25:3128
ENV https_proxy http://192.168.156.25:3128
RUN apk update && apk upgrade && apk add netcat-openbsd
RUN mkdir -p /usr/local/licensingservice
ADD #project.build.finalName#.jar /usr/local/licensingservice/
ADD run.sh run.sh
RUN chmod +x run.sh
CMD ./run.sh
it build without error:
---> Using cache
---> 8fa60876c229
Step 5/9 : RUN mkdir -p /usr/local/licensingservice
---> Using cache
---> bca46b1256e1
Step 6/9 : ADD licensing-service-0.0.1-SNAPSHOT.jar /usr/local/licensingservice/
---> a66979ed3755
Step 7/9 : ADD run.sh ./run.sh
---> 95b492565374
Step 8/9 : RUN chmod +x run.sh
---> Running in eec3075c30f3
Removing intermediate container eec3075c30f3
---> 96a2d7b89b80
Step 9/9 : CMD ./run.sh
---> Running in c338e9d33371
Removing intermediate container c338e9d33371
---> 324d5a83cf84
ProgressMessage{id=null, status=null, stream=null, error=null, progress=null, progressDetail=null}
Successfully built 324d5a83cf84
Successfully tagged johncarnell/tmx-licensing-service:chapter4
but docker run -it 324d5a83cf84:
/bin/sh: ./run.sh: not found
I debug using docker run --rm -it 324d5a83cf84 cat ./run.sh,it print the file well.
run.sh:
#!/bin/sh
echo "hello1"
I suspect you are working on Windows and you are using the default Windows newline: CR LF. Change to LF in your run.sh and it will work like a charm.
But how to do that you ask? Open run.sh in Notepad++ and look at the bottom right of the window. Click on Windows (CR LF) and select Unix (LF).

Resources