I'm building what I think is a simple dockerfile and have got one line in the code that is throwing an error.
# dockerfile
...
RUN curl -k --output bin/theta `curl -k 'https://mainnet-data.thetatoken.org/binary?os=linux&name=theta'`
RUN curl -k --output bin/thetacli `curl -k 'https://mainnet-data.thetatoken.org/binary?os=linux&name=thetacli'`
RUN curl -k --output guardian_mainnet/node/config.yaml `curl -k 'https://mainnet-data.thetatoken.org/config?is_guardian=true'`
...
The first two curl commands run without any issue. The third curl command throws an error:
Warning: Failed to create the file guardian_mainnet/node/config.yaml: No such file or directory
The directory is created and does exist. The prior two curl commands use exactly the same format and result in both the theta and thetacli files being created.
I've actually setup a docker container with the base image predicated on the FROM for this dockerfile. From there I've run the code in the dockerfile line-by-line and it has executed without any problem (including the third line). In other words, if I manually run the dockerfile commands from the CLI for the base container it works - it's only when building the container from the dockerfile at the host level that the error is thrown.
The only differences are (i) the file type of .yaml and (ii) the ? in the https link. But I've found nothing that says that would be a problem. [I've tried saving without the extension and it didn't make a difference.]
What am I missing?
Related
I'm trying to populate data using REST endpoints of another container (i.e. my-blog-container) by creating a container (data-loader), for which I'm running a shell script dataloader.sh inside container (i.e. data-loader). In the shell script I'm trying to access the json file (i.e. mydata.json) to make the POST call in shell script while running the container.
Basically Goal is to populate the data by spinning up the container on its
own.
But I'm getting an error in my container:
Warning: Couldn't read data from file
Warning: "mydata.json", this makes an empty
Warning: POST.
docker-compose.yaml
data-loader:
<<: *my-base-service
build:
context: .
dockerfile: Dockerfile.dataloader
args:
IMAGE_NAME: <alpine base image>
container_name: data-loader
volumes:
- $MY_PATH/config/mydata.json
Dockerfile.dataloader
ARG IMAGE_NAME
FROM ${IMAGE_NAME}
USER root
RUN apk update && apk upgrade && apk add curl
COPY dataloader.sh /
RUN chmod +x /dataloader.sh
ENTRYPOINT [ "sh", "/dataloader.sh" ]
dataloader.sh
response=$(curl -X POST "http://myblogcontainer:8080/blog/create" -H "accept: */*" -H "Content-Type: application/json" \
-d #mydata.json)
echo ${response}
I want to populate the data using this json file. But unable to make it. Please help me out.
If you have any questions, please feel free to ask.
Thanks in advance :)
Given the assumption that you have a colon as part of MY_PATH, you mount the file into /config/ but you are using a relative path in your script.
If your workdir is not /config, then your script will not find the file. To be sure you could use an absolute path.
-d '#/config/mydata.json'
In general, it is useful in these kinds of scenarios to do some debugging by entering the container interactively to poke around in the file system. Or by adding some print statements to your script to ensure your assumptions are correct.
Also note that your script is missing a shebang. The default interpreter is actually sh, so it works out, but it is good practice to add one anyway.
You probably should also double quote the variable response when using to prevent word splitting and globbing.
I am trying to fork this docker image so that if anything changes on the original it won't affect me.
I have forked the repo corresponding to that image to my own repo.
I have cloned the repo and am trying to build it:
docker build . -t davcal/gcc-cross-x86_64-elf
I am getting this error:
+ cd /usr/local/src
+ ./build-binutils.sh 2.31.1
/bin/sh: 1: ./build-binutils.sh: not found
The command '/bin/sh -c set -x && cd /usr/local/src && ./build-binutils.sh ${BINUTILS_VERSION} && ./build-gcc.sh ${GCC_VERSION}' returned a non-zero code: 127
What makes no sense to me is that if I use the original image, it builds successfully:
FROM randomdude/gcc-cross-x86_64-elf
...
Maybe Docker Hub stores a pre-built image?
How do I fix this?
Note: I am using Windows. This shouldn't make a difference since the error originates within the container.
Edit
I tried patching the Dockerfile to chmod executable permissions to the sh files in case that was causing problems on Windows. Unfortunately, the exact same error occurs.
RUN set -x \
&& chmod +x /usr/local/src/build-binutils.sh \
&& chmod +x /usr/local/src/build-gcc.sh \
&& cd /usr/local/src \
&& ./build-binutils.sh ${BINUTILS_VERSION} \
&& ./build-gcc.sh ${GCC_VERSION}
Edit 2
Following this method, I inspected the container to see if the sh files actually exist. Here is the output.
I ran docker run --rm -it c53693f11514 bash, including the hash of the intermediate container of the previous successful step of the Dockerfile.
This is the output showing that the files do exist:
root#9b8a64ac2090:/# cd usr/local/src
root#9b8a64ac2090:/usr/local/src# ls
binutils-2.31.1 build-binutils.sh build-gcc.sh gcc-8.2.0
From the described symptoms, file exists, is a shell script, and works on other machines, the "file not found" error is most likely from Winidows linefeeds being added to the file. When the Linux kernel processes a shell script, it looks at the first line, the #!/bin/sh or similar, and then finds that interpreter to run the shell script. If that interpreter isn't found, you'll get a "file not found" error.
In this case, the file it's looking for won't be /bin/sh, but instead /bin/sh\r or /bin/sh^M depending on how you want to represent the carriage return character. You can fix that for single files with a tool like dos2unix but in general, you'll want to fix git itself since there are likely other files that have had their linefeeds corrupted. For details on adjusting the behavior of git, see this post.
I'm trying to run a docker container from snakemake. The jobs run and produce the correct output but when they complete I get
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
and snakemake tries to remove the files.
Things I've tried to debug this:
Adding || true, set +u or exit 0 to the shell command in the rule
Running the commands shown in --printshellcmds (they run fine)
Putting the docker command in a bash script in 'strict' mode to see
if there are any issues (there are not)
Printing the actual exit
code in the shell: command of the rule and it prints 0
Running the R script which is being called in the container directly
from the snakemake (it works fine)
Adding --user $(id -u):$(id -g) to the docker run command suggested here: Snakemake claims rule exits with non-zero exit code, even with "|| true"?. This
fails as the R script in the container has nowhere to write out
intermediate files as my current user does not properly exist in the container
Here is the snakefile rule:
rule run_biospyder:
input:
counts = "{dir}/{name}_counts.csv",
metadata = "{dir}/{name}_metadata.csv",
output: directory("{dir}/{name}_output")
params:
dockervol = "/usr/src/data"
shell:
"""
docker run \
-v $PWD:{params.dockervol} biospyderpipeline \
--config-file {params.dockervol}/config.yml \
--counts {params.dockervol}/{input.counts} \
--samples {params.dockervol}/{input.metadata} \
--output {params.dockervol}/{output} \
--name {wildcards.name}
"""
The issue is a docker permissions issue when mapping volumes. The problem seems to be due to inside container being run as root (I'm just developing) and snakemake tries to write out .snakemake_timestamp as my current user to output directory of the mapped volume (which is owned by root).
I managed to fix it with this: https://denibertovic.com/posts/handling-permissions-with-docker-volumes/
The shell script at the entrypoint creates a user with the same id and runs any commands as that user. This means the mapped volume contents are owned by by user and Snakemake can write to it at the end of the run. This also doesn't fix the user id at the docker build so anyone can use it.
I have created a declarative jenkins pipeline and one of it's stages is as follows:
stage('Docker Image'){
steps{
bat 'docker build -t HMT/demo-application:%BUILD_NUMBER% --no-cache -f Dockerfile .'
}
}
This is the docker file:
FROM tomcat:alpine
RUN wget -O /usr/local/tomcat/webapps/launchstation04.war http://localhost:8082/artifactory/demoArtifactory/com/demo/0.0.1-SNAPSHOT/demo-0.0.1-SNAPSHOT.war
EXPOSE 9100
CMD /usr/local/tomcat/bin/cataline.bat run
I am getting the below error.:
[91m/bin/sh:
01:33:28 [0mThe command '/bin/sh -c wget -O /usr/local/tomcat/webapps/launchstation04.war http://localhost:8082/artifactory/demoArtifactory/com/demo/0.0.1-SNAPSHOT/demo-0.0.1-SNAPSHOT.war' returned a non-zero code: 127
UPDATE:
I have updated the command to
RUN wget -O /usr/local/tomcat/webapps/launchstation04.war -U jenkinsuser:Learning#% http://localhost:8082/artifactory/demoArtifactory/com/demo/0.0.1-SNAPSHOT/demo-0.0.1-20200823.053346-18.war
There is no problem in my command.Jfrog artifactory was unable to authorize this action.So I added username and password details but it still didn't work.
Error:
wget: server returned error: HTTP/1.1 401 Unauthorized
It didnt work after modifiying the password policy to unsupported.But it worked when I allowed anonymous access.
How to provide access using credentials.
Need more clarification on your question. Not sure where you are using curl command.
Image tomcat:alpine doesn't contains curl command. Unless you install it manually.
bash-4.4# type curl
bash: type: curl: not found
bash-4.4#
If your ask is regarding the sh -c option, if the script is invoked through CMD option, yes it will use sh. Instead you can give a try with ENTRYPOINT.
You can provide username & password via command line:
wget --user user --password pass
Using curl :
curl -u username:password -O
But void using special characters:
Change your password to another once in: [a-z][A-Z][0-9]
Try an API Key instead of password, I have a feeling that "#" may be throwing you off. Quotes can help there too or separating the password with -p
Also look at the request logs for whether the entry comes as 401 for the user, or anonymous/unauthenticated
Lastly, see if you can cURL from outside the image and then ADD the file in, as that will remove any external factors that may vary from the host (where I assume the command works)
I'm running this docker image to use the TICK Kapacitor locally.
The problem I face is that when I try to use User Defined Functions, e.g any of these examples I get the error message that /usr/bin/python2 does not exist.
I add the following to the kapacitor.conf:
[udf.functions]
[udf.functions.tTest]
prog = "/usr/bin/python2"
args = ["-u", "/tmp/kapacitor_udf/mirror.py"]
timeout = "10s"
[udf.functions.tTest.env]
PYTHONPATH = "/tmp/kapacitor_udf/kapacitor/udf/agent/py"
Further attempts from my side including altering the image used to build Kapacitor to install python works but the agent seems to fail to compile anyway.
Is there anyone who managed to get UDFs running using the Kapacitor Docker image?
Thanks
Docker image from the official repository: docker pull kapacitor does not have python installed inside. You can verify this by running shell in the container:
PS> docker exec -it kapacitor bash
and execute one of the command options:
$ python -VERSION
$ python: command not found
or
$ readlink -f $(which python) | xargs -I% sh -c 'echo -n "%:"; % -V'
$ readlink: missing operand
or
$ find / -type f -executable -iname 'python *'
void returns. And oppositely if python is available, commands return version and list of executable files
Note: Here and further all command snippets are given for Powershell on Windows. And all command snippets inside docker container are given for bash shell as Linux images are used.
Basicly, there is two options to get kapacitor image with python inside to execute UDFs:
Install the python in the kapacitor image, i.e. build new docker image from very kapacitor image.
Example could be found here:
Build a new verion of kapacitor image from one of the python official images
The second option is more natural as you get consistent python installation and keep efforts on doing work of installing python which already done by the docker community.
So following option 2 we'll perform:
Examine the Dockefile of the official kapacitor image
Choose an appropriate python image
Create new project and Dockerfile for kapacitor
Build and Check the kapacitor image
Examine Dockefile of official kapacitor image
General note:
For any image, the original Dockerfiles can be obtained in this way:
https://hub.docker.com/
-> Description Tab
-> Supported tags and respective Dockerfile links section
-> each of the tags is a link that leads to the Dockerfile
So for kapacitor everything is in the influxdata-docker git repository
Then in the Dockerfile we find that the image is created based on
FROM buildpack-deps: stretch-curl
here:
buildpack-deps
the image provided by the project of the same name https://hub.docker.com/_/buildpack-deps
curl
This variant includes just the curl, wget, and ca-certificates packages. This is perfect for cases like the Java JRE, where downloading JARs is very common and
necessary, but checking out code isn't.
stretch
short version name of the OS, in this case Debian 9 stretch https://www.debian.org/News/2017/20170617
Buildpack-deps images are in turn built based on
FROM debian: stretch
And Debian images from the minimum docker image
FROM: scratch
Choose appropriate python image
Among python images, for example 3.7, you can find similar versions inheriting from buildpack-deps
FROM buildpack-deps: stretch
Following the inheritance, we'll see:
FROM buildpack-deps: stretch
FROM buildpack-deps: stretch-smc
FROM buildpack-deps: stretch-curl
FROM debian: stretch
In other words, the python: 3.7-stretch image only adds functionality to the Debian compared to the kapacitor image.
This means that we can to rebuild kapacitor image on top of the python image: 3.7-stretch with no risk or gaining incompatibility.
Docker context folder preparation
Clone the repository
https://github.com/influxdata/influxdata-docker.git
Create the folder influxdata-docker/kapacitor/1.5/udf_python/python3.7
Copy the following three files into it from influxdata-docker/kapacitor/1.5/:
Dockerfile
entrypoint.sh
kapacitor.conf
In the copied Dockerfile FROM buildpack-deps: stretch-curl replace with FROM python: 3.7-stretch
Be carefuly! If we work on Windows and because of scientific curiosity open the entrypoint.sh file in the project folder, then be sure to check that it does not change the end-line character from Linux (LF) to Windows variant: (CR LF).
Otherwise, when you start the container later, you get an error:
or in the container log:
exec: bad interpreter: No such file or directory
or if you'll start debugging and, running the container with bash, will do:
$ root # d4022ac550d4: / # exec /entrypoint_.sh
$ bash: /entrypoint_.sh: / bin / bash ^ M: bad interpreter: No such file or directory
Building
Run PS> docker build -f. \ Dockerfile -t kapacitor_python_udf
Again, in case of Windows environment
If during the build execution an error occurs of the form:
E: Release file for http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease is not valid yet (invalid for another 9h 14min 10s). Updates for this repository will not be applied.
then your computer clock probably went out of sync and/or Docker Desktop incorrectly initialized the time after the system returned from sleep. See the issue)
To fix it, restart Docker Desktop and / or Windows settings -> Date and time settings -> Clock synchronization -> perform Sync
You can also read more here
Launch and check
Launching the container with the same actions as for the standard image. Example:
PS> docker run --name=kapacitor -d `
--net=influxdb-network `
-h kapacitor `
-p 9092:9092 `
-e KAPACITOR_INFLUXDB_0_URLS_0=http://influxdb:8086 `
-v ${PWD}:/var/lib/kapacitor `
-v ${PWD}/kapacitor.conf:/etc/kapacitor/kapacitor.conf:ro `
kapacitor
Check:
PS> docker exec -it kapacitor_2 bash
$ python -VERSION
$ Python 3.7.7
$ readlink -f $(which python) | xargs -I% sh -c 'echo -n "%:"; % -V'
$ /usr/local/bin/python3.7: Python 3.7.7