Trying to install ckan from Dockerfile in:
Docker Community Edition
Version 17.06.2-ce-mac27 (19124)
Channel: stable
428bd6ceae*
FIRST ATTEMPT
These are the steps followed:
$ docker pull ckan/solr
$ docker pull ckan/ckan
$ docker pull ckan/postgresql
After downloaded the images I get:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ckan/solr latest 4acd7db7b1f7 3 days ago 517MB
ckan/ckan latest 77dd30c92740 3 days ago 642MB
ckan/postgresql latest 3c3ecd94ae7e 3 days ago 265MB
For each Dockerfile I made:
For solr
$ docker build . (for SOLR)
Step 7/9
ADD failed: stat /var/lib/docker/tmp/docker-builder234804113/solrconfig.xml: no such file or directory
For postgresql
$ docker build . (for POSTGRESQL)
Successfully built 8b296a6e3153
For ckan
$ docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan
Error in Step 7/26 :
Get:1 http://security.debian.org/ jessie/updates/main perl-base amd64 5.20.2-3+deb8u9 [1226 kB]
debconf: delaying package configuration, since apt-utils is not installed
And Fail in:
Step 12/26 : ADD ./requirements.txt $CKAN_HOME/src/ckan/requirements.txt
ADD failed: stat /var/lib/docker/tmp/docker-builder814560705/requirements.txt: no such file or directory
SECOND ATTEMPT
Removed images
Pull again images
$ docker pull ckan/solr
$ docker pull ckan/ckan
$ docker pull ckan/postgresql
And Launch:
$ docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan
for the Dockerfile in: https://hub.docker.com/r/ckan/ckan/~/dockerfile/
and I get the same error in the same step:
Step 12/26 : ADD ./requirements.txt $CKAN_HOME/src/ckan/requirements.txt
ADD failed: stat /var/lib/docker/tmp/docker-builder254648764/requirements.txt: no such file or directory
but the log in Docker Hub is correct.
Am I doing something wrong?????
Dockerfile
FROM debian:jessie
MAINTAINER Open Knowledge
ENV CKAN_HOME /usr/lib/ckan/default
ENV CKAN_CONFIG /etc/ckan/default
ENV CKAN_STORAGE_PATH /var/lib/ckan
ENV CKAN_SITE_URL http://localhost:5000
# Install required packages
RUN apt-get -q -y update && apt-get -q -y upgrade && DEBIAN_FRONTEND=noninteractive apt-get -q -y install \
python-dev \
python-pip \
python-virtualenv \
libpq-dev \
git-core \
build-essential \
libssl-dev \
libffi-dev \
&& apt-get -q clean
# SetUp Virtual Environment CKAN
RUN mkdir -p $CKAN_HOME $CKAN_CONFIG $CKAN_STORAGE_PATH
RUN virtualenv $CKAN_HOME
RUN ln -s $CKAN_HOME/bin/pip /usr/local/bin/ckan-pip
RUN ln -s $CKAN_HOME/bin/paster /usr/local/bin/ckan-paster
# SetUp Requirements
ADD ./requirements.txt $CKAN_HOME/src/ckan/requirements.txt
RUN ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/requirements.txt
# TMP-BUGFIX https://github.com/ckan/ckan/issues/3388
ADD ./dev-requirements.txt $CKAN_HOME/src/ckan/dev-requirements.txt
RUN ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/dev-requirements.txt
# TMP-BUGFIX https://github.com/ckan/ckan/issues/3594
RUN ckan-pip install --upgrade urllib3
# SetUp CKAN
ADD . $CKAN_HOME/src/ckan/
RUN ckan-pip install -e $CKAN_HOME/src/ckan/
RUN ln -s $CKAN_HOME/src/ckan/ckan/config/who.ini $CKAN_CONFIG/who.ini
# SetUp EntryPoint
COPY ./contrib/docker/ckan-entrypoint.sh /
RUN chmod +x /ckan-entrypoint.sh
ENTRYPOINT ["/ckan-entrypoint.sh"]
# Volumes
VOLUME ["/etc/ckan/default"]
VOLUME ["/var/lib/ckan"]
EXPOSE 5000
CMD ["ckan-paster","serve","/etc/ckan/default/ckan.ini"]
UPDATE 1
As suggested, added the content of the ckan package from:
http://packaging.ckan.org/
Copied to my Dockerfile directory /data/usr/lib/ckan/default/src/ckan/*
New error in step 15 (see comments)
UPDATE 2
Cloned ckan from: https://github.com/ckan/ckan
$ docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan
Result:
Step 26/26 : CMD ckan-paster serve /etc/ckan/default/ckan.ini
---> Running in eeeb6ace6ee1
---> 3cd87cf4a1af
Removing intermediate container eeeb6ace6ee1
Successfully built 3cd87cf4a1af
Successfully tagged ckan:latest
docker: Error response from daemon: could not get container for db: No such container: db.
Trying to access to ckan:
docker run -it ckan bash
I get this error:
Distribution already installed:
ckan 2.8.0a0 from /usr/lib/ckan/default/src/ckan
Creating /etc/ckan/default/ckan.ini
Now you should edit the config files
/etc/ckan/default/ckan.ini
Traceback (most recent call last):
File "/usr/local/bin/ckan-paster", line 11, in <module>
sys.exit(run())
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
result = self.command()
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 348, in command
self._load_config(cmd!='upgrade')
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 321, in _load_config
self.site_user = load_config(self.options.config, load_site_user)
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 230, in load_config
load_environment(conf.global_conf, conf.local_conf)
File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 111, in load_environment
p.load_all()
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 129, in load_all
unload_all()
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 182, in unload_all
unload(*reversed(_PLUGINS))
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 210, in unload
plugins_update()
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 121, in plugins_update
environment.update_config()
File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 289, in update_config
engine = sqlalchemy.engine_from_config(config, client_encoding='utf8')
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 428, in engine_from_config
return create_engine(url, **options)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
return strategy.create(*args, **kwargs)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 50, in create
u = url.make_url(name_or_url)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 194, in make_url
return _parse_rfc1738_args(name_or_url)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 240, in _parse_rfc1738_args
return URL(name, **components)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 60, in __init__
self.port = int(port)
ValueError: invalid literal for int() with base 10: ''
UPDATE 3 (matt fullerton suggestion)
I have cloned https://github.com/parksandwildlife/ckan.git in a folder called ckan
I have cloned https://github.com/parksandwildlife/ckan/tree/3649-docker-upgrade in another folder called ckan-3649
In ckan folder
docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan
In Step 15/26 :
RUN ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/dev-requirements.txt
ERROR:
Cleaning up...
Command /usr/lib/ckan/default/bin/python2 -c "import setuptools, tokenize;__file__='/tmp/pip-build-BLM6DJ/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-Vynm83-record/install-record.txt --single-version-externally-managed --compile --install-headers /usr/lib/ckan/default/include/site/python2.7 failed with error code 1 in /tmp/pip-build-BLM6DJ/cryptography
Storing debug log for failure in /root/.pip/pip.log
The command '/bin/sh -c ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/dev-requirements.txt' returned a non-zero code: 1
In ckan-3649/contrib/docker "docker-compose up"
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b6d7a5a595ca redis:latest "docker-entrypoint..." 9 minutes ago Up 9 minutes 6379/tcp redis
4551ccea28f4 ckan/solr:latest "docker-entrypoint..." 9 minutes ago Up 9 minutes 8983/tcp solr
86cea84b6ab5 docker_db "docker-entrypoint..." 9 minutes ago Up 9 minutes 5432/tcp db
fca64f1bee5a clementmouchet/datapusher "python datapusher..." 9 minutes ago Up 9 minutes 0.0.0.0:8800->8800/tcp datapusher
But I think ckan image is missing....
There is a major upgrade of the docker functionality in CKAN underway. I would suggest cloning CKAN from here:
https://github.com/parksandwildlife/ckan
git clone https://github.com/parksandwildlife/ckan.git
Then checkout the origin/3649-docker-upgrade branch
(https://github.com/parksandwildlife/ckan/tree/3649-docker-upgrade)
git checkout origin/3649-docker-upgrade
Then use docker-compose in the contrib/docker folder:
docker-compose up
This should assemble Solr, Postgres etc. for you.
Comments on mileage with this at https://github.com/ckan/ckan/pull/3692 will also be appreciated.
Related
I am trying to integrate my anaconda package build system with Codeship CI/CD service, therefore I need my build process to happen inside a Docker container.
Currently my Dockerfile looks like so:
FROM continuumio/miniconda3
COPY . .
RUN conda create --yes --name build-env python=3.8 \
&& conda install -n build-env conda-build -y \
&& conda run -n build-env conda-build --channel haasad .
RUN conda create --yes --name testing-env python=3.8 \
&& conda install -n testing-env --use-local sten -c haasad \
&& conda install -n testing-env -c anaconda pytest
When the build runs, the following error occurs:
/opt/conda/envs/build-env/conda-bld/chardet_1591782226225/work/conda_build.sh: line 4: /tmp/build/80754af9/chardet_1573033772973/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/bin/python: No such file or directory
Traceback (most recent call last):
File "/opt/conda/envs/build-env/bin/conda-build", line 11, in <module>
sys.exit(main())
File "/opt/conda/envs/build-env/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 469, in main
execute(sys.argv[1:])
File "/opt/conda/envs/build-env/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 458, in execute
outputs = api.build(args.recipe, post=args.post, build_only=args.build_only,
File "/opt/conda/envs/build-env/lib/python3.8/site-packages/conda_build/api.py", line 208, in build
return build_tree(absolute_recipes, config, stats, build_only=build_only, post=post,
File "/opt/conda/envs/build-env/lib/python3.8/site-packages/conda_build/build.py", line 2339, in build_tree
packages_from_this = build(metadata, stats,
File "/opt/conda/envs/build-env/lib/python3.8/site-packages/conda_build/build.py", line 1491, in build
utils.check_call_env(cmd, env=env, rewrite_stdout_env=rewrite_env,
File "/opt/conda/envs/build-env/lib/python3.8/site-packages/conda_build/utils.py", line 398, in check_call_env
return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
File "/opt/conda/envs/build-env/lib/python3.8/site-packages/conda_build/utils.py", line 378, in _func_defaulting_env_to_os_environ
raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-o', 'errexit', '/opt/conda/envs/build-env/conda-bld/chardet_1591782226225/work/conda_build.sh']' returned non-zero exit status 127.
The repo
How to fix this and what am I doing wrong?
My final purpose is to run experiment from an Api.
the experiment come from :
https://github.com/mlflow/mlflow/tree/master/examples/tensorflow/tf2
but export the file in my custom git where I clone it, in the image below ->
I have 2 images in my docker compose :
tree project :
|_app/
| |_Dockerfile
|
|_mlflow/
| |_Dockerfile
|
|_docker-compose.yml
app/Dockerfile
FROM continuumio/anaconda3
ENV APP_HOME ./
WORKDIR ${APP_HOME}
RUN conda config --append channels conda-forge
RUN conda install --quiet --yes \
'mlflow' \
'psycopg2' \
'tensorflow'
RUN pip install pylint
RUN pwd;ls \
&& git clone https://github.com/MChrys/QuickSign.git
RUN pwd;ls \
&& cd QuickSign \
&& pwd;ls
COPY . .
#RUN conda install jupyter
#CMD jupyter notebook --ip=0.0.0.0 --port=8888 --allow-root --no-browser
CMD cd QuickSign && mlflow run .
mlflow/Dockerfile
FROM python:3.7.0
RUN pip install mlflow
RUN mkdir /mlflow/
CMD mlflow server \
--backend-store-uri /mlflow \
--host 0.0.0.0
docker-compose.yml
version: '3'
services:
notebook:
build:
context: ./app
ports:
- "8888:8888"
depends_on:
- mlflow
environment:
MLFLOW_TRACKING_URI: 'http://mlflow:5000'
mlflow:
build:
context: ./mlflow
expose:
- "5000"
ports:
- "5000:5000"
when I docker-compose up the image I obtain :
notebook_1_74059cdc20ce | response = requests.request(**kwargs)
notebook_1_74059cdc20ce | File "/opt/conda/lib/python3.7/site-packages/requests/api.py", line 60, in request
notebook_1_74059cdc20ce | return session.request(method=method, url=url, **kwargs)
notebook_1_74059cdc20ce | File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
notebook_1_74059cdc20ce | resp = self.send(prep, **send_kwargs)
notebook_1_74059cdc20ce | File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
notebook_1_74059cdc20ce | r = adapter.send(request, **kwargs)
notebook_1_74059cdc20ce | File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
notebook_1_74059cdc20ce | raise ConnectionError(e, request=request)
notebook_1_74059cdc20ce | requests.exceptions.ConnectionError: HTTPConnectionPool(host='mlflow', port=5000): Max retries exceeded with url: /api/2.0/mlflow/runs/create (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd5db4edc50>: Failed to establish a new connection: [Errno 111] Connection refused'))
The problem look like that I run a project which is not found in the server images, as I run it in the app image, but I don't know how figure it out I have to trigger the experiment from a futur flask app
It looks like the server is not reachable from the app. I assume docker-compose.yml is used by the project running in the app, so that is trying to contact the mlflow server at MLFLOW_TRACKING_URI: 'http://mlflow:5000'. Is http://mlflow:5000 a domain you have set up? Where is it supposed to be reachable from?
The problem came from docker for windows, I was unable to make working docker compose on it but there are no problem to build it when I run it on virtual machine with ubuntu.
I'm using fastlane container that stores at google container registry to upload APK to google play store using Google Cloud Build.
APK has been succesfully created.However, when processing last step (fastlane), it face errors:
Step #2: 487ea6dabc0c: Pull complete
Step #2: a7ae4fee33c9: Pull complete
Step #2: Digest: sha256:2e31d5ae64984a598856f1138c6be0577c83c247226c473bb5ad302f86267545
Step #2: Status: Downloaded newer image for gcr.io/myapp789-app/fastlane:latest
Step #2: gcr.io/myapp789-app/fastlane:latest
Step #2: docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"supply\": executable file not found in $PATH": unknown.
Step #2: time="2019-08-29T23:22:55Z" level=error msg="error waiting for container: context canceled"
Finished Step #2
ERROR
ERROR: build step 2 "gcr.io/myapp789-app/fastlane" failed: exit status 127
Note:
1) Docker Source file was taken from https://hub.docker.com/r/fastlanetools/fastlane and then I build my own image.
2) Docker Image Build on Google Cloud VM using Debian GNU/Linux 9 (stretch)
Docker Source File for fastlane:
# Final image #
###############
FROM circleci/ruby:latest
MAINTAINER milch
ENV PATH $PATH:/usr/local/itms/bin
# Java versions to be installed
ENV JAVA_VERSION 8u131
ENV JAVA_DEBIAN_VERSION 8u131-b11-1~bpo8+1
ENV CA_CERTIFICATES_JAVA_VERSION 20161107~bpo8+1
# Needed for fastlane to work
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
# Required for iTMSTransporter to find Java
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/jre
USER root
# iTMSTransporter needs java installed
# We also have to install make to install xar
# And finally shellcheck
RUN echo 'deb http://archive.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/jessie-
backports.list \
&& apt-get -o Acquire::Check-Valid-Until=false update \
&& apt-get install --yes \
make \
shellcheck \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
USER circleci
COPY --from=xar_builder /tmp/xar /tmp/xar
RUN cd /tmp/xar \
&& sudo make install \
&& sudo rm -rf /tmp/*
CloudBuild.yaml:
- name: 'gcr.io/$PROJECT_ID/fastlane'
args: ['supply', '--package_name','${_ANDROID_PACKAGE_NAME}', '--track', '${_ANDROID_RELEASE_CHANNEL}', '--json_key_data', '${_GOOGLE_PLAY_UPLOAD_KEY_JSON}', '--apk', '/workspace/${_REPO_NAME}/build/app/outputs/bundle/release/app.aab']
timeout: 1200s
Any Idea to solve this?
I solve this by building docker image using docker source from Google Cloud Official other than fastlane on hub.docker.com (where's it never update since 5 month ago)
I run these command:
docker run -ti --rm alpine
apk add --no-cache curl
curl https://registry.npmjs.org/monaco-editor/-/monaco-editor-0.15.6.tgz | tar xvz --strip 1 package/min
and got error tar: package/min: not found in archive.
I run the same command (curl | tar) in Mac terminal and docker ubuntu, all of them are success.
The tar included in Alpine isn't the usual (GNU) tar, but a component of BusyBox:
/scratch # tar --version
tar (busybox) 1.28.4
Apparently, this version of tar generates a (bogus) error message when run with
tar xvz --strip 1 package/min
(however, at first glance, it created the target directory just fine, so ignoring the error message might be ok).
To get rid of the annoying error, you should install GNU tar and use that:
/scratch # apk add --no-cache tar
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
(1/1) Installing tar (1.31-r0)
Executing busybox-1.28.4-r2.trigger
OK: 7 MiB in 19 packages
/scratch # tar --version
tar (GNU tar) 1.31
<rest of message omitted>
Afterwards, running your original command works without any error messages.
I have a project running on Docker with docker-compose for dev environment.
I want to get it running on GitLabCI with a gitlab-ci-multi-runner "Docker mode" instance.
Here is my .gitlab-ci.yml file:
image: soullivaneuh/docker-bash
before_script:
- apk add --update bash curl
- curl --silent --location https://github.com/docker/compose/releases/download/1.5.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
- chmod +x /usr/local/bin/docker-compose
- ./configure
- docker-compose up -d
Note that soullivaneuh/docker-bash image is just a docker image with bash installed.
The script fails on docker-compose up -d command:
gitlab-ci-multi-runner 0.7.2 (998cf5d)
Using Docker executor with image soullivaneuh/docker-bash ...
Pulling docker image soullivaneuh/docker-bash:latest ...
Running on runner-1ee5079f-project-3-concurrent-1 via sd-59984...
Fetching changes...
Removing app/config/parameters.yml
Removing docker-compose.env
HEAD is now at 5c5e7ff remove docker service
From https://git.dummy.net/project/project
5c5e7ff..45e643d docker-ci -> origin/docker-ci
Checking out 45e643dd as docker-ci...
Previous HEAD position was 5c5e7ff... remove docker service
HEAD is now at 45e643d... Remove docker info commands
$ apk add --update bash curl
fetch http://dl-4.alpinelinux.org/alpine/v3.2/main/x86_64/APKINDEX.tar.gz
OK: 10 MiB in 28 packages
$ curl --silent --location https://github.com/docker/compose/releases/download/1.5.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
$ chmod +x /usr/local/bin/docker-compose
$ ./configure
$ docker-compose up -d
bash: line 30: /usr/local/bin/docker-compose: No such file or directory
ERROR: Build failed with: exit code 1
I have absolutly no idea why this is failing.
Thanks for help.
The No such file or directory is misleading. I've received that many times while trying to run dynamically linked binaries using alpine linux (which it appears you are using).
The problem (as I understand it) is that the binary was compiled and linked against glibc, but alpine uses musl, not glibc.
You could use ldd /usr/local/bin/docker-compose to tell you which libraries are missing (or run it with strace if all else fails).
To get it working, it might be easier to install from python source (https://docs.docker.com/compose/install/#install-using-pip), which is what the official compose image does (https://github.com/docker/compose/blob/master/Dockerfile.run).
Or you could use an image built on debian or some other distro that uses glibc.