I am doing MLperf, object_detection project test.
https://github.com/mlperf/training/tree/master/object_detection
Question 1:
When doing:
nvidia-docker run -v .:/workspace -t -i --rm --ipc=host mlperf/object_detection \
"cd mlperf/training/object_detection && ./install.sh"
It responses:
docker: Error response from daemon: create .: volume name is too short, names should be at least two alphanumeric characters.
I need to change -v .: to -v $(pwd):/workspace
Question 2:
When applied to the modification above, I got a new error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"cd mlperf/training/object_detection && ./install.sh\": stat cd mlperf/training/object_detection && ./install.sh: no such file or directory": unknown.
It seems docker can't accept a string with space, ex: "cd xxxxxxx && ./install.sh"
If I modified the string to single command (./install.sh)
nvidia-docker run -v $(pwd):/workspace -t -i --rm --ipc=host mlperf/object_detection \
"./install.sh"
This will work, it doesn't look like an incorrect path problem, I tested to use an absoluted path it got the same error.
Question 3:
After followed the steps in the webpage, I always got an error:
ModuleNotFoundError: No module named 'maskrcnn_benchmark'
root#nvme:/markkang/mlperf/training/object_detection# nvidia-docker run -v $(pwd):/workspace -t -i --rm --ipc=host mlperf/object_detection "./run_and_time.sh"
/workspace/pytorch /workspace
Traceback (most recent call last):
File "tools/train_mlperf.py", line 8, in <module>
from maskrcnn_benchmark.utils.env import setup_environment # noqa F401 isort:skip
ModuleNotFoundError: No module named 'maskrcnn_benchmark'
Edit train_mlperf.py and insert the following path code before the invocation of maskrcnn_benchmark.utils.env, e.g.
import sys
sys.path.append('/workspace/pytorch/')
from maskrcnn_benchmark.utils.env import setup_environment # noqa F401 isort:skip
Related
I am trying to install Apex on Docker Desktop. Althouhg I am following the oficial guide on https://container-registry.oracle.com/ords/f?p=113:10:::::: , I get errors regarding the conn_string.txt file.
This was the commands I apply following the guide:
mkdir ords_secrets ords_config
echo 'CONN_STRING=pdbadmin/##mypass###localhost:1521/XEPDB1' > ords_secrets/conn_string.txt
docker run --rm --name ords -v ${PWD}/ords_secrets/:/opt/oracle/variables -p 8181:8181 container-registry.oracle.com/database/ords:latest
And then I get the error:
INFO : This container will start a service running ORDS 22.4.1 and APEX 22.2.0.
ERROR: CONN_STRING_FILE has not added, create a file with CONN_STRING variable and added as docker volume:
mkdir volume ; echo 'CONN_STRING=user/password#hostname:port/service_name' > volume/conn_string.txt
But if i list the directory ords_secrets, the file is there.
Can anyone help? What am I doing wrong?
Thank you
I need to extract the filesystem of a debian image onto the host, modify it, then repackage it back into a docker image. I'm using the following commands:
docker export container_name > archive.tar
tar -xf archive.tar -C debian/
modifying the file system here
tar -cpjf archive-modified.tar debian/
docker import archive-modified.tar debian-modified
docker run -it debian-modified /bin/bash
After I try to run the new docker image I get the following error:
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown.
ERRO[0000] error waiting for container: context canceled
I've tried the above steps without modifying the file system at all and I get the same behavior. I've also tried importing the output of docker export directly, and this works fine. This probably means I'm creating the new tar archive incorrectly. Can anyone tell me what I'm doing wrong?
Take a look at the archive generated by docker export:
# tar tf archive.tar | sort | head
bin/
bin/bash
bin/cat
bin/chgrp
bin/chmod
bin/chown
bin/cp
bin/dash
bin/date
bin/dd
And then at the archive you generate with your tar -cpjf ... command:
# tar tf archive-modified.tar | sort | head
debian/
debian/bin/
debian/bin/bash
debian/bin/cat
debian/bin/chgrp
debian/bin/chmod
debian/bin/chown
debian/bin/cp
debian/bin/dash
debian/bin/date
You've moved everything into a debian/ top-level directory, so there is no /bin/bash in the image (it would be /debian/bin/bash, and probably wouldn't work anyway because your shared libraries aren't in the expected location, either.
You probably want to create the updated archive like this:
# tar -cpjf archive-modified.tar -C debian/ .
In a build process I need to call zipalign which is on a certain path in the docker container that I'm using:
$ docker run nathansamson/flutter-builder-docker:v0.7.3 find . -iname zipalign
./opt/android-sdk-linux/build-tools/28.0.1/zipalign
This path can change, if the docker container is updated and there is a new android sdk. For example this could be the path in the future:
$ docker run nathansamson/flutter-builder-docker:v0.9.9 find . -iname zipalign
./opt/android-sdk-linux/build-tools/42.0.0/zipalign
So instead of hardcoding the call to
docker run nathansamson/flutter-builder-docker:v0.7.3 \
/opt/android-sdk-linux/build-tools/28.0.1/zipalign -h
I would like a generic solution that finds the path to zipalign automatically. I have tried it with a subshell
$ docker run nathansamson/flutter-builder-docker:v0.7.3 $(find . -iname zipalign) -h
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:348: starting container process caused "exec: \"-h\":
executable file not found in $PATH": unknown.
ERRO[0001] error waiting for container: context canceled
and with a wildcard for the folder:
$ docker run nathansamson/flutter-builder-docker:v0.7.3 /opt/android-sdk-linux/build-tools/*/zipalign -h
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:348: starting container process caused "exec:
\"/opt/android-sdk-linux/build-tools/*/zipalign\": stat /opt/android-
sdk-linux/build-tools/*/zipalign: no such file or directory": unknown.
ERRO[0001] error waiting for container: context canceled
So subshells and wildcard don't work in Docker. Any ideas how I can find the path to zipalign whenever I'm calling it?
In your Dockerfile you control the entire environment. It's often easiest to cause things to appear in their "natural" places, like /usr/bin. You also have the advantage that, within a single Docker image, there will only be one version of the tools installed.
I might do something like this:
RUN for f in $PWD/opt/android-sdk-linux/build-tools/*/*; do \
ln -s $f /usr/local/bin; \
done
CMD ["zipalign", "-h"]
Another approach that might work is to use a build argument or an environment variable to hold the version number. If you do that then you can set up a known path name.
ARG version
RUN curl -LO http://.../android-sdk-linux-${version}.tar.gz \
&& tar xzf android-sdk-linux-${version}.tar.gz \
&& rm -f android-sdk-linux-${version}.tar.gz \
&& cd opt/android-sdk-linux-build-tools \
&& ln -s ${version} current
CMD ["./opt/android-sdk-linux/build-tools/current/zipalign"]
find can execute something by using the -exec option.
find . -name zipalign -exec bash -c '"$0"' {} \;
In your example:
docker run nathansamson/flutter-builder-docker:v0.7.3 find . -name zipalign -exec bash -c '"$0"' {} \;
If there is nothing that can give you any hints, and you need to just find it, then find will probably do it. Something like:
find /opt -name zipalign -type f
If you can give it more specific starting point, instead of just /opt, then it will run faster.
If you want to execute it (and pass in "-h") in one line, you could do:
$(find /opt -name zipalign -type f) -h
docker-compose exec is failing when reading from stdin:
$ echo "use mydb; show measurements;" | docker-compose exec influxdb influx
Connected to http://localhost:8086 version 1.2.4
InfluxDB shell version: 1.2.4
> Traceback (most recent call last):
File "/usr/local/bin/docker-compose", line 9, in <module>
load_entry_point('docker-compose==1.9.0', 'console_scripts', 'docker-compose')()
File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 65, in main
command()
File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 117, in perform_command
handler(command, command_options)
File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 462, in exec_command
pty.start()
File "/usr/local/lib/python2.7/dist-packages/dockerpty/pty.py", line 338, in start
io.set_blocking(pump, flag)
File "/usr/local/lib/python2.7/dist-packages/dockerpty/io.py", line 32, in set_blocking
old_flag = fcntl.fcntl(fd, fcntl.F_GETFL)
ValueError: file descriptor cannot be a negative integer (-1)
Using the -T flag gets rid of the problem, but it seems to suppress reading from stdin:
$ echo "use mydb; show measurements;" | docker-compose exec -T influxdb influx
(I need to kill that command, since it shows nothing and is not able to read from stdin)
This seems like a docker-compose bug, maybe I should file this?
$ docker-compose --version
docker-compose version 1.9.0, build 2585387
EDIT
Related:
https://github.com/docker/compose/issues/3352
https://github.com/docker/compose/issues/4290
In https://github.com/docker/compose/issues/3352 endzyme suggests using the docker command for doing this. I created a tiny bash function I have in my bash profile based on his suggestion:
# Like `docker-compose exec` but forwarding stdin to the container
# See https://github.com/docker/compose/issues/3352
function docin() {
local service="$1"; shift
docker exec -i "$(docker-compose ps -q $service)" "$#"
}
In your case you would use it like:
echo "use mydb; show measurements;" | docin influxdb influx
It's a good enough work-around until they fix this in docker-compose.
Trying to install ckan from Dockerfile in:
Docker Community Edition
Version 17.06.2-ce-mac27 (19124)
Channel: stable
428bd6ceae*
FIRST ATTEMPT
These are the steps followed:
$ docker pull ckan/solr
$ docker pull ckan/ckan
$ docker pull ckan/postgresql
After downloaded the images I get:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ckan/solr latest 4acd7db7b1f7 3 days ago 517MB
ckan/ckan latest 77dd30c92740 3 days ago 642MB
ckan/postgresql latest 3c3ecd94ae7e 3 days ago 265MB
For each Dockerfile I made:
For solr
$ docker build . (for SOLR)
Step 7/9
ADD failed: stat /var/lib/docker/tmp/docker-builder234804113/solrconfig.xml: no such file or directory
For postgresql
$ docker build . (for POSTGRESQL)
Successfully built 8b296a6e3153
For ckan
$ docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan
Error in Step 7/26 :
Get:1 http://security.debian.org/ jessie/updates/main perl-base amd64 5.20.2-3+deb8u9 [1226 kB]
debconf: delaying package configuration, since apt-utils is not installed
And Fail in:
Step 12/26 : ADD ./requirements.txt $CKAN_HOME/src/ckan/requirements.txt
ADD failed: stat /var/lib/docker/tmp/docker-builder814560705/requirements.txt: no such file or directory
SECOND ATTEMPT
Removed images
Pull again images
$ docker pull ckan/solr
$ docker pull ckan/ckan
$ docker pull ckan/postgresql
And Launch:
$ docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan
for the Dockerfile in: https://hub.docker.com/r/ckan/ckan/~/dockerfile/
and I get the same error in the same step:
Step 12/26 : ADD ./requirements.txt $CKAN_HOME/src/ckan/requirements.txt
ADD failed: stat /var/lib/docker/tmp/docker-builder254648764/requirements.txt: no such file or directory
but the log in Docker Hub is correct.
Am I doing something wrong?????
Dockerfile
FROM debian:jessie
MAINTAINER Open Knowledge
ENV CKAN_HOME /usr/lib/ckan/default
ENV CKAN_CONFIG /etc/ckan/default
ENV CKAN_STORAGE_PATH /var/lib/ckan
ENV CKAN_SITE_URL http://localhost:5000
# Install required packages
RUN apt-get -q -y update && apt-get -q -y upgrade && DEBIAN_FRONTEND=noninteractive apt-get -q -y install \
python-dev \
python-pip \
python-virtualenv \
libpq-dev \
git-core \
build-essential \
libssl-dev \
libffi-dev \
&& apt-get -q clean
# SetUp Virtual Environment CKAN
RUN mkdir -p $CKAN_HOME $CKAN_CONFIG $CKAN_STORAGE_PATH
RUN virtualenv $CKAN_HOME
RUN ln -s $CKAN_HOME/bin/pip /usr/local/bin/ckan-pip
RUN ln -s $CKAN_HOME/bin/paster /usr/local/bin/ckan-paster
# SetUp Requirements
ADD ./requirements.txt $CKAN_HOME/src/ckan/requirements.txt
RUN ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/requirements.txt
# TMP-BUGFIX https://github.com/ckan/ckan/issues/3388
ADD ./dev-requirements.txt $CKAN_HOME/src/ckan/dev-requirements.txt
RUN ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/dev-requirements.txt
# TMP-BUGFIX https://github.com/ckan/ckan/issues/3594
RUN ckan-pip install --upgrade urllib3
# SetUp CKAN
ADD . $CKAN_HOME/src/ckan/
RUN ckan-pip install -e $CKAN_HOME/src/ckan/
RUN ln -s $CKAN_HOME/src/ckan/ckan/config/who.ini $CKAN_CONFIG/who.ini
# SetUp EntryPoint
COPY ./contrib/docker/ckan-entrypoint.sh /
RUN chmod +x /ckan-entrypoint.sh
ENTRYPOINT ["/ckan-entrypoint.sh"]
# Volumes
VOLUME ["/etc/ckan/default"]
VOLUME ["/var/lib/ckan"]
EXPOSE 5000
CMD ["ckan-paster","serve","/etc/ckan/default/ckan.ini"]
UPDATE 1
As suggested, added the content of the ckan package from:
http://packaging.ckan.org/
Copied to my Dockerfile directory /data/usr/lib/ckan/default/src/ckan/*
New error in step 15 (see comments)
UPDATE 2
Cloned ckan from: https://github.com/ckan/ckan
$ docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan
Result:
Step 26/26 : CMD ckan-paster serve /etc/ckan/default/ckan.ini
---> Running in eeeb6ace6ee1
---> 3cd87cf4a1af
Removing intermediate container eeeb6ace6ee1
Successfully built 3cd87cf4a1af
Successfully tagged ckan:latest
docker: Error response from daemon: could not get container for db: No such container: db.
Trying to access to ckan:
docker run -it ckan bash
I get this error:
Distribution already installed:
ckan 2.8.0a0 from /usr/lib/ckan/default/src/ckan
Creating /etc/ckan/default/ckan.ini
Now you should edit the config files
/etc/ckan/default/ckan.ini
Traceback (most recent call last):
File "/usr/local/bin/ckan-paster", line 11, in <module>
sys.exit(run())
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
result = self.command()
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 348, in command
self._load_config(cmd!='upgrade')
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 321, in _load_config
self.site_user = load_config(self.options.config, load_site_user)
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 230, in load_config
load_environment(conf.global_conf, conf.local_conf)
File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 111, in load_environment
p.load_all()
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 129, in load_all
unload_all()
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 182, in unload_all
unload(*reversed(_PLUGINS))
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 210, in unload
plugins_update()
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 121, in plugins_update
environment.update_config()
File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 289, in update_config
engine = sqlalchemy.engine_from_config(config, client_encoding='utf8')
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 428, in engine_from_config
return create_engine(url, **options)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
return strategy.create(*args, **kwargs)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 50, in create
u = url.make_url(name_or_url)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 194, in make_url
return _parse_rfc1738_args(name_or_url)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 240, in _parse_rfc1738_args
return URL(name, **components)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 60, in __init__
self.port = int(port)
ValueError: invalid literal for int() with base 10: ''
UPDATE 3 (matt fullerton suggestion)
I have cloned https://github.com/parksandwildlife/ckan.git in a folder called ckan
I have cloned https://github.com/parksandwildlife/ckan/tree/3649-docker-upgrade in another folder called ckan-3649
In ckan folder
docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan
In Step 15/26 :
RUN ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/dev-requirements.txt
ERROR:
Cleaning up...
Command /usr/lib/ckan/default/bin/python2 -c "import setuptools, tokenize;__file__='/tmp/pip-build-BLM6DJ/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-Vynm83-record/install-record.txt --single-version-externally-managed --compile --install-headers /usr/lib/ckan/default/include/site/python2.7 failed with error code 1 in /tmp/pip-build-BLM6DJ/cryptography
Storing debug log for failure in /root/.pip/pip.log
The command '/bin/sh -c ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/dev-requirements.txt' returned a non-zero code: 1
In ckan-3649/contrib/docker "docker-compose up"
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b6d7a5a595ca redis:latest "docker-entrypoint..." 9 minutes ago Up 9 minutes 6379/tcp redis
4551ccea28f4 ckan/solr:latest "docker-entrypoint..." 9 minutes ago Up 9 minutes 8983/tcp solr
86cea84b6ab5 docker_db "docker-entrypoint..." 9 minutes ago Up 9 minutes 5432/tcp db
fca64f1bee5a clementmouchet/datapusher "python datapusher..." 9 minutes ago Up 9 minutes 0.0.0.0:8800->8800/tcp datapusher
But I think ckan image is missing....
There is a major upgrade of the docker functionality in CKAN underway. I would suggest cloning CKAN from here:
https://github.com/parksandwildlife/ckan
git clone https://github.com/parksandwildlife/ckan.git
Then checkout the origin/3649-docker-upgrade branch
(https://github.com/parksandwildlife/ckan/tree/3649-docker-upgrade)
git checkout origin/3649-docker-upgrade
Then use docker-compose in the contrib/docker folder:
docker-compose up
This should assemble Solr, Postgres etc. for you.
Comments on mileage with this at https://github.com/ckan/ckan/pull/3692 will also be appreciated.