I am using google cloud platform datalab and I have the following "install_sw" notebook.
%%bash
apt-get update
apt-get -y install --allow-unauthenticated libsndfile1 librosa
and
!pip install librosa soundfile google-cloud-storage
Everytime I restart work on my notebooks, I have to run the install software notebook before I work on my regular notebook.
I tried to ssh to the compute instance and then connect to the docker container using the docker exec command. Once I am in the docker container I tried to apt-get install and pip install and the will install, but they do not change the notebook environment.
I do not how to configure dockers. Do I need to do something at the docker level to get this to work?
See https://cloud.google.com/datalab/docs/how-to/adding-libraries#adding_python_libraries_to_a_cloud_datalab_instance.
You can follow option 3 to create your own custom image that extends the Datalab image.
Related
I have the following scenario:
A docker or podman container is setup ready and deployed to several production instances, that are NOT connected to the internet.
A new release has been developed, that needs only a new package, like a python module of a few kilobytes in size.
The new package is installed on dev container, and the dockerfile has been updated to also load the latest module (just for documentation, because the target system cannot reach docker.io).
We have packed the new container release, which is more than a Gigabyte in size. And could transport the new container to the target environments.
My question is: is there a way, to pack, create and transport only a delta of the container compared to the previously deployed version?
podman version 3.4.7
echo "\
FROM jupyter/scipy-notebook
USER root
RUN apt-get update && apt-get install --no-install-recommends -y mupdf-tools python3-dev
USER user
RUN pip -V
RUN pip install fitz==0.0.1.dev2
RUN pip install PyMuPDF==1.20.2
RUN pip install seaborn
RUN pip install openpyxl==3.0.10
RUN pip install flask==2.1.3
" > sciPyDockerfile
podman build --tag python_runner -f ./sciPyDockerfile
sudo podman save -o python_runner.tar python_runner
gzip python_runner.tar
The result is a file
1.1G Nov 28 15:27 python_runner.tar.gz
Is there any way to pack the delta only?
I am preparing a test automation which require me to install network manager so that the code api(which uses python3-networkmanager) could be tested.
In the docker file, I tried installing:
apt-get install dbus \
network-manager
start receiving error:
networkmanager.systems do not have hostname property.
I looked for solutions, but appears that will require:
Privilege user (cannot use privilege user, project requirement)
Reboot after installing same. (in docker, hence, can't reboot)
This leaves me with an only option for mocking debian networkmanager that can communicate with python3-networkmanager.
Trying to figure out, how I can mock same?
RUN apt-get update && apt-get install -y \
network-manager
worked for me.
I would like to contribute as I had to spend some time getting it to work.
Inside the dockerfile you have to add:
RUN apt-get update && apt-get install -y network-manager dbus
Also, I added a script to start the network manager:
#!/bin/bash
service dbus start
service NetworkManager start
Then in the Dockerfile you have to call this start script at the end:
COPY start_script.sh /etc/init/
ENTRYPOINT ["/etc/init/start_script.sh"]
Now you can build your container and run it like this:
docker run --net="host" -v /run/dbus/system_bus_socket:/run/dbus/system_bus_socket container
For me, it is enough to work with OrangePi and Docker without a privileged container.
I'm following this SageMaker guide and using the 1.12 cpu docker file.
https://github.com/aws/sagemaker-tensorflow-serving-container
If I use the requirements.txt file to install Pillow, my container works great locally, but when I deploy to SageMaker, 'pip3 install' fails with an error indicating my container doesn't have internet access.
To work around that issue, I'm trying to install Pillow in my container before deploying to SageMaker.
When I include the lines "RUN pip3 install Pillow" and "RUN pip3 show Pillow" in my docker file, when building, I see output saying "Successfully installed Pillow-6.2.0" and the show command indicates the lib was installed at /usr/local/lib/python3.5/dist-packages. Also running "RUN ls /usr/local/lib/python3.5/dist-packages" in the docker files shows "PIL" and "Pillow-6.2.0.dist-info" in dist-packages, and the PIL directory includes many code files.
However, when I run my container locally, trying to import in python using "from PIL import Image" results in error "No module named PIL". I've tried variations like "import Image", but PIL doesn't seem to be installed in the context in which the code is running when I start the container.
Before the line "from PIL import Image", I added "import subprocess" and 'print(subprocess.check_output("ls /usr/local/lib/python3.5/dist-packages".split()))'
This ls output matches what I get when running it in the docker file, except "PIL" and "Pillow-6.2.0.dist-info" are missing. Why are those two in /usr/local/lib/python3.5/dist-packages when I run the docker file but not when my container is started locally?
Is there a better way to include Pillow in my container? The referenced Github page also shows that I can deploy libraries by including the files (in code/lib of model package), but to get files compatible with Ubuntu 16.04 (which the docker container uses; I'm on a Mac), I'd probably copy them from the docker container after running "RUN pip3 install Pillow" in my docker file, and it seems odd that I would need to get files from the docker container to deploy to the docker container.
My docker file looks like:
ARG TFS_VERSION
FROM tensorflow/serving:${TFS_VERSION} as tfs
FROM ubuntu:16.04
LABEL com.amazonaws.sagemaker.capabilities.accept-bind-to-port=true
COPY --from=tfs /usr/bin/tensorflow_model_server /usr/bin/tensorflow_model_server
# nginx + njs
RUN \
apt-get update && \
apt-get -y install --no-install-recommends curl && \
curl -s http://nginx.org/keys/nginx_signing.key | apt-key add - && \
echo 'deb http://nginx.org/packages/ubuntu/ xenial nginx' >> /etc/apt/sources.list && \
apt-get update && \
apt-get -y install --no-install-recommends nginx nginx-module-njs python3 python3-pip python3-setuptools && \
apt-get clean
RUN pip3 install Pillow
# cython, falcon, gunicorn, tensorflow-serving
RUN \
pip3 install --no-cache-dir cython falcon gunicorn gevent requests grpcio protobuf tensorflow && \
pip3 install --no-dependencies --no-cache-dir tensorflow-serving-api
COPY ./ /
ARG TFS_SHORT_VERSION
ENV SAGEMAKER_TFS_VERSION "${TFS_SHORT_VERSION}"
ENV PATH "$PATH:/sagemaker"
RUN pip3 show Pillow
RUN ls /usr/local/lib/python3.5/dist-packages
I've tried installing Pillow on the same line as cython and other dependencies, but the result is the same...those dependencies are in /usr/local/lib/python3.5/dist-packages both at the time the container is built and when the container is started locally, while "PIL" and "Pillow-6.2.0.dist-info" are only present when the container is built.
Apologies for the late response.
If I use the requirements.txt file to install Pillow, my container works great locally, but when I deploy to SageMaker, 'pip3 install' fails with an error indicating my container doesn't have internet access.
If restricted internet access isn't a requirement, then you should be able to enable internet access by making enable_network_isolation=False when instantiating your Model class in the SageMaker Python SDK, as shown here: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/model.py#L85
If restricted internet access is a requirement, this means that you will need to either install your dependencies in your own container beforehand or make use of the packaging as you mentioned in your correspondence.
I have copied your provided Dockerfile and created an image to run as an image in order to reproduce the error you are seeing. I was not able to reproduce the error as quoted below:
However, when I run my container locally, trying to import in python using "from PIL import Image" results in error "No module named PIL". I've tried variations like "import Image", but PIL doesn't seem to be installed in the context in which the code is running when I start the container.
I created a similar Docker image and ran it as a container with the following command:
docker run -it --entrypoint bash <DOCKER_IMAGE>
from within the container I started a Python3 session and ran the following commands locally without error:
root#13eab4c6e8ab:/# python3 -s
Python 3.5.2 (default, Oct 8 2019, 13:06:37)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image
Can you please provide the code for how you're starting your SageMaker jobs?
Please double check that the Docker image you have created is the one being referenced when starting your SageMaker jobs.
Please let me know if there is anything I can clarify.
Thanks!
I'm trying to install some libraries (specifically pytorch) in my docker image. I have a Dockerfile that installs anaconda properly, but now I'd like to use conda to install a few other things in the image. Is there a way to do this? I've tried
RUN /opt/anaconda/bin/conda install -y pytorch torchvision
And anaconda is installed in the right location. Am I doing this the right way?
First, check if the way you have added/installed anaconda in your own image reflects the ContinuumIO/docker-images/anaconda.
Second, you can test the installation dynamically, as the README recommends
docker run -it yourImage /bin/bash -c "/opt/conda/bin/conda install ...
If that is working correctly, you can do the same in your Dockerfile.
I have jenkins build and I am trying to invoke a ansible playbook file for an s3 upload. When I execute a post-build-script for invoking an ansible playbook file, I am ending with below error.
Cannot run program "ansible-playbook" (in directory "/var/jenkins_home/workspace/mybuild"): error=2, No such file or directory
Below screenshot is ansible post build script configuration.
FYI: There is a file(ansibledemo.yml) in my build folder. I tried giving absolute path(/var/jenkins_home/workspace/mybuild/ansibledemo.yml). Still no go.
When I try running ansible-playbook myplaybook.yml directly in jenkins image(terminal) I am ending up with bash: ansible-playbook: command not found
When I tried installing ansible in my jenkins server, I couldn't execute any installation commands. Please see the below screenshot.
Ansible is not install on your Jenkins machine, first you need to install the ansible on the jenkins machine:
On Ubuntu/Debian:
sudo apt-add-repository ppa:ansible/ansible
sudo apt-get update
sudo apt-get install ansible
On CentOS/RedHat:
sudo yum install epel-release
sudo yum install ansible
After that you will be able to run the ansible-playbook.
You can try to install using pip version as an alternative and try, Please see the below steps,
$ virtualenv venv
$ source venv/bin/activate
$ pip install ansible-container[docker,openshift]
You can see more options to install in docs: https://docs.ansible.com/ansible-container/installation.html
But always it is a good option to keep a separate vm / docker like "ansible-controller" and use that as a slave to jenkins, So that you don't need ansible plugins in ansible. And jenkins will be always stable without much load
Download package information from the configured sources.
# apt update
Install ansible
# apt install ansible
That's it.
If you run official jenkins container (based on debian) than repo with ansible build in already and you don't need "apt-add-repository". But you could install apt-add-repository by installing software-properties-common for further using.
dpkg -S apt-add-repository tells that this packet belongs to software-properties-common.
Error appears because the author of container always tries to make it as light as possible and remove package information.
You don't need sudo, because you become root in container by default. You become another user only if you mention it in intentionally.
Please, add information that you work in container to your question.