The background is, I'm responsible for maintaining a fancy Docker image that is used by our team for analytics. It uses a Jupyter notebook image as the base, and then adds various customisations, extra packages, etc.
One of the team members recently wanted to run Tensorflow. No problem, I'll just run mamba install and add it to the image. However, this created an issue: Tensorflow 2.4.3 (the latest version) is somehow incompatible with R 4.1.1 (also the latest version) or something else in the ecosystem, causing R to to be downgraded to 3.6.3. So I created a new environment and installed TF into that:
FROM hongooi/jupytermodelrisk:1.0.0
RUN mamba create -n tensorflow --clone base
# Make RUN commands use the new environment
RUN echo "conda activate tensorflow" >> ~/.bashrc
SHELL ["/bin/bash", "--login", "-c"]
RUN mamba install -y 'tensorflow=2.4.3'
But when I rebuilt the image, I found that while the tensorflow env had been created, the Tensorflow package had been installed into the base env, not the tensorflow env. Has anyone else encountered this? I can verify, if I login to the container, that the tensorflow env has been created: it just doesn't contain the Tensorflow package.
I don't get this problem if run the create, activate and install commands from inside the container. It's only when I try to do it in the Dockerfile.
I use mamba instead of conda because the latter takes forever to run, given the number of packages installed. In fact, trying to run conda install tensorflow crashes after ~5 hours.
Not an expert on dockerfiles, but in general you could just use the -n flag to the install command to specify the target environment for the installation like so:
mamba install -n tensorflow -y tensorflow=2.4.3
Related
Background: I am new to the Python world and am using Plotly for creating basic graphs in Python. I am using AWS Sagemaker's JupyterLab for creating the python scripts.
Issue: I have been trying to run the basic codes mentioned on Plotly's website however even those are returning blank graphs.
Issue Resolution Tried by myself:
pip installed plotly version 4.6.0
Steps mentioned on https://plotly.com/python/getting-started/ for JupyterLab support have already been executed
Code Example:
import plotly.graph_objects as go
fig = go.Figure(data=go.Bar(y=[2, 3, 1]))
fig.show()
I recently had the same issue. Simple change suggested here helped me. I know this is a temporary workaround until a proper fix is found.
// fig = go.Figure()
fig = go.FigureWidget() // replace with this
// fig.show()
fig // remove .show()
Sagemaker notebook instances are using (As of Jan 2022), for some reason, jupyterlab==1.2.21. You can verify that by running pip freeze | grep lab from the terminal or !pip freeze | grep lab from a notebook.
According to the documentation, you'll need to install the following jupyterlab extensions (which are not needed if sagemaker was running jupyterlab 3):
jupyterlab-plotly
jupyter-widgets/jupyterlab-manager
You can install those on a up-and-running instance by running
jupyter labextension install jupyterlab-plotly#5.5.0 #jupyter-widgets/jupyterlab-manager in the terminal or notebook (using ! if you are running on the notebook ofcourse). Notice that the jupyterlab-plotly extension version (here 5.5.0) should match the plotly version you are installing. Mismatches my cause issues. In this case by plotly version is 5.5.0 and thus that's also the jupyterlab-plotly version I've installed.
If you need, like I did, to have it ready upon spinning up a notebook instance, you'll need to:
Create a lifecycle script
To it, add:
PATH=$PATH:/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin - To ensure nodejs path which is needed for the extension installation
pip install plotly==5.5.0 - To ensure a specific version
jupyter labextension install jupyterlab-plotly#5.5.0 #jupyter-widgets/jupyterlab-manager - To ensure same version
of coures, you can change the version according to the most up to date.
I think that documentation is not on par. You now need to install jupyterlab-plotly extension.
jupyter labextension install jupyterlab-plotly
UPDATE
I followed a mix of instructions here and here.
First Enable Extention manager from jupyter-lab
then from terminal
conda install -c conda-forge "nbformat" "ipywidgets>=7.5" -y
jupyter labextension install jupyterlab-plotly
jupyter labextension install #jupyter-widgets/jupyterlab-manager plotlywidget
And within your environment
conda install nbformat
I have a Docker image that is used for running tests in Jenkins and Bamboo. I need to upgrade the version of g++ used (to something with C++11 support).
I tried using a Dockerfile that looks roughly like the following one:
FROM docker.blahblahblah/centos/6.6:latest
RUN yum install -y git gcc-c++ imake centos-release-scl-rh devtoolset-7-toolchain
# I've tried putting this into /etc/bashrc, ~/.bashrc, ~/.bash_profile
RUN echo "source scl_source enable devtoolset-7" >> ~/.bashrc
My issue is that when g++ is used within the container, it uses the older one, instead of the newer one in devtoolset-7, even though the newer one should be sourced from the bashrc. (Maybe I'm misunderstanding how Docker will try to run everything.)
Could anyone point me in the right direction here?
The instructions at https://docs.yugabyte.com/latest/quick-start/docker/install/ state that Docker for Windows is supported, however the yb-docker-ctl utility in the step that follows seems to be a *nix app and does not run on Windows 10 Pro. How do I install a 3-node local YugaByte cluster on Docker for Windows? (by the way StackOverflow would not let me add a YugaByte tag to the question, I could only add Docker)
The yb-docker-ctl utility is actually a Python2 script that will run on Windows 10 Pro if you have Python2 installed. I prefer to use Chocolately (https://chocolatey.org) to manage my package installations, so you could install python2 (not python -- as that will default to python3) using choco install python2 from PowerShell or CMD. You can also install wget in the same manner.
You will then need to a couple of changes to yb-docker-ctl. The script utilizes os.path.join which will utilize the Windows default of \\ for path separator. Add the line import posixpath after line 10 of yb-docker-ctl and substitute posixpath.join for os.path.join at lines 227 and 377.
After you have made those modifications, you can run python yb-docker-ctl create to create your 3 node cluster.
I have a requirement that before an application runs, some part of it needs to read the environmental variable. For this I have the following docker file
FROM nodesource/jessie:0.12.7
# install gettext for envsubst
RUN apt-get update
RUN apt-get install -y gettext-base
# cache package.json and node_modules to speed up builds
ADD package.json package.json
RUN npm install
# Add source files
ADD src src
# Substiture value for backend endpoint env var
RUN envsubst < src/js/envapp.js > src/js/app.js
ADD node_modules node_modules
EXPOSE 8000
CMD ["npm","start"]
The above envsubst line reads (should read) an env variable $MYENV and substitutes it. But when I open the file app.js, its empty.
I checked if the environmental variable exists in the container and it does. Any reason its value is not read and substituted?
I also tried the same command in teh container and it works. It only does not work when I run the image
This is likely because $MYENV is not available for envsubst when you run the image.
Each RUN command runs on its own shell.
From the Docker documentations:
RUN (the command is run in a shell - /bin/sh -c - shell form)
You need to source your profile as well, for example if the $MYENV environment variable is available in the .bashrc file, you can modify your Dockerfile like this:
RUN source ~/.bashrc && envsubst < src/js/envapp.js > src/js/app.js
I encountered the same issues, and after much research and fishing through the internet. I managed to find a few work arounds to this issue. Below I'll list them and identifiable risks at the time of this "Answer post"
Solutions:
1.) apt-get install -y gettext its a standard GNU package language library, one of these libraries that it includes is envsubst` and I can confirm that it works for docker UBUNTU:latest and it should work for every flavored version.
2.) npm install envsub dependent on the "use case" - this approach would be better supported by node based projects.
3.) enstub cli project lib in my opinion it seems a bit overkill to downloading a custom cli from a random stranger but, it's also another option.
Risk:
apt-get install -y gettext:
1.) gettext - this approach would NOT be ideal for VM's as with any package library, it requires maintenance and updates as time passes. However, this isn't necessary for docker because once an a container is initialized and up and running we can create a bashscript to add the package, substitute env vars and then remove the package.
2.) It's a bad idea for VM's because it can be used to execute arbitrary code
npm install ensub
1.) envsub - updating packages and this approach wouldn't be ideal if your dealing with a different stack and not using nodejs.
NOTE:
There's also a PHP version for those developing a PHP application and it seems to work PHP's cli if you need a custom environment.
Resources:
GetText package library info: https://www.gnu.org/software/gettext/
GetText Risk - https://ubuntu.com/security/notices/USN-3815-2
PHP-GetText - apt-get install -y php-gettext
Custom ensubst cli: https://github.com/a8m/envsubst
I suggest that since your are using Node, you use the npm envsub module.
This module is well tested and is developed with docker in mind.
It avoids the need for relying on other dependencies when you already have the full Node arsenal at your fingertips.
envsub is described as
envsub is envsubst for NodeJS
NodeJS global CLI module providing file-level environment variable substitution via Handlebars
I am the author of the package. I think you will enjoy it.
I had some issues with envsubst in Docker.
For some reasons envsubst doesn't work when I try to copy the output in the same file. For example, this is not working:
RUN envsubst < file.conf > file.conf
But when I when I tried to use a temp file the issue disappeared:
RUN envsubst < file.conf > file.conf.temp && cp -f file.conf.temp file.conf
I followed the application to run the tests of pylons project:
http://pylonshq.com/docs/en/0.9.7/i18n/#testing-the-application
But when I run:
nosetests --with-pylons test.ini
It reports an error:
E:\pylons\helloworld>nosetests --with-pylons test.ini
Usage: nosetests-script.py [options]
nosetests-script.py: error: no such option: --with-pylons
Why nosetests doesn't know the --with-pylons, how to fix it?
If you are using Pylons 1.0.1, the nose plugin is not registered by Pylons itself any more.
A workaround is to add this to the entry_points section of your own project's setup.py:
[nose.plugins]
pylons = pylons.test:PylonsPlugin
This error happens in cases where nose cannot find installed pylons.
This can happen if nose is installed system-wide (for example, via apt-get install python-nose), but Pylons is installed in virtual environment. In that case you can either:
Install Pylons system-wide, that would pollute your global environment and defeat the purpose of having virtual environment
Install nose in virtual environment (easy_install -U nose when virtual environment is activated)
If you've installed the latest version of pylons using pip, version 1.0.1rc1 is installed. Nose is not able to find the pylons-plugin.
To fix this downgrade to pylons 1.0.
pip uninstall pylons
pip install pylons==1.0
I had the same problem and found the solution here
I never used --with-pylons. When I am in the directory of the project, nosetests does the job without any parameters.
I'm on Linux, with the proper virtualenv activated. Maybe it's different on Windows.