Docker compose jupyter pyspark notebook NameError: name 'TerminalsExtensionApp' is not defined - docker

version: "3.7"
services:
pyspark:
image: jupyter/pyspark-notebook:latest
environment:
JUPYTER_ENABLE_LAB: "yes"
ports:
- "8888:8888"
volumes:
[C:\spark\shared_data:/home/jovyan/work/data]
requirements:
image: jupyter/pyspark-notebook:latest
volumes:
[C:\spark\shared_data:/home/jovyan/work/data]
command: pip install JayDeBeApi
I run the docker-compose yaml file of jupyter pyspark notebook & it has other Python modules also installed via requirements.txt. While composing up the docker I get UserWarning: Could not import submodules warnings.warn("Could not import submodules") and NameError: name 'TerminalsExtensionApp' is not defined.

Related

How to authenticate to Azurite using pyspark?

I am building an application using two docker containers in the same network:
mcr.microsoft.com/azure-storage/azurite
jupyter/pyspark-notebook
Here is my docker-compose file:
version: "3.9"
services:
azurite:
image: mcr.microsoft.com/azure-storage/azurite:latest
ports:
- "10000:10000"
- "10001:10001"
- "10002:10002"
volumes:
- azurite_volume:/data
pyspark:
image: jupyter/pyspark-notebook:latest
ports:
- 10003:8888
user: root
working_dir: /home/${NB_USER}
environment:
- NB_USER=${NB_USER}
- CHOWN_HOME=yes
- GRANT_SUDO=yes
command: start-notebook.sh --NotebookApp.password="" --NotebookApp.token=""
volumes:
- /my/local/folder:/home/${NB_USER}/work
volumes:
azurite_volume:
driver: local
from the jupyter notebook I am trying to connect to and read data from azurite. Here is my code:
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName('test') \
.config(
'fs.azure.account.key.devstoreaccount1.blob.core.windows.net',
'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==') \
.getOrCreate()
df = spark.read.json('wasb://my-container#devstoreaccount1/path/to/file.json')
However, this code returns an error:
org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Unable to access container bronze in account devstoreaccount1 using anonymous credentials, and no credentials found for them in the configuration.
The container in azurite has already been set to "public" although it wouldn`t be necessary because I am providing the credential in the spark config. Even though, the error tells me that I am using anonymous credentials...
I am probably setting the credentials wrongly but I couldn't find anywhere how to set them properly.
How can I set up the credentials to be able to read from azurite using pyspark?

Run multiple commands services in yml file when docker-compose up

I want to run multiple command when I docker-compose up this service, but always failed
Win 10, Powershell, docker: 2.4.0.0, Engine: 19.03.13, Compose:1.27.4
version: '3.7'
services:
cypress:
container_name: tax_t_cypress
image: cypress/included:5.4.0
command: bash -c "npm i cypress-file-upload#5.0.3" && cypress run
volumes:
- ./:/tax
working_dir: /tax
environment:
- CYPRESS_BASE_URL=http://nginx:8888
depends_on:
- webpack
It will error
You passed: npm i cypress-file-upload#5.0.3
The error was: Cannot read property 'split' of undefined
If I use single command like command: cypress run, It's okay, But I really need install the dependency first in this image, How can I run multiple command in yml successfully.
if you need to install something on the image, you should do it on the image level, by writing a dockerfile and referencing it in your docker compose.
your dockerfile could be
FROM cypress/included:5.4.0
RUN npm i cypress-file-upload#5.0.3
CMD ["cypress" , "run"]
in your docker-compose.yml
version: '3.7'
services:
cypress:
container_name: tax_t_cypress
build:
context: .
dockerfile: dockerfile
volumes:
- ./:/tax
working_dir: /tax
environment:
- CYPRESS_BASE_URL=http://nginx:8888
depends_on:
- webpack

How to upload extra modules to odoo container in digital ocean

I have a docker-compose.yml with the configuration for Odoo container, and I have some custom modules.
version: '2'
services:
web:
image: odoo:11.0
restart: always
depends_on:
- db
ports:
- "8069:8069"
volumes:
- ./custom-modules:/mnt/extra-addons
db:
image: postgres:10
environment:
- POSTGRES_DB=postgres
- POSTGRES_PASSWORD=odoo
- POSTGRES_USER=odoo
And I want to deploy this containers to digital ocean, so I create a docker droplet
$ docker-machine create --driver=digitalocean --digitalocean-access-token=$DO_TOKEN --digitalocean-size=s-1vcpu-1gb odoo
$ eval $(docker-machine env odoo)
$ docker-compose up -d
And i was expecting something like docker uploads my custom-modules/ or something like that but the folder is not available in the docker machine. Any idea on how to do this? Of course I know how to install odoo from scratch in a normal ubuntu droptlet but I want to do this with Docker, but I am new with this technology
Have you added the relative paths of your extra module in the addons_path of your odoo.conf configuration file ?
Your .yml file looks correct.
Don't forget to update the list of applications in the 'applications' app, and install/update your custo module...

Any rake command is waiting few minutes before running within Docker For Mac container

Run simple docker architecture with 2 containers db and rails application. Any rake command related to db is very slow. Like rake db:create, rake db:migrate
Tried to test speed between 2 containers by iperf. It shows 26-27 Gbits/sec. So it looks like not network problem. And it is working like charm in any linux host.
Docker For Mac specs
MacOS Mojave 10.14.3;
Engine: 18.09.1;
Compose: 1.23.2;
Machine 0.16.1;
Here is sample docker-compose.yml
version: '3.7'
services:
postgres_10_5:
image: postgres:10.5
ports:
- "5432"
networks:
- backend
web_app:
build:
context: .
dockerfile: Dockerfile-dev
env_file:
- ./.env
ports:
- "3000:3000"
- "1080:1080"
environment:
- RAILS_ENV=development
volumes:
- .:/home/app
networks:
- backend
networks:
backend:
driver: bridge
Expect not wait for result of any rake command around 5 minutes. Don't know where to dig down. Any hints?
I had this exact same issue too. It's to do with the very poor performance of Docker on OSX, and how you've setup your volumes/mounts in docker.
I found this article that has a good overview of how to setup a Dockerfile and docker-compose.yml for Rails, and have it actually perform OK.
The main thing to understand:
To make Docker fast enough on MacOS follow these two rules: use :cached to mount source files and use volumes for generated content (assets, bundle, etc.).
You haven't setup your volumes properly for ruby gems, postgresql data (and possibly other things).
Key statements you need in your Dockerfile:
...
# Configure bundler and PATH
ENV LANG=C.UTF-8 \
GEM_HOME=/bundle \
BUNDLE_JOBS=4 \
BUNDLE_RETRY=3
ENV BUNDLE_PATH $GEM_HOME
ENV BUNDLE_APP_CONFIG=$BUNDLE_PATH \
BUNDLE_BIN=$BUNDLE_PATH/bin
ENV PATH /app/bin:$BUNDLE_BIN:$PATH
# Upgrade RubyGems and install required Bundler version
RUN gem update --system && \
gem install bundler:$BUNDLER_VERSION
# Create a directory for the app code
RUN mkdir -p /app
...
And in your docker-compose.yml
version: '3.7'
postgres_10_5:
image: postgres:10.5
volumes:
- postgresql:/var/lib/postgresql/data
ports:
- "5432"
web_app:
build:
context: .
dockerfile: Dockerfile-dev
env_file:
- ./.env
stdin_open: true
tty: true
volumes:
- .:/app:cached
- rails_cache:/app/tmp/cache
- bundle:/bundle
environment:
- RAILS_ENV=${RAILS_ENV:-development}
depends_on:
- postgres_10_5
volumes:
postgres:
bundle:
rails_cache:
See the article for a more in-depth discussion on how it all works.

How to setup PyCharm for Docker inside Vagrant?

I've a project that's running Docker in Vagrant.
The python interpreter is inside the Docker containter.
How do I set up PyCharm so that it can use this interpreter.
The Dockerfile is:
FROM python:3.5.1-onbuild
The docker-compose.yaml is the following:
web:
restart: always
build: .
ports:
- "80:80"
expose:
- "80"
links:
- postgres:postgres
volumes:
- .:/usr/src/app/
env_file: .env
command: /usr/local/bin/gunicorn --reload -w 2 -b :80 hello:app
data:
image: postgres:latest
volumes:
- /var/lib/postgresql
command: "true"
postgres:
restart: always
image: postgres:latest
volumes_from:
- data
ports:
- "5432:5432"
I already tried with the standard options but it seems that you only can choose either docker or vagrant:
docker or vagrant
Thanks
Docker-compose support is added to the last version of PyCharm. Here's how to solve the issue, using the latest version of PyCharm.
Configure PyCharm interpreter with docker-compose inside Vagrant

Resources