In dsx is there a way to use "display" in a scala 2.11 with Spark 2.0 notebook (I know it can be done in a python notebook with pixiedust). Eg:
display(spark.sql("SELECT COUNT(zip), SUM(pop), city FROM hive_zips_table
WHERE state = 'CA' GROUP BY city ORDER BY SUM(pop) DESC"))
But I want to do the same in a scala notebook. Currently I am just doing a show command below that just give data in a tabular format with no graphics etc.
spark.sql("SELECT COUNT(zip), SUM(pop), city FROM hive_zips_table
WHERE state = 'CA' GROUP BY city ORDER BY SUM(pop) DESC").show()
Note:
Pixiedust currently works with Spark 1.6 and Python 2.7.
Pixiedust currently supports Spark DataFrames, Spark GraphFrames and Pandas
Reference:-
https://github.com/ibm-cds-labs/pixiedust/wiki
But if you can use Spark 1.6 ,here is a quick way around to use that fancy display function:-
You can go the other way around, Since Pixidust let you use scala and python in one python notebook with %%scala line magic.
https://github.com/ibm-cds-labs/pixiedust/wiki/Using-Scala-language-within-a-Python-Notebook
Step 1. Create a notebook with python 2 and spark 1.6
Install pixidust and import it
!pip install --user --no-deps --upgrade pixiedust
import pixiedust
Define your variables or your dataframe in Scala under
%%scala
import org.apache.spark.sql._
print(sc.version)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val __df = sqlContext.read.json("people.json")
__df.show()
or
do whatever to create your dataframe
val __df = dataframe1.sql("SELECT COUNT(zip), SUM(pop), city FROM hive_zips_table
WHERE state = 'CA' GROUP BY city ORDER BY SUM(pop) DESC").show()
Step 2: In separate cell run following to access df variable in your python shell.
display(__df)
Reference to my sample Notebook:-
IBM Notebook: https://apsportal.ibm.com/analytics/notebooks/095520cb-c9ff-4f4a-a829-f458f20b4505/view?access_token=d4de7944ad7d6bfc179632a3036a7971c130e54d1a30ecf5df34ece8c4f8c3b5
Github: https://github.com/charles2588/bluemixsparknotebooks/blob/master/pixiedust/PixiedustTestCase.ipynb
Thanks,
Charles.
You can get similar result in Zeppelin
z.show(dataframe)
Related
Expected Behavior (local environment: fresh MacOS 12.4 installation)
With no environment updates except $ pip3 install matplotlib, I can successfully run this simple plot from the Matplotlib documentation:
Example Code:
# testplot.py
import matplotlib.pyplot as plt
import numpy as np
# Data for plotting
t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2 * np.pi * t)
fig, ax = plt.subplots()
ax.plot(t, s)
ax.set(xlabel='time (s)', ylabel='voltage (mV)',
title='About as simple as it gets, folks')
ax.grid()
fig.savefig("test.png")
plt.show()
Actual Output (saved to a .png after window opens):
Run $ python3 testplot.py in the terminal:
Observed Behavior (vscode python 3.8 dev container)
Disclaimer: This post does not address notebook-based plots (which work fine but are not always preferred)
However, when I run this in my dev container, I get the following error:
testplot.py:16: UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure.
plt.show()
First Attempted Solution:
Following this previously posted solution, I specified the backend (export MPLBACKEND=TKAgg) before running the interpreter, but the error persists.
Second Attempted Solution:
Following the comments, I added the following lines to the script:
import matplotlib
matplotlib.use('tkagg')
In the v3.8 dev container, this addition changes the error to:
Traceback (most recent call last):
File "testplot.py", line 5, in <module>
matplotlib.use('tkagg')
File "/usr/local/python/lib/python3.8/site-packages/matplotlib/__init__.py", line 1144, in use
plt.switch_backend(name)
File "/usr/local/python/lib/python3.8/site-packages/matplotlib/pyplot.py", line 296, in switch_backend
raise ImportError(
ImportError: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, as 'headless' is currently running
Note: adding these two lines broke the local script as well. The point of the local example was to show that it plots stuff without installing anything except matplotlib.
I used Drake Binary Installation for Python from this site
https://drake.mit.edu/python_bindings.html#installation
Then when i checked to ensure I can import pydrake, I have got this:
dmitriy#dmitriy-Lenovo-ideapad-310-15ISK:/opt$ python3 -c 'import pydrake.all; print(pydrake.__file__)'
/opt/drake/lib/python3.8/site-packages/pydrake/__init__.py
But when I try execute this example:
import pydrake.all
builder = pydrake.systems.framework.DiagramBuilder()
plant, _ = pydrake.multibody.plant.AddMultibodyPlantSceneGraph(builder, 0.0)
pydrake.multibody.parsing.Parser(plant).AddModelFromFile(
pydrake.common.FindResourceOrThrow(
"drake/examples/pendulum/Pendulum.urdf"))
plant.Finalize()
diagram = builder.Build()
simulator = pydrake.systems.analysis.Simulator(diagram)
I get this:
No module named 'pydrake'
The documentation you cite shows how to export PYTHONPATH (or alternatively, how to us a virtualenv). Since the first command you posted works, you must have done those steps correctly. Thus, the second command should have also worked.
Maybe you ran the second command in a new terminal, without the correct environment set up?
I tried to run the snippet of code from the tutorial about NER from deeppavlov.
Link tutorial
First I run command:
python -m deeppavlov install ner_ontonotes_bert_mult
python -m deeppavlov interact ner_ontonotes_bert_mult [-d]
Then
from deeppavlov import configs, build_model
ner_model = build_model(configs.ner.ner_ontonotes_bert_mult, download=True)
ner_model(['World Curling Championship will be held in Antananarivo'])
Catch this error:
ImportError: cannot import name 'convert_examples_to_features'
So if paste code in VSCode, line
configs.ner.ner_ontonotes_bert_mult
will give a signal that "Instance of 'Struct' has no 'ner' member"
(Config is Struct)
How I can fix it?
Google didn 't find an answer
This question already has answers here:
anaconda - graphviz - can't import after installation
(14 answers)
Closed 3 years ago.
I cannot get graphviz to install and import properly, either using conda install or pip install. I am using jupyter on anaconda.
conda install -c anaconda graphviz
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-2-0e78fd6401bd> in <module>()
----> 1 import graphviz
2 dot_data = tree.export_graphviz(clf, out_file=None)
3 graph = graphviz.Source(dot_data)
4 graph.render("iris")
ModuleNotFoundError: No module named 'graphviz'
When I install using pip I can successfully import but when I run the below code, I get a very long error message that ends with the following when running graph.render:
from sklearn.datasets import load_iris
from sklearn import tree
import graphviz
iris = load_iris()
clf = tree.DecisionTreeClassifier()
clf = clf.fit(iris.data, iris.target)
dot_data = tree.export_graphviz(clf, out_file=None)
graph = graphviz.Source(dot_data)
graph.render("iris")
Below is the end of the end of a long error message:
ExecutableNotFound: failed to execute ['dot', '-Tpdf', '-O', 'iris'], make sure the Graphviz executables are on your systems' PATH
I have the directory where the graphviz folders are in PATH.
Anyone have an idea what I am doing wrong?
http://www.graphviz.org/Download_windows.php
Install windows package from: http://www.graphviz.org/Download_windows.php
Install python graphviz package
Add C:\Program Files (x86)\Graphviz2.38\bin to User path
Add C:\Program Files (x86)\Graphviz2.38\bin\dot.exe to System Path
RESTART COMPUTER
Worked for me!
If you're running Linux, you may have to run this command in addition to the Conda command (I had to do this):
sudo apt-get install graphviz
I was getting the exact same error in Pycharm and this solved it.
I have a single Python script called myscript.py and would like to package it up as a nix derivation with mkDerivation.
The only requirement is that my Python script has a run-time dependency, say, for the consul Python library (which itself depends on the requests and six Python libraries).
For example for myscript.py:
#!/usr/bin/env python3
import consul
print('hi')
How to do that?
I can't figure out how to pass mkDerivation a single script (its src seems to always want a directory, or fetchgit or similar), and also can't figure out how to make the dependency libraries available at runtime.
When you have a single Python file as your script, you don't need src in your mkDerivation and you also don't need to unpack any source code.
The default mkDerivation will try to unpack your source code; to prevent that, simply set dontUnpack = true.
myscript-package = pkgs.stdenv.mkDerivation {
name = "myscript";
propagatedBuildInputs = [
(pkgs.python36.withPackages (pythonPackages: with pythonPackages; [
consul
six
requests2
]))
];
dontUnpack = true;
installPhase = "install -Dm755 ${./myscript.py} $out/bin/myscript";
};
If your script is executable (which we ensure with install -m above) Nix will automatically replace your #!/usr/bin/env python3 line with one which invokes the right specific python interpreter (the one for python36 in the example above), and which does so in an environment that has the Python packages you've specifified in propagatedBuildInputs available.
If you use NixOS, you can then also put your package into environment.systemPackages, and myscript will be available in shells on that NixOS.
This helper function is really nice:
pkgs.writers.writePython3Bin "github-owner-repos" { libraries = [ pkgs.python3Packages.PyGithub ]; } ''
import os
import sys
from github import Github
if __name__ == '__main__':
gh = Github(os.environ['GITHUB_TOKEN'])
for repo in gh.get_user(login=sys.argv[1]).get_repos():
print(repo.ssh_url)
''
https://github.com/nixos/nixpkgs/blob/master/pkgs/build-support/writers/default.nix#L319