I have just started working on Azure Data bricks.
I am facing some error while running a already created Python NoteBook.
Here is the code:
import mmlspark
from mmlspark import *
lime = TabularLIME()\
.setModel(randomForestModel)\
.setPredictionCol("predict")\
.setOutputCol("weights")\
.setInputCol("feat")
lime_model = lime.fit(eng_train_pipe)
lime_model.save('dbfs:/mnt/aimodels/rf-eng-lag-lime')
Getting this error:
NameError Traceback (most recent call last)
<command-2244283784192060> in <module>
2 from mmlspark import *
3
----> 4 lime = TabularLIME()\
5 .setModel(rfModel)\
6 .setPredictionCol("prediction")\
NameError: name 'TabularLIME' is not defined
Cluster Runtime is :6.5 (includes Apache Spark 2.4.5, Scala 2.11)
The namespace is incorrect. The correct namespace is mmlspark.lime.TabularLIME.TabularLIME
from mmlspark.lime.TabularLIME import TabularLIME
Related
I'm trying to install a production server for Django Channels application I've been running locally with success. However, when starting Daphne and reaching to the application via browser, I get an Internal Server Error from Daphne. Console output is as follows:
2021-08-07 11:57:09,584 DEBUG HTTP b'GET' request for ['127.0.0.1', 33566]
2021-08-07 11:57:10,071 ERROR Exception inside application: 'module' object is not callable
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/asgiref/compatibility.py", line 34, in new_application
instance = application(scope)
TypeError: 'module' object is not callable
2021-08-07 11:57:10,072 DEBUG HTTP 500 response started for ['127.0.0.1', 33566]
As the project/asgi.py might have some relevance here, it is below:
import os
import django
from django.core.asgi import get_asgi_application
os.environ['DJANGO_SETTINGS_MODULE'] = "proj.settings"
django.setup()
from channels.auth import AuthMiddlewareStack
from channels.routing import ProtocolTypeRouter, URLRouter
import app.routing
application = ProtocolTypeRouter({
"http": get_asgi_application(),
"websocket": AuthMiddlewareStack(
URLRouter(
app.routing.websocket_urlpatterns
)
),
})
However, I've been poking around with the said asgi.py, and I have a feeling that at least the application variable does not have anything to do with this, as that block could be commented out with no impact on the error message.
Relevant packages:
channels 3.0.4
daphne 3.0.2
Django 3.2.6
Any ideas?
I have an ML model (trained locally) in python. Previously the model has been deployed to a Windows IIS server and it's working fine.
Now, I am trying to deploy it as a service on Azure container instance (ACI) with 1 core, and 1 GB of memory. I took references from one and two Microsoft docs. The docs use SDK for all the steps, but I am using the GUI feature from the Azure portal.
After registering the model, I created an entry script and a conda environment YAML file (see below), and uploaded both to "Custom deployment asset" (at Deploy model area).
Unfortunately, after hitting deploy, the Deployment state is stuck at Transitioning state. Even after 4 hours, the state remains the same and there were no Deployment logs too, so I am unable to find what I am doing wrong here.
NOTE: below is just an excerpt of the entry script
import pandas as pd
import pickle
import re, json
import numpy as np
import sklearn
def init():
global model
global classes
model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'randomForest50.pkl')
model = pickle.load(open(model_path, "rb"))
classes = lambda x : ["F", "M"][x]
def run(data):
try:
namesList = json.loads(data)["data"]["names"]
pred = list(map(classes, model.predict(preprocessing(namesList))))
return str(pred[0])
except Exception as e:
error = str(e)
return error
name: gender_prediction
dependencies:
- python
- numpy
- scikit-learn
- pip:
- pandas
- pickle
- re
- json
The issue was in the YAML file. The dependencies/libraries in the YAML should be according to conda environment. So, I changed everything accordingly, and it worked.
Modified YAML file:
name: gender_prediction
dependencies:
- python=3.7
- numpy
- scikit-learn
- pip:
- azureml-defaults
- pandas
- pickle4
- regex
- inference-schema[numpy-support]
I created a docker image with spark 3.0.0 that is to be used for executing pyspark from a jupyter notebook. The issue I'm having though, when running the docker image locally and testing the following script:
import os
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
print("*** START ***")
sparkConf = SparkConf()
sc = SparkContext(conf=sparkConf)
rdd = sc.parallelize(range(100000000))
print(rdd.sum())
print("*** DONE ***")
I get the following error:
Traceback (most recent call last):
File "test.py", line 9, in <module>
sc = SparkContext(conf=sparkConf)
File "/usr/local/lib/python3.7/dist-packages/pyspark/context.py", line 136, in __init__
conf, jsc, profiler_cls)
File "/usr/local/lib/python3.7/dist-packages/pyspark/context.py", line 213, in _do_init
self._encryption_enabled = self._jvm.PythonUtils.getEncryptionEnabled(self._jsc)
File "/usr/local/lib/python3.7/dist-packages/py4j/java_gateway.py", line 1487, in __getattr__
"{0}.{1} does not exist in the JVM".format(self._fqn, name))
py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getEncryptionEnabled does not exist in the JVM
I've tried using findspark and pip installing py4j fresh on the image, but nothing is working and I can't seem to find any answers other than using findspark. Has anyone else been able to solve this issue using spark 3.0.0?
Probably your are mixing different version of Pyspark and Spark
Check my see my complete answer here:
https://stackoverflow.com/a/66927923/14954327
I just cloned VnPy, and I am trying to run VnTrader on a Ubuntu 16.04 Machine, as mentioned in the VnPy Starter Guide. I followed step by step, but when I run
python vnpy/examples/VnTrader/run.py
I get the following Import Error. What is the problem?
Traceback (most recent call last):
File "run.py", line 28, in <module>
from vnpy.trader.gateway import (ctpGateway, ibGateway)
File "/home/alessandro/anaconda2/lib/python2.7/site-packages/vnpy-1.9.0-py2.7.egg/vnpy/trader/gateway/ctpGateway/__init__.py", line 5, in <module>
from .ctpGateway import CtpGateway
File "/home/alessandro/anaconda2/lib/python2.7/site-packages/vnpy-1.9.0-py2.7.egg/vnpy/trader/gateway/ctpGateway/ctpGateway.py", line 16, in <module>
from vnpy.api.ctp import MdApi, TdApi, defineDict
File "/home/alessandro/anaconda2/lib/python2.7/site-packages/vnpy-1.9.0-py2.7.egg/vnpy/api/ctp/__init__.py", line 4, in <module>
from .vnctpmd import MdApi
ImportError: No module named vnctpmd
ImportError: No module named vnctpmd
The vnctpmd module, is the API interface for CTP broker of the VnPy package. As for every other API interface, you need to first build it, and then import it.
In your case, you probably didn't build the CTP interface when prompted during the installation of VnPy, so now run.py can't import the module.
Do you need 'CTP' interface? No
Solution A: I don't need CTP interface
If you don't need CTP interface you can open run.py commenting the parts relative to CTP (and also relative to all the other interfaces you didn't build)
# /examples/VnTrader/run.py script
# this version runs with only the IB interface built
# encoding: UTF-8
# 重载sys模块，设置默认字符串编码方式为utf8
try:
reload # Python 2
except NameError: # Python 3
from importlib import reload
import sys
reload(sys)
try:
sys.setdefaultencoding('utf8')
except AttributeError:
pass
# 判断操作系统
import platform
system = platform.system()
# vn.trader模块
from vnpy.event import EventEngine
from vnpy.trader.vtEngine import MainEngine
from vnpy.trader.uiQt import createQApp
from vnpy.trader.uiMainWindow import MainWindow
# 加载底层接口
from vnpy.trader.gateway import ibGateway
# ### here comment the interfaces you don't need
# from vnpy.trader.gateway import (ctpGateway, ibGateway)
if system == 'Linux':
# from vnpy.trader.gateway import xtpGateway
pass
elif system == 'Windows':
from vnpy.trader.gateway import (femasGateway, xspeedGateway,
secGateway)
# 加载上层应用
from vnpy.trader.app import (riskManager, ctaStrategy,
spreadTrading, algoTrading)
#----------------------------------------------------------------------
def main():
"""主程序入口"""
# 创建Qt应用对象
qApp = createQApp()
# 创建事件引擎
ee = EventEngine()
# 创建主引擎
me = MainEngine(ee)
# 添加交易接口
# me.addGateway(ctpGateway)
me.addGateway(ibGateway)
if system == 'Windows':
me.addGateway(femasGateway)
me.addGateway(xspeedGateway)
me.addGateway(secGateway)
if system == 'Linux':
# me.addGateway(xtpGateway)
pass
# 添加上层应用
me.addApp(riskManager)
me.addApp(ctaStrategy)
me.addApp(spreadTrading)
me.addApp(algoTrading)
# 创建主窗口
mw = MainWindow(me, ee)
mw.showMaximized()
# 在主线程中启动Qt事件循环
sys.exit(qApp.exec_())
if __name__ == '__main__':
main()
Solution B: I need CTP interface
In case you need CTP, you can just reinstall vnpy with the command
bash install.sh
and when prompted 'do you need CTP' answer Yes
I installed MicroPython v1.9.3-8 on my ESP8266 board. Here is the beginning of my main.py file:
from machine import Pin
led = Pin(2, Pin.OUT, value=1)
#---MQTT Sending---
from time import sleep_ms
from ubinascii import hexlify
from machine import unique_id
#import socket
from umqtt import MQTTClient
SERVER = "10.6.6.192"
CLIENT_ID = hexlify(unique_id())
TOPIC1 = b"/server/tem"
TOPIC2 = b"/server/hum"
TOPIC3 = b"/server/led"
The line from umqtt import MQTTClient throws an error when I reset the module:
File "main.py", line 11, in < module >
ImportError: no module named 'umqtt.MQTTClient'
Here is my umqtt.py file.
I have the umqtt.py file uploaded to my esp8266 with webrepl. When I run:
import os
os.listdir()
I get this output:
>>> os.listdir()
['boot.py', 'webrepl_cfg.py', 'umqtt.py', 'main.py']
Since in the umqtt.py file in line 8 the class MQTTClient is defined, I do not know what am I doing wrong to get this code to work.
I think you need to either specify the simple or the robust versions:
from umqtt.simple import MQTTClient