I'm trying to make CTR (Click through rate) prediction using a python module named 'xlearn'.
It enables me to implement a FFM (field-aware factorisation machine) quite easily.
However, I have a problem with the fit function ( supposed to train the model) which crashes the kernel of my jupyter notebook without any error messages.
Here is the code :
import xlearn as xl
ffm_model = xl.create_ffm()
param = {'task':'binary', 'lr':0.2, 'lambda':0.002, 'metric':'acc'}
ffm_model.setTrain('ffm_train.txt')
ffm_model.fit(param, "./model.out") #this line crashes the kernel
I've already tried to fit the model just after python ffm_model = xl.create_ffm() this also crashes the kernel without any error messages ...
Don't hesitate to share your ideas I'm really stuck here.
I didn't realize the xLearn module was showing error messages in the terminal :
Xlearn Imgae Error Messages
Related
I am implementing a small monitoring system to compute the error between (actual, desired) for a controller (and also recording them for quick and simple analysis). In some of my code, I've added my Systems to my DiagramBuilder and have connected everything using Connect().
I have a controller system that takes in desired input, then produces the actual as output.
Rather than try to remember the inputs connected to the desired output, I'd rather just trace it back.
How do I do that?
From quick perusal, there is both DiagramBuilder.connection_map() and Diagram.connection_map().
The following code seems to work as of v1.11.0:
def trace_to_output(diagram_or_builder, input_port):
system = input_port.get_system()
input_locator = (system, input_port.get_index())
connection_map = diagram_or_builder.connection_map()
output_system, output_index = connection_map[input_locator]
output_port = output_system.get_output_port(output_index)
return output_port
EDIT: I had dict_inverse() in there incorrectly. Fixed.
I am trying to create several .qgs project files to be served at a later time by an instance of qgis Server.
To this end I need to start a new PyQGIS application several times upon request. The application runs smoothly the first time it is called, but if I try to run it a second time I get a Segmentation Fault error.
Here is an example code that triggers the problem:
from qgis.core import QgsApplication
import os
os.environ["QT_QPA_PLATFORM"] = "offscreen"
for i in range(2):
print(f'Iteration number {i}')
print('\tSet prefix path')
QgsApplication.setPrefixPath('/usr', False)
print('\tInstantiating application')
qgs = QgsApplication([], False)
print('\tInitializing application')
qgs.initQgis()
print('\tExiting')
qgs.exitQgis()
When executed, I get this output:
Iteration number 0
Set prefix path
Instantiating application
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
Initializing application
Exiting
Iteration number 1
Set prefix path
Instantiating application
Initializing application
Segmentation fault
Something similar happens if I enclose the content of the loop inside a function and call it multiple times. In this case the segmentation fault happens upon calling qgs.exitQgis() the second time (and any vector or raster layers added before that would be invalid).
Maybe the problem is that for some reason qgs.exitQgis() is not really cleaning up before exit?
The code is running on a Python:3.9 docker container that comes with Debian Bullseye.
Qgis has been installed following the instruction from the QGIS docs:
https://qgis.org/en/site/forusers/alldownloads.html#debian-ubuntu. QGIS version is QGIS 3.22.3-Białowieża 'Białowieża'.
To prevent an import error when loading qgis.core I had to set the environment variable PYTHONPATH = /usr/lib/python3/dist-packages/.
UPDATE: Following a suggestion of one comment found on this post:
https://gis.stackexchange.com/questions/250933/using-exitqgis-in-pyqgis,
I substituted qgs.exitQgis() with qgs.exit() and now the app can be instantiated again any number of times without crashing.
It is still not clear what causes the segmentation fault, but at least I found this workaround.
It seems like the problem was fixed in QGIS ver. 3.24 Tisler. Now qgs.exitQgis() can be called in a loop without triggering a segmentation fault.
I would like to visualize pointcloud in drake-visualizer using python binding.
I imitated how to publish images through lcm from here, and checked out these two issues (14985, 14991). The snippet is as follows :
point_cloud_to_lcm_point_cloud = builder.AddSystem(PointCloudToLcm())
point_cloud_to_lcm_point_cloud.set_name('pointcloud_converter')
builder.Connect(
station.GetOutputPort('camera0_point_cloud'),
point_cloud_to_lcm_point_cloud.get_input_port()
)
point_cloud_lcm_publisher = builder.AddSystem(
LcmPublisherSystem.Make(
channel="DRAKE_POINT_CLOUD_camera0",
lcm_type=lcmt_point_cloud,
lcm=None,
publish_period=0.2,
# use_cpp_serializer=True
)
)
point_cloud_lcm_publisher.set_name('point_cloud_publisher')
builder.Connect(
point_cloud_to_lcm_point_cloud.get_output_port(),
point_cloud_lcm_publisher.get_input_port()
)
However, I got the following runtime error:
RuntimeError: DiagramBuilder::Connect: Mismatched value types while connecting output port lcmt_point_cloud of System pointcloud_converter (type drake::lcmt_point_cloud) to input port lcm_message of System point_cloud_publisher (type drake::pydrake::Object)
When I set 'use_cpp_serializer=True', the error becomes
LcmPublisherSystem.Make(
File "/opt/drake/lib/python3.8/site-packages/pydrake/systems/_lcm_extra.py", line 71, in _make_lcm_publisher
serializer = _Serializer_[lcm_type]()
File "/opt/drake/lib/python3.8/site-packages/pydrake/common/cpp_template.py", line 90, in __getitem__
return self.get_instantiation(param)[0]
File "/opt/drake/lib/python3.8/site-packages/pydrake/common/cpp_template.py", line 159, in get_instantiation
raise RuntimeError("Invalid instantiation: {}".format(
RuntimeError: Invalid instantiation: _Serializer_[lcmt_point_cloud]
I saw the cpp example here, so maybe this issue is specific to python binding.
I also saw this python example, but thought using 'PointCloudToLcm' might be more convenient.
P.S.
I am aware of the development in recent commits on MeshcatVisualizerCpp and MeshcatPointCloudVisualizerCpp, but I am still on the drake-dev stable build 0.35.0-1 and want to stay on drake visualizer until the meshcat c++ is more mature.
The old version in pydrake.systems.meshcat_visualizer.MeshcatVisualizer is a bit too slow on my current use-case (multiple objects drop). I can visualize the pointcloud with this visualization setting, but it took too much machine resources.
Only the message types that are specifically bound in lcm_py_bind_cpp_serializers.cc can be used on an LCM message input/output port connection between C++ and Python. For all other LCM message types, the input/output port connection must be from a Python system to a Python system or a C++ System to a C++ System.
The lcmt_image_array is listed there, but not the lcmt_point_cloud.
If you're stuck using Drake's v0.35.0 capabilities, then I don't see any great solutions. Some options:
(1) Write your own PointCloudToLcm system in Python (by re-working the C++ code into Python, possibly with a narrower set of supported features / channels for simplicity).
(2) Write your own small C++ helper function MakePointCloudPublisherSystem(...) that calls LcmPublisherSystem::Make<lcmt_point_cloud> function in C++, and bind it into Python. Then your Python code can call MakePointCloudPublisherSystem() and successfully connect that to the existing C++ PointCloudToLcm.
I have some existing PyTorch codes with cuda() as below, while net is a MainModel.KitModel object:
net = torch.load(model_path)
net.cuda()
and
im = cv2.imread(image_path)
im = Variable(torch.from_numpy(im).unsqueeze(0).float().cuda())
I want to test the code in a machine without any GPU, so I want to convert the cuda-code into CPU version. I tried to look at some relevant posts regarding the CPU/GPU switch of PyTorch, but they are related to the usage of device and thus doesn't apply to my case.
As pointed out by kHarshit in his comment, you can simply replace .cuda() call with .cpu():
net.cpu()
# ...
im = torch.from_numpy(im).unsqueeze(0).float().cpu()
However, this requires changing the code in multiple places every time you want to move from GPU to CPU and vice versa.
To alleviate this difficulty, pytorch has a more "general" method .to().
You may have a device variable defining where you want pytorch to run, this device can also be the CPU (!).
for instance:
if torch.cuda.is_available():
device = torch.device("cuda")
else:
device = torch.device("cpu")
Once you determined once in your code where you want/can run, simply use .to() to send your model/variables there:
net.to(device)
# ...
im = torch.from_numpy(im).unsqueeze(0).float().to(device)
BTW,
You can use .to() to control the data type (.float()) as well:
im = torch.from_numpy(im).unsqueeze(0).to(device=device, dtype=torch.float)
PS,
Note that the Variable API has been deprecated and is no longer required.
net = torch.load(model_path, map_location=torch.device('cpu'))
Pytorch docs: https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-on-cpu-load-on-gpu
It is a Python Flask app.
The same code works in my local when I run this app on local. But on a server that I rent from DigitalOcean, it gives me this problem.
I have been trying to load this machine learning model(a classification model I trained using sklearn) at run. But it gives me this error OR sometimes, with some solutions I saw in Stackoverflow added, hangs forever.
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0
I tried every solution in Stackoverflow: like adding encoding="latin-1" or "bytes" on loading the pickle. I tried below and many other combinations with different arguments that people recommended in Stackoverflow.
def load_model(file_path):
script_directory = os.path.split(os.path.abspath(__file__))[0]
abs_filepath = os.path.join(script_directory, file_path)
with open(abs_filepath, 'rb') as f:
classifier = pickle.loads(f.read())
return classifier
def load_model(file_path):
script_directory = os.path.split(os.path.abspath(__file__))[0]
abs_filepath = os.path.join(script_directory, file_path)
with open(abs_filepath, 'r') as f:
# also with 'rb'
classifier = pickle.load(f, encoding="bytes")
# also with "latin-1" "latin1" etc.. and load, loads, f, and f.read()
return classifier
model = load_model("modelname.pickle")
What is wrong with this?
It seems as there is a problem when you pickle the model using python2 and try to load this model using python3.
Did you try to load this model using Python 2?
Do you have the same mistake when you run it with parameter encoding=latin1? If it is a different error maybe you need to run
dill._dill._reverse_typemap["ObjectType"] = object
before loading it.
The problem is described pretty well here:
https://rebeccabilbro.github.io/convert-py2-pickles-to-py3/