i am using pioneer 3dx with rosaria library where sonar publishes obstacles position from where i can extract ranges from obstacles. i mounted intel realsense D435i on pioneer and installed all the libraries required for camera. now the question is what data should i extract from camera for fusing both data so that they complement each other.
i used yolov3 algorithm on realsense camera and try to detect objects ,bounding box and x min,y min,x max, y max and depth info from objects detected.
so which info should i use from vast topics that camera is giving to fuse it with sonar range data so to improve obstacle detection fo avoidance
i am attaching sonar range data and camera publishes topics below
sonar ranges from 8 sonar rings:
pointcloud of sonar:
camera published topics:
Related
have IMU sensor that gives me the raw data such as orientation, Angular and Linear acceleration. Im using ROS and doing some Gazebo UUV simulation. Furthermore, I want to get linear velocity from the raw IMU data.
So the naive integration method is as follow: The simplest method for IMU state estimation is the naive integration of the measured data. We estimate the attitude by integrating 3D angular velocity data obtained by the IMU. Assuming that the time step Δ𝑡 is small, the attitude at each time step can be incrementally calculated.
If I do integration over time there will be accumulated error and will not be accurate with the time when for example the robot makes turns. So Im looking for some methods ( ROS packages or outside ROS framework) or code that can correct that error.
Any help?
I would first recommend that you try fitting your input sensor data into an EKF or UKF node from the robot_localization package. This package is the most used & most optimized pose estimation package in the ROS ecosystem.
It can handle 3D sensor input, but you would have to configure the parameters (there are no real defaults, all config). Besides the configuration docs above, the github has good examples of yaml parameter configurations (Ex.) (you'll want a separate file from the launch file) and example launch files (Ex.).
I am going to set up my RGBD camera with Panasonic LUMIX GH5 and azure Kinect like a depthkit cinema.
The depthkit does not provide raw depth data but the obj sequence files. I require the depth buffer which aligns with an RGB image.
So I started to write the software for it. (I have fixed my Panasonic LUMIX GH5 camera and azure Kinect with SmallRig fixture.)
After getting the extrinsic param of the GH5 and Azure Kinect RGB sensor with OpenCV's solvePnP function, how can I use them to aligning the GH5 colour image with the Azure Kinect Depth image?
Or should I do another approach to accomplish this?
I can't find any idea or resources for this issue.
In the Azure Kinect documentation, I found "k4a_transformation_depth_image_to_color_camera_custom" function in Azure Kinect SDK.
Is this method useful for my case?
And if true, how can I get the k4a_transformation_t value for its parameter?
I am currently doing a project with a FLIR Blackfly camera (BFS-U3-120S4C to be exact), and have not been getting a low enough reprojection error in my calibration experiments.
I know that manufacturers often have the intrinsic parameters and distortion coefficients available, and FLIR has a Spinnaker Python API to operate their cameras and they advertise them as Machine Vision cameras, so I strongly suspected that this would be the case for them.
However, I have not been successful in finding this in my Google searches. Does anyone know of FLIR have given out the values for these parameters somewhere?
Thanks for the help!
I can't figure out how can I get raw depth data from my Xiaomi Redmi Note 8 Pro with rear ToF depth sensor.
I've tried:
Camera2Format from android samples. No results (and I'm not alone)
get physicalId from rear camera as described in android developers blog. No result.
ARCore API can't handle depth.
I saw several similar questions (one, two) without answers. Is it so difficult to capture raw depth from depth camera in Android?
Raw Depth API vs Full Depth API
At Google I/O 2021, the Raw Depth API for ARCore 1.24 version was announced in addition to the existing Full Depth API, working since ARCore 1.18. This second-Gen depth API gives you the ability to merge Raw Depth data coming from iToF sensor with data coming from Depth-from-Motion ML-algorithm. The resulted data can be retrieved through 16-bit buffer stream, where 13-bit is reserved for depth map and 3-bit is reserved for confidence map.
At Google I/O 2022, Full Depth API was updated for users running ARCore 1.31. Now, all 16 bits per pixel are used, allowing app to represent the depth within a distance of 65,535 millimeters.
Today ARCore developers can use Raw Depth API for Precise Measurements and for ops on Reconstructed Mesh. On the other hand, Full Depth API is ideally suitable for People/Objects Occlusion and Scene Relighting purposes.
The following table represents three important parameters:
Full Depth API (v1.31+)
Raw Depth API (v1.24+)
Full Depth API (v1.18+)
Accuracy
Bad
Awesome
Bad
Coverage
All pixels
Not all pixels
All pixels
Distance
0 to 65.5 m
0.5 to 5.0 m
0 to 8.2 m
Raw Depth is available on all devices that support the Depth API. The Raw Depth API, like the Full Depth API, does not require a supported hardware depth sensor, such as a time-of-flight (iToF) sensor. However, both the Raw Depth API and the Full Depth API make use of any supported hardware sensors that a device may have.
The list of ARCore Depth API supported devices is growing. Pay attention to the last column called Comments - info about Depth API support is there.
I would like to experiment with machine learning (especially CNNs) on the aligned RGB and depth stream of either an Intel RealSense or an Orbbec Astra camera. My goal is to do some object recognisation and highlight/mark them in the output video stream (as a starting point).
But after having read many articles I am still confused about the involved frameworks and how the data flows from the camera through the involved software components. I just can't get a high level picture.
This is my assumption regarding the processing flow:
Sensor => Driver => libRealSense / Astra SDK => TensorFlow
Questions
Is my assumption correct regarding the processing?
Orbbec provides an additional Astra OpenNI SDK besides the Astra SDK where as Intel has wrappers (?) for OpenCV and OpenNI. When or why would I need this additional libraries/support?
What would be the quickest way to get started? I would prefer C# over C++
Your assumptions are correct: the data acquisition flow is: sensor -> driver -> camera library -> other libraries built on top of it (see OpenCV support for Intel RealSense)-> captured image. Once you got the image, you can do whatever you want of course.
The various libraries allow you to work easily with the device. In particular, OpenCV compiled with the Intel RealSense support allows you to use OpenCV standard data acquisition stream, without bothering about the image format coming from the sensor and used by the Intel library. 10/10 use these libraries, they make your life easier.
You can start from the OpenCV wrapper documentation for Intel RealSense (https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv). Once you are able to capture the RGBD images, you can create your input pipeline for your model using tf.data and develop in tensorflow any application that uses CNNs on RGDB images (just google it and look on arxiv to have ideas about the possible applications).
Once your model has been trained, just export the trained graph and use it in inference, hence your pipeline will become: sensor -> driver -> camera library -> libs -> RGBD image -> trained model -> model output