I'm reading lots of 16MB tiff files into a numpy array, then into a python Queue (that never fills) from an external USB disk drive.
iotop in Linux reports file reading speed of around 11-12MB/s.
If I use ImageJ(a Java application) to read the same files, I get speeds of around 26MB/s.
This is the snippet that does the reading: It runs on a separate single thread to the rest of the application.
def fileReader(self):
for file_ in self.files:
im_array = np.asarray(Image.open(file_))
self.im_array_queue.put(im_array)
I'm wondering whether the lower speed is due to a Python IO limitation or whether it's the creation of the array.
Ive found tifffile to be excellent for loading tiffs, comparable to imagej: http://www.lfd.uci.edu/~gohlke/code/tifffile.py.html
It also supports 16bit and multipage tiffs
It's not a Python limitation.
Using openCV
import cv2
im = cv2.imread(file_)
I get speeds up to 26MB/s comparable with ImageJ
Related
We need to append two very large TIF files vertically using ImageMagick, but we are very limited on memory and disk resources because we are attempting to do this on AWS Lambda.
We currently use the very simply approach here...
magick convert image1.tif image2.tif -append result.tif
This works, but because of the size of each image, the memory and disk consumption is too high and we run into resource issues.
ImageMagick has a "stream" command (https://www.imagemagick.org/script/stream.php) but I cannot find any examples of how we might use it for what we are attempting to do.
We have tried other approaches, such a the -limit option, but we still run into issues. I am trying to determine how this could be done using the "stream" command, if it is possible at all. I have seen "stream" suggested for this use case, but no examples.
Any help greatly appreciated!
I'm not at a machine to test, but I suspect you can achieve that using much less memory, and time, with vips.
I think you'd want this at the command line:
vips join input1.tif input2.tif result.tif vertical
Add a final parameter of --vips-leak to check total memory used.
The join operation is documented here:
http://libvips.github.io/libvips/API/current/libvips-conversion.html#vips-join
There are node, PHP, Python, Ruby etc. bindings as well.
I created two 10,000x10,000 pixels TIF files and did the same append operation with ImageMagick and vips:
ImageMagick: 11 seconds and 4.86GB memory used
vips: 4 seconds and 157 MB memory used
We are working on project where we need to do some image processing on FPGA. For that purpose we are using ZedBoard with linaro (Ubuntu Version) running on it.
What we have already done is we have stored the image in binary form pixel by pixel in DDR using python script on Processing System of Zedboard.
Now our task is to read the content of DDR memory, process it and send back the processed output to DDR Memory again. We are using vivado xilinx tool for FPGA part. We tried to use AXI-DMA with AXI-Interconnect to read and write data from DDR.
My question is, Do We need to use SDK and some sort of C coding to read and write DDR Memory on Programmable Logic side? As we want to make our module start reading the data from DDR with a control signal and then start actual processing of Image data. Once we read specific block of data, process it and store the result back to DDR memory on the fly. We are not sure which IP Block do we need in our block design for vivado. Also do we need Block Ram Memory at the end before sending the date to DDR.
Can anyone who already done this sort of project or have any knowledge ? Any help from your side will be appreciated !
Thanks
The zynq FPGA provide an AMBA AXI interconnect for that purpose.
This is the interconnect on the right.
If anyone has used the iOS wrapper for the LZMA SDK available at https://github.com/mdejong/lzmaSDK and have been able to tweak it in order to see the progress of unarchiving, please help.
I am going to use this SDK in iOS to extract a 16MB file, which uncompresses to a 150MB file, and this takes around 40seconds to complete. It would be good to have some kind of callback for showing the progress of uncompression.
Help is greatly appreciated.
Thanks
So, I looked at this issue quite a bit recently, and honestly the best you are going to be able to do is look for all the files in a specific tmp dir where decompression is going on and then count them and compare to a known size N. The problem with attempting to do this in the library is that it spans multiple runtimes and the callback idea makes the code a mess. Also, a callback would not help that much because of the way 7z compression works. To decode, one needs to build up the decompression dictionary before specific files can be decompressed, and that process of building up the dictionary takes a long time before the first file can even be written. So, if you put a "percent done" counter in your app showing how much was done, it would show 0% done for a long time, then jump to 50% and then 90 or 100 %. Basically, it would not be that useful even if it was implemented.
You could try C++ port of the latest LZMA SDK(15.06) without described above limitations(C version). Memory allocations and IO read/write can be tuned in runtime, plus work with password encrypted archives, smoothed progress, Lzma & Lzma2 archive types etc.
GitHub: https://github.com/OlehKulykov/LzmaSDKObjC
I wonder if it is ok (apart from the fact it is done million times daily...) to feed ImageMagic's convert command with user-uploaded files?
Obviously, ImageMagic has a large attack front, as it is capable of loading tons of formats and thus using vast amounts of code processing input data.
Excessive CPU or memory consumption do no harm, as this can be kept under control by simple means (for example by ulimit). Abitrary file access or runing injected code with network access however must be prevented by convert. Can we expect this?
Are there procedures to follow for the ImageMagic authors that would provide a feasable amount of security against such exploits?
I have a beagleboard with TMS320C64x+ DSP. I'm working on an image processing beagleboard application. Here's how it's going to work:
The ARM reads an image from a file and puts the image in a 2D array.
The arm sends the matrix to the DSP. The DSP receives the matrix.
The DSP performs the image processing algorithm on the received matrix (the algorithm code uses about 5MB of dynamically allocated memory).
The DSP sends the processed image (matrix) to the ARM. The arm receives the matrix.
The arm saves the processed image to a file.
I'v already written the code for steps 1,3,5. What is the easiest way to do steps 3+4 (sending the data)? Code examples are welcome.
The easiest way is to use shared memory:
Use the CMEM kernel module to allocate a chunk of memory on the ARM that can be accessed from ARM and DSP. Then pass the pointer down to the DSP using the DspBios NOTIFY component.
Once the DSP is done with processing you can notify the ARM via NOTIFY.
This way there is no need to copy the data from the ARM to the DSP or vice versa. All you have to make sure is, that the data comes from the CMEM component. This makes sure the memory is contiguous (the DSP does not know about the ARM memory manager).
Shared memory is the right approach, but learning how to do it can be a pain. The C6Run tool can abstract the ARM/DSP communications for you making it easier. Although NOTIFY is really the right API to use, C6Run utilizes CMEM using an older API.
If you want to try C6Run out on the BeagleBoard, the easiest way is by following the instructions on the eLinux wiki for setting up C6Run for the ECE597 course given by Mark Yoder at Rose-Hulman. These instructions depend on running the Angstrom demo image(2). A stable version that was used to demonstrate functionality of the hardware is documented as well(3).
(2): www.angstrom-distribution.org/demo/beagleboard
(3): code.google.com/p/beagleboard/wiki/BeagleBoardDiagnosticsNext