We ran into a strange problem with some of our in-house developed applications and thought it was something deep in the code but then we wrote a quick sample to test it, we experienced the same issue.
Here's the code for the sample:
#include <stdio.h>
int main(void)
{
int i;
int j;
int COUNT = 750000;
double x[100];
double y[100];
FILE *OutputFile1;
FILE *OutputFile2;
FILE *OutputFile3;
FILE *OutputFile4;
FILE *OutputFile5;
FILE *OutputFile6;
FILE *OutputFile7;
FILE *OutputFile8;
FILE *OutputFile9;
OutputFile1 = fopen("Output_file_1.dat","w");
OutputFile2 = fopen("Output_file_2.dat","w");
OutputFile3 = fopen("Output_file_3.dat","w");
OutputFile4 = fopen("Output_file_4.dat","w");
OutputFile5 = fopen("Output_file_5.dat","w");
OutputFile6 = fopen("Output_file_6.dat","w");
OutputFile7 = fopen("Output_file_7.dat","w");
OutputFile8 = fopen("Output_file_8.dat","w");
OutputFile9 = fopen("Output_file_9.dat","w");
/* Do stuff in here */
/* Initialize the arrays */
for( i = 0; i < 100; i++)
{
x[i] = 2.50 * (double)i;
y[i] = 10.0 * (double)i;
}
printf("Initialized the x and y arrays\n");
/* Write junk to files */
for( i = 0; i < COUNT; i++)
{
printf("Outer loop %d\n", i);
for( j = 0; j < 100; j++)
{
fprintf(OutputFile1," %e", x[j]);
fprintf(OutputFile2," %e", x[j]);
fprintf(OutputFile3," %e", x[j]);
fprintf(OutputFile4," %e", x[j]);
fprintf(OutputFile5," %e", x[j]);
fprintf(OutputFile6," %e", y[j]);
fprintf(OutputFile7," %e", y[j]);
fprintf(OutputFile8," %e", y[j]);
fprintf(OutputFile9," %e", y[j]);
}
fprintf(OutputFile1,"\n");
fprintf(OutputFile2,"\n");
fprintf(OutputFile3,"\n");
fprintf(OutputFile4,"\n");
fprintf(OutputFile5,"\n");
fprintf(OutputFile6,"\n");
fprintf(OutputFile7,"\n");
fprintf(OutputFile8,"\n");
fprintf(OutputFile9,"\n");
}
/* End doing stuff here */
fflush(OutputFile1);
fclose(OutputFile1);
fflush(OutputFile2);
fclose(OutputFile2);
fflush(OutputFile3);
fclose(OutputFile3);
fflush(OutputFile4);
fclose(OutputFile4);
fflush(OutputFile5);
fclose(OutputFile5);
fflush(OutputFile6);
fclose(OutputFile6);
fflush(OutputFile7);
fclose(OutputFile7);
fflush(OutputFile8);
fclose(OutputFile8);
fflush(OutputFile9);
fclose(OutputFile9);
return(0);
}
So, here's what happens when you run this. If you run this in one terminal window and run top in another while it's running, you'll notice your memory being eaten away. It takes about 8 minutes for it to run and when it's finished, the system doesn't give the memory back, until the files are deleted. Once the files are deleted, all of the memory is released back to the system.
It's just C with the latest gcc compiler, CentOs 6.3.
Are we missing something?
Thanks!
"The system doesn't give the memory back". How do you know? There is a difference between "memory reported as free by top", and "memory you can use". This is because disk I/O is stored in cache when possible - that way, if you need to use the same file again, the information is already available in memory - faster to access. Once you delete a file, it's no longer useful to keep in cache - so the cache is cleared.
Another way to look at this is to use the free command. Doing this on my Linux box, I see the following:
total used free shared buffers cached
Mem: 66005544 65559292 446252 0 199832 60332160
-/+ buffers/cache: 5027300 60978244
Swap: 1044216 1884 1042332
The key line is the one that says "-/+ buffers/cache" . You can see that the first line tells me "446M free" - not a lot on a 64G machine. But the second line says "only joking, you have 60 G free". That is the real "free memory".
See whether that line "gives back memory" without having to delete the files. I think you will find it does.
The system caches these files for quick access at the next time. Since the memory isn't used by an application it's used for caching. The cache is freed if the files are gone or an other application needs more memory to run.
See: Linux Memory Management
Related
I'm trying to add some features to an older Qt4 application, and I'm new to Qt. The application uses the foreach keyword which I believe is implemented by Qt. However all foreach loops in the application only run once, regardless of the number of items that are in the container.
I added this sanity check to the application:
QString test("1234");
int i = 0;
foreach (QChar c, test) {
i++;
}
int stl = 0;
for (QString::iterator j = test.begin(); j != test.end(); j++) {
stl++;
}
qDebug()
<< "string:" << test
<< "size:" << test.size()
<< "foreach:" << i
<< "stl:" << stl
;
It always shows this message:
string: "1234" size: 4 foreach: 1 stl: 4
I've tested it with the above QString and with a QModelIndexList and each time it only runs the loop once, even when the container reports having more than one item, and in both cases the STL-style loop works fine, it's only the foreach that exits the loop early.
What am I doing wrong? The application is built against Qt 4.8.7.
For the record, it turns out this is a change in behaviour with GCC 9 (bug report) to do with where break; statements should appear and what they do.
It seems GCC versions before 9 did the wrong thing, but Qt 4 was written around that behaviour, so once it was fixed in GCC 9, Qt's foreach looping broke.
It looks like it has been addressed in recent Qt versions but not unfortunately in Qt 4.
Assume that the img is a 3 channel Mat in the code.
There is a memory leak with the code.
I guess that the pointers(reference) in the slice "matsplits" are not deleted with garbage-collection in go. How can I fix it?
for{
matsplits := gocv.Split(img)
matsplits[0].Close()
matsplits[1].Close()
matsplits[2].Close()
}
Kind of above codes cause memory leak. I am sure that the Mat objects in imgarr are closed, but the memory usage is still growing up. Why?
Update: part of codes in my project
processed := 0
for processed < proc.imgNumber {
grayhconcatImg := <-proc.processedImg[0][chindex]
var roiList roilist
var numStartPosList numStartPos
for x := 0; x < 11520-w; x++ {
test := gocv.NewMat()
testRegion := grayhconcatImg.img.Region(image.Rect(x, 0, x+w, h))
gocv.BitwiseXor(chimg, testRegion, &test)
testRegion.Close()
//testsplit := gocv.Split(test)
test.Close()
//testsplit[0].Close()
//testsplit[1].Close()
processed++
}
The memory leak occurs if "testsplit"'s are unmarked.
len(testsplit) is 2.
I have checked that testsplit[0] and testsplit[1] have been closed correctly after testsplit[i].Close().
After Close(), the memory will be clear partly after the gc comes.
Check it like
matsplits := gocv.Split(img)
matsplits[0].Close()
matsplits[1].Close()
matsplits[2].Close()
runtime.GC()
exec like GODEBUG=gctrace=1 go run main.go 2>xx.log
and you can find the what gc actual do in xx.log
So I am working on a program to read values from a game (Mitos.is: The Game).
It is similiar to Agar.io
You have a size (mass), and I want to get the mass amount, it is a program, not an online game like Agar.io.
I have found this Auto Hotkey script:
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn ; Enable warnings to assist with detecting common errors.
SendMode Input ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.
ReadMemory(MADDRESS=0,PROGRAM="",BYTES=4)
{
Static OLDPROC, ProcessHandle
VarSetCapacity(MVALUE, BYTES,0)
If PROGRAM != %OLDPROC%
{
WinGet, pid, pid, % OLDPROC := PROGRAM
ProcessHandle := ( ProcessHandle ? 0*(closed:=DllCall("CloseHandle"
,"UInt",ProcessHandle)) : 0 )+(pid ? DllCall("OpenProcess"
,"Int",16,"Int",0,"UInt",pid) : 0)
}
If (ProcessHandle) && DllCall("ReadProcessMemory","UInt",ProcessHandle,"UInt",MADDRESS,"Str",MVALUE,"UInt",BYTES,"UInt *",0)
{ Loop % BYTES
Result += *(&MVALUE + A_Index-1) << 8*(A_Index-1)
Return Result
}
return !ProcessHandle ? "Handle Closed:" closed : "Fail"
}
mass := ReadMemory(Address here, "Mitos.is: The Game")
MsgBox, %mass%
It works seamlessly but there is one slight problem, in Cheat Engine I took the liberty of finding the base address as shown below:
So I took the address circled here:
And inserted that into the program where it says "Address Here", correct me if this is not the right address, but when I restart the game and run my script it says "Fail", but in Cheat Engine the address is still valid. Help?
Check the address if it changes after restarting the game, or do not restart game just run the script without restart and you haven't defined bytes so try following,
ReadMemory(MADDRESS=0, PROGRAM="", BYTES=4 )
mass := ReadMemory("0x123456", "Mitos.is: The Game", "4")
"PROGRAM" should be correct window title use spy to get correct window title, address has to be in hex value i.e. "0x15B29DD0", I don't know how your cheat engine reads program memory address.
I have tried to release a Mat image from my program, however, no matter how I tried it, the same image (result) still appear when I click on the "process button" to process another image (from loading or snapping an image). The old results will always be displayed.
I have to close the whole program, open the next image I want to process and click the "process button" to get the actual result. However, this is not ideal, as I want my program to immediately process the image I load or snap and not having to close the whole program(or rather stop run and click run again)
Most of the results I searched from google suggested these methods:
imagep.release(); //where imagep is the image after going through processing
imagep=Mat();
UPDATE:
My code is goes something along this line. In my program.h file, the image and image p and declared like this:
class program : public QMainWindow
{
Q_OBJECT
public:
program(QWidget *parent = 0);
~program();
cv::Mat image, imagep; //original image and image processed
The coding for the button in the .cpp file goes something like this:
imagep = image.clone(); //also tried: imagep=image, and imagep=image+0 already.
.
.
.
processing/segmenting steps
.
.
cv::imshow("new image", imagep);
cvWaitKey(10);
imagep=Mat(); //also tried cvDestroyWindow("new image"), //imagep.release();
The setting of the picture of imagep(where I changed the values of the pixels based on results from the image processing. ) is created via:
for (int i=0; i< imagep.rows; ++i)
{
for (int j=0; j<imagep.cols; ++j)
{
//imagep.at<cv::Vec3b>(i,j)= v_char[i]; Note that this is commented. This method doesn't work, hence the below method. But it doesn't work too.
Vec3b temp=v_char[i];
imagep.at<cv::Vec3b>(i,j)[0]=temp[0];
imagep.at<cv::Vec3b>(i,j)[1]=temp[1];
imagep.at<cv::Vec3b>(i,j)[2]=temp[2];
}
}
However, these methods doesn't work. Any suggestions of how I can release the Mat image?
waitKey(-1) wait for a keyboard input while waitKey(10) uses 10 ms to update UI and then passes to control further on; It seems to that your problem is not release but proper update.
Normally one don’t need to release Mat (though I saw some release() calls in Java), just reassign it. Your problem may also be related to smart pointers that got killed when the number of references goes to zero but it is hard to say without seeing your code.
There are some tricky things happening when creating, assigning, and declaring Mats. Also, note for example that Mat m = n, copies the reference only but Mat m = n + 0, copies the whole object.
I just recently started to work with ImageJ (and thus do not have much experience with macro programming) to analyze my microscopy pictures.
In order to generate FRET pixel-by-pixel images that are corrected for spectral bleed through I am using the plug in: pixFRET. This plug in requires a stack of 3 images to work: FRET, Donor, Acceptor. So far, I have to open every picture myself and this is REALLY inconvenient for large time stacks (> 1000 images). I am looking for a way to loop the plug in or create some kind of macro to do this.
A short description of my Data structure:
workfolder\filename_t001c1 (Channel 1 Image - Donor at time point 001),
filename_t001c2 (Channel 2 Image - FRET at time point 001),
...t001c3 (can be neglected)
...t001c4 (Channel 4 Image - Acceptor at time point 001).
I would have to create a stack of C2/C1/C4 at each time point that is automatically analyzed by pixFRET (with set parameters) and the result should be saved in an output folder.
I am grateful for every suggestion as my biggest problem is the looping of this whole stack generation/pixFRET analysis (can only do this manual right now).
Thanks
David
I did not find a way to directly include the parameters and commands from the pixFRET PlugIn. However, here I show a work around that works with IJ_Robot to add these commands. I further included some stuff to perform a alignment of the camera channels based on the first images of the time series.
// Macro for creating time resolved pixFRET images with a alignment of both cameras used
// a separate setting file is required for pixFRET -> put this into the same folder as the pixFRET plugin
// the background region has to be set manually in this macro
// IJ_robot uses cursor movements - DO NOT move the cursor while excuting the macro + adjust IJ_robot coordinates when changing the resolution/system.
dir = getDirectory("Select Directory");
list = getFileList(dir);
//single alignment
run("Image Sequence...", "open=[dir] number=2 starting=1 increment=1 scale=100 file=[] or=[] sort");
rename(File.getName(dir));
WindowTitle=getTitle()
rename(WindowTitle+toString(" Main"))
MainWindow=getTitle()
NSlices=getSliceNumber()
xValue=getWidth()/2
yValue=getHeight()/2
//setTool("rectangle");
makeRectangle(0, 0, xValue, yValue);
run("Align slices in stack...", "method=5 windowsizex="+toString(xValue*2-20)+" windowsizey="+toString(yValue*2-20)+" x0=10 y0=10 swindow=0 ref.slice=1 show=true");
selectWindow("Results");
XShift=getResult("dX", 0);
YShift=getResult("dY", 0);
File.makeDirectory(toString(File.getParent(dir))+toString("\\")+"test"+" FRET");
for(i=0;i<list.length;i+=4){
open(dir+list[i+1]);
run("Translate...", "x=XShift y=YShift interpolation=None stack");
open(dir+list[i]);
open(dir+list[i+3]);
run("Translate...", "x=XShift y=YShift interpolation=None stack");
wait(1000);
run("Images to Stack", "name=Stack title=[] use");
selectWindow("Stack");
makeRectangle(15, 147, 82, 75); //background region
run("PixFRET...");
run("IJ Robot", "order=Left_Click x_point=886 y_point=321 delay=500 keypress=[]");
run("IJ Robot", "order=Left_Click x_point=874 y_point=557 delay=500 keypress=[]");
selectWindow("NFRET (x100) of Stack");
save(toString(File.getParent(dir))+toString("\\")+"test"+" FRET"+toString(i) +".tif");
selectWindow("Stack");
close();
selectWindow("FRET of Stack");
close();
selectWindow("NFRET (x100) of Stack");
close();
run("IJ Robot", "order=Left_Click x_point=941 y_point=57 delay=300 keypress=[]");
}
Thanks for your help Jan. If you can think of a way to call these pixFRET commands directly rather than using Ij_robot, please let me know.
Take this tutorial from Fiji (is just ImageJ) as a starting point, and use the macro recorder (Plugins > Macros > Record...) to get the neccessary commands.
Your macro code could then look something like this:
function pixfret(path, commonfilename) {
open(path + commonfilename + "c2");
open(path + commonfilename + "c1");
open(path + commonfilename + "c4");
run("Images to Stack", "name=Stack title=[] use");
run("PixFRET"); // please adjust this to your needs
}
setBatchMode(true);
n_timepoints = 999;
dir = "/path/to/your/images/";
for (i = 0; i < n_timepoints; i++)
pixfret(dir, "filename_t" + IJ.pad(i, 4));
setBatchMode(false);
Hope that helps.