I'm new to Lua and I have question regarding to memory management in Lua.
Question 1) When calling function using io.popen(), I saw many Lua programmers wrote a close statement after using popen() function. I wonder what is the reason for that? For example, to demonstrate look at this code:
handle = io.popen("ls -a")
output = handle:read("*all")
handle:close()
print(output)
handle = io.popen("date")
output = handle:read("*all")
handle:close()
print(output)
I heard Lua can manage memory itself. So do I really need to write handle:close like above? What will happen to memory if I just ignore the handle:close() statement and just write it like this?
handle = io.popen("ls -a")
handle = io.popen("date")
output = handle:read("*all")
Question 2) From the code in question 1, in term of memory usage, can we write the handle:close() statement at the end with only one line instead of two like this ?:
handle = io.popen("ls -a")
output = handle:read("*all")
-- handle:close() -- dont close it yet do at the end
print(output)
handle = io.popen("date") -- this use the same variable `handle` previously
output = handle:read("*all")
handle:close() -- only one statement to close all above
print(output)
You can see that I didn't close this from the first statement when I use io.popen but I close it at the end, will this make the program slow because I close it only with one close statement at the end?
Lua will close the file handle automatically when the garbage collector gets around to collecting it.
Lua Manual 5.4: file:close
Closes file. Note that files are automatically closed when their handles are garbage collected, but that takes an unpredictable amount of time to happen.
BUT, it is best practice to close the handles yourself as soon as you are done with the handle, this is because it will take an unknown amount of time for the GC to do it.
This is not an issue of memory but of a much more limited resource of open file handles, something like 512 on a windows machine, a small pool for all the applications running on it.
As for the second question, when you reassign a variable AND there are no other remaining references to the previous value, that value will eventually be collected by the GC.
Question 1
In this case the close is not for memory reasons, but to close the file. When a file handle gets collected, it will be closed automatically, but if a program doesn't generate much garbage (which some programmers specifically optimize for), the GC might not run for quite a while after the program is done with the file handling and the file would stay open.
Also, if the variable stays in scope, then the GC won't get to collect it at all until the scope ends, which might be a very long time.
Question 2
That wouldn't work. Methods get called on values, not on variables, so when you assign a new value to a variable, the old one just disappears. Calling a method on the new value won't affect any other value that used to be stored in the variable.
Related
Say I've this statement:
final output = MyEnum.values[myPreferences.getIndex('some_key') ?? getDefaultIndex('some_key')];
I can store the value of those two expressions in a local variable like:
final savedIndex = myPreferences.getIndex('some_key');
final defaultIndex = getDefaultIndex('some_key');
final output = MyEnum.values[savedIndex ?? defaultIndex];
Will this cause performance / memory issues if I do the same refactoring (create two additional local variables to make the code look more readable) on 1000s of my output type variables?
PS: Please don't advise on how I can get the default value in a better way from SharedPreferences and how I could reduce the above statement. I am just using the example to give an illustration.
Most likely not. I wouldn't worry about it.
Dart can be hard to optimize in some cases, but local variables is one thing compilers are very good at dealing with.
Unless your code allocates a gargantuan object, puts it into a local variable, uses it once and never needs it again, and then goes on to do a very long computation, I wouldn't worry about a local variable keeping the value alive. Even in that situation, the compiler will almost certainly still not keep the value alive after it's no longer needed (and if it does, it would probably do it even if you don't use a variable for your intermediate value), and I'd only start investigating if I'm actually seeing issues.
Don't worry, trust the compiler.
i'm currently trying to find a way to return a static value which would represent how much memory standart a function takes or its time of execution (as a static thread), I thought about using coroutines, however I cannot make any working prototypes, thanks for help in advance ! (:
The Lua function collectgarbage with the string "count" as an argument returns a number that reflects the amount of memory currently in use by the interpreter. Here is a link to an example and more information; I will reproduce the example here:
function memuse()
local i = math.modf(collectgarbage("count") * 1024)
return i
end
This function returns the amount of memory, in kilobytes, currently in use by Lua.
As for time, the simplest way is to call os.time(), which returns the current system time. Note, though, that this only returns the number of seconds to the nearest whole number. If you need greater precision, there are a few options: one, place a system call with io.popen to retrieve the current system time, which includes the non-integer portion; or two, implement some time-related functions in C/C++ and call them from Lua. I have used both options and the second one produces excellent accuracy, but for the sake of simplicity I will just show the first one.
-- Function called 'tick' to retrieve the current OS time.
function tick()
local fil = assert(io.popen("date +%s.%N"))
local str = fil:read("*all")
return tonumber(str)
end
Lua file handles--one of which is produced by the call to io.popen--have their own destructors so they need not be explicitly closed; however, you may want to call fil:close() for the sake of garbage collection and avoiding any open file-related errors.
If you want to pursue the second, more complicated option, I suggest creating a timer class in C++ that makes use of the chrono library to retrieve the system time.
I am not sure that these two functions are of relevance to you, but I hope they help.
I'm writing a package which makes heavy use of buffers internally for temporary storage. I have a single global (but not exported) byte slice which I start with 1024 elements and grow by doubling as needed.
However, it's very possible that a user of my package would use it in such a way that caused a large buffer to be allocated, but then stop using the package, thus wasting a large amount of allocated heap space, and I would have no way of knowing whether to free the buffer (or, since this is Go, let it be GC'd).
I've thought of three possible solutions, none of which is ideal. My question is: are any of these solutions, or maybe ones I haven't thought of, standard practice in situations like this? Is there any standard practice? Any other ideas?
Screw it.
Oh well. It's too hard to deal with this, and leaving allocated memory lying around isn't so bad.
The problem with this approach is obvious: it doesn't solve the problem.
Exported "I'm done" or "Shrink internal memory usage" function.
Export a function which the user can call (and calling it intelligently is obviously up to them) which will free the internal storage used by the package.
The problem with this approach is twofold. First, it makes for a more complex, less clean interface to the user. Second, it may not be possible or practical for the user to know when calling such a function is wise, so it may be useless anyway.
Run a goroutine which frees the buffer after a certain period of the package going unused, or which shrinks the buffer (perhaps halving the length) whenever its size hasn't been increased in a while.
The problem with this approach is primarily that it puts unnecessary strain on the scheduler. Obviously a single goroutine isn't so bad, but if this were accepted practice, it wouldn't scale well if every package you imported were doing this under the hood. Also, if you have a time-sensitive application, you may not want code running when you're not aware of it (that is, you may assume that the package isn't doing any work when its functions are not being called - a reasonable assumption, I'd say).
So... any ideas?
NOTE: You can see the existing project here (the relevant code is only a few tens of lines).
A common approach to this is letting the client pass an existing []byte (or whatever) as an argument to some call/function/method. For example:
// The returned slice may be a sub-slice of dst if dst was large enough
// to hold the entire encoded block. Otherwise, a newly allocated slice
// will be returned. It is valid to pass a nil dst.
func Foo(dst []byte, whatever Bar) (ret []byte, err error)
(Example)
Another approach is to get a new []byte from a, for example cache and/or for example pool (if you prefer the later name for that concept) and rely on clients to return used buffers to such "recycle-bin".
BTW: You're doing it right by thinking about this. Where it's possible to reasonably reuse []byte buffers, there's a potential for lowering the GC load and thus making your program better performing. Sometimes the difference can be critical.
You could reslice your buffer at the end of every operation.
buffer = buffer[:0]
Then your function extendAndSliceBuffer would have the original backing array most likely available if it needs to grow. If not, you would suffer a new allocation, which you might get anyway when you do extendAndSliceBuffer.
Overall, I think a cleaner solution is to do like #jnml said and let the users pass their own buffer if they care about performance. If they don't care about performance, then you should not use a global var and simply allocate the buffer as you need and let it go when it gets out of scope.
I have a single global (but not exported) byte slice which I start
with 1024 elements and grow by doubling as needed.
And there's your problem. You shouldn't have a global like this in your package.
Generally the best approach is to have an exported struct with attached functions. The buffer should reside in this struct unexported. That way the user can instantiate it and let the garbage collector clean it up when they let go of it.
You also want to avoid requiring globals like this as it can hamper unit tests. A unit test should be able to instantiate the exported struct, as the user can, and do it each time for every test.
Also depending on what kind of buffer you need, bytes.Buffer may be useful as it already provides io.Reader and io.Writer functions. bytes.Buffer also automatically grows and shrinks its buffer. In buffer.go you'll see various calls to b.Truncate(0) that does the shrinking with the comment "reset to recover space".
It's generally really really bad form to write Go code that is not thread-safe. If two different goroutines call functions that modify the buffer at the same time, who knows what state the buffer will be in when they finish? Just let the user provide a scratch-space buffer if they decide that the allocation performance is a bottleneck.
I have a program which calculates 'Printer Queues Total' value using '/usr/bin/lpstat' through popen() system call.
{
int n=0;
FILE *fp=NULL;
printf("Before popen()");
fp = popen("/usr/bin/lpstat -o | grep '^[^ ]*-[0-9]*[ \t]' | wc -l", "r");
printf("After popen()");
if (fp == NULL)
{
printf("Failed to start lpstat - %s", strerror(errno));
return -1;
}
printf("Before fscanf");
fscanf(fp, "%d", &n);
printf("After fscanf");
printf("Before pclose()");
pclose(fp);
printf("After pclose()");
printf("Value=%d",n);
printf("=== END ===");
return 0;
}
Note: In the command line, '/usr/bin/lpstat' command is hanging for some time as there are many printers available in the network.
The problem here is, the execution is hanging at popen() system call, Where as I would expect it to hang at fscanf() which reads the output from the file stream fp.
If anybody can tell me the reasons for the hang at popen() system call, it will help me in modifying the program to work for my requirement.
Thanks for taking time in reading this post and your efforts.
What people expect does not always have a basis in reality :-)
The command you're running doesn't actually generate any output until it's finished. That would be why it would seem to be hung in the popen rather than the fscanf.
There are two possible reasons for that which spring to mind immediately.
The first is that it's implemented this way, with popen capturing the output in full before delivering the first line. Based on my knowledge of UNIX, this seems unlikely but I can't be sure.
Far more likely is the impact of the pipe. One thing I've noticed is that some filters (like grep) batch up their lines for efficiency. So, while popen itself may be spewing forth its lines immediately (well, until it gets to the delay bit anyway), the fact that grep is holding on to the lines until it gets a big enough block may be causing the delay.
In fact, it's almost certainly the pipe-through-wc, which cannot generate any output until all lines are received from lpstat (you cannot figure out how many lines there are until all the lines have been received). So, even if popen just waited for the first character to be available, that would seem to be where the hang was.
It would be a simple matter to test this by simply removing the pipe-through-grep-and-wc bit and seeing what happens.
Just one other point I'd like to raise. Your printf statements do not have newlines following and, even if they did, there are circumstances where the output may still be fully buffered (so that you probably wouldn't see anything until that program exited, or the buffer filled up).
I would start by changing them to the form:
printf ("message here\n"); fflush (stdout); fsync (fileno (stdout));
to ensure they're flushed fully before continuing. I'd hate this to be a simple misunderstanding of a buffering issue :-)
It sounds as if popen may be hanging whilst lpstat attempts to retrieve information from remote printers. There is a fair amount of discussion on this particular problem. Have a look at that thread, and especially the ones that are linked from that.
I've been writing some scripts for a game, the scripts are written in Lua. One of the requirements the game has is that the Update method in your lua script (which is called every frame) may take no longer than about 2-3 milliseconds to run, if it does the game just hangs.
I solved this problem with coroutines, all I have to do is call Multitasking.RunTask(SomeFunction) and then the task runs as a coroutine, I then have to scatter Multitasking.Yield() throughout my code, which checks how long the task has been running for, and if it's over 2 ms it pauses the task and resumes it next frame. This is ok, except that I have to scatter Multitasking.Yield() everywhere throughout my code, and it's a real mess.
Ideally, my code would automatically yield when it's been running too long. So, Is it possible to take a Lua function as an argument, and then execute it line by line (maybe interpreting Lua inside Lua, which I know is possible, but I doubt it's possible if all you have is a function pointer)? In this way I could automatically check the runtime and yield if necessary between every single line.
EDIT:: To be clear, I'm modding a game, that means I only have access to Lua. No C++ tricks allowed.
check lua_sethook in the Debug Interface.
I haven't actually tried this solution myself yet, so I don't know for sure how well it will work.
debug.sethook(coroutine.yield,"",10000);
I picked the number arbitrarily; it will have to be tweaked until it's roughly the time limit you need. Keep in mind that time spent in C functions etc will not increase the instruction count value, so a loop will reach this limit far faster than calls to long-running C functions. It may be viable to set a far lower value and instead provide a function that sees how much os.clock() or similar has increased.