I'm repeatedly calling a python module I wrote in C++ using the Python C API. My python program repeatedly calls my module's pyParse function, which does a bunch of stuff and returns a PyTuple that contains more PyTuple objects as elements. Every returned object ends up with a PyObject->refcnt of 1, so you know the object should be deleted when it goes out of scope in python. I repeatedly call this module with something like the following python code:
import my_module #this is my c++ module.
import os
path = 'C:/data/'
for filename in os.listdir(path):
data = my_module.pyParse(path+filename)
The longer this loop runs, the more the memory usage blows up. Every iteration produces about 2kb of tuples (which should be destroyed at end of every iteration). Yet when I take "heap snapshots" and compare an early one to another many more iterations later, you can see the allocation of memory called by PyTuple_New and other python objects keeps growing.
Yet because every returned object has 1 as a reference count, I would expect that it would be destroyed after going out of scope in python. Finally, my program ends in a read access violation in a random part of the code. Is there something I am missing? Or does anyone know how to possibly debug this and get a better handle on what's going on? I'm desperate!
Related
I am working on a Fortran code with MPI on ray-tracing.
The issue is that the code has a large array that is needed in all processors.
But this array is read-only for all processors at every time step,
then it is updated at the beginning of the next time step.
The current code uses one processor to read this array from an external file,
then simply broadcast it to all processors. In this way, every processor has a copy of this array.
To save some memory, I was planning to use OpenMP, but this question MPI Fortran code: how to share data on node via openMP? shows that MPI3.0 can do the trick.
But in the answers of the above question, unfortunately there is no example on how to broadcast the array to nodes.
I wonder anyone can help to provide an example?
Thanks in advance!
EDIT: I included some more details of the code structure as suggested by Vladimir.
Below is a pseudo-code showing the current treatment in the code:
allocate(largearray,size)
do i=1,10
!!! Update largearray
if(myrank==0) then
~~~read from file~~~ largearray
call MPI_BCAST(largearray,size,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,ierr)
do while(.true.)
call MPI_RECEIVE(results)
call collector(results,final_product)
exit
end do
call output(final_product,i)
else
call MPI_BCAST(largearray,size,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,ierr)
call radiation(largearray,beams)
call raytracing(beams,largearray,results)
call MPI_SEND(results,0)
end if
end do
deallocate(largearray)
I'm reading some source codes of a project, which is a combination of c++ and lua, they are interwined through luabind.
There is a la.lua file, in which there is a function exec(arg). The lua file also uses functions/variables from other lua file, so it has statements as below in the beginning
module(..., package.seeall);
print("Loading "..debug.getinfo(1).source.."...")
require "client_config"
now I want to run la.exec() from interactive terminal(on linux), but I get errors like
attempt to index global 'lg' (a nil value)
if I want to import la.lua, I get
require "la"
Loading #./la.lua...
./la.lua:68: attempt to index global 'ld' (a nil value)
stack traceback:
./lg.lua:68: in main chunk
[C]: in function 'require'
stdin:1: in main chunk
[C]: ?
what can I do?
Well, what could be going wrong?
(Really general guesswork following, there's not much information in what you provided…)
One option is that you're missing dependencies because the files don't properly require all the things they depend on. (If A depends on & requires B and then C, and C depends on B but doesn't require it because it's implicitly loaded by A, directly loading C will fail.) So if you throw some hours at tracking down & fixing dependencies, things might suddenly work.
(However, depending on how the modules are written this may be impossible without a lot of restructuring. As an example, unless you set package.loaded["foo"] to foo's module table in foo before loading submdules, those submodules cannot require"foo". (Luckily, module does that, in newer code without module that's often forgotten – and then you'll get an endless loop (until the stack overflows) of foo loading other modules which load foo which loads other modules which …) Further, while "fixing" things so they load in the interpreter you might accidentally break the load order used by the program/library under normal operation which you won't notice until you try to run that one normally again. So it may simply cost too much time to fix dependencies. You might still be able to track down enough to construct a long lua -lfoo-lbar… one-off dependency list which might get things to run, but don't depend on it.)
Another option is that there are missing parts provided by C(++) modules. If these are written in the style of a Lua library (i.e. they have luaopen_FOO), they might load in the interpreter. (IIRC that's unlikely for C++ because it expects the main program to be C++-aware but lua is (usually? always?) plain C.) It's also possible that these modules don't work that way and need to be loaded in some other way. Yet another possibility might be that the main program pre-defines things in the Lua state(s) that it creates, which means that there is no module that you could load to get those things.
While there are some more variations on the above, these should be all of the general categories. If you suspect that your problem is the first one (merely missing dependency information), maybe throw some more time at this as you have a pretty good chance of getting it to work. If you suspect it's one of the latter two, there's a very high chance that you won't get it to work (at least not directly).
You might be able to side-step that problem by patching the program to open up a REPL and then do whatever it is you want to do from there. (The simplest way to do that is to call debug.debug(). It's really limited (no multiline, no implicit return, crappy error information), but if you need/want something better, something that behaves very much like the normal Lua REPL can be written in ~30 lines or so of Lua.)
I am trying to hack a game (not for cheating though) by introducing new built-in methods and functions in order to communicate with the game using sockets. Here is a small "pseudo code" example of what I want to accomplish:
Inside the Lua code I am calling my_hack() and pass the current game state:
GameState = {}
-- Game state object to be passed on
function GameState:new()
-- Data
end
local gameState = GameState:new()
-- Collect game state data and pass it to 'my_hack' ..
my_hack(gameState)
and inside my_hack the object is getting sent away:
int my_hack(lua_State * l)
{
void* gameState= lua_topointer(l, 1);
// Send the game state:
socket->send_data(gameState);
return 0;
}
Now, the big question is how to introduce my_hack() to the game?
I assume, that all built in functions must be kept in some sort of lookup table. Since all Lua code is getting interpreted, functions like import etc. will have to be statically available, right? If that is correct, then it should be "enough" to find out where this code is residing in order to smuggle my code into the game that would allow me to call my_hack() in a Lua script.
There should be two options: The first is that the Lua built is embedded inside the executable and is completely static and the second is that all Lua code gets loaded dynamically from a DLL.
This question goes out to anybody who has a slightest clue about where and how I should keep looking for the built in functions. I've tried a few things with Cheat Engine but I wasn't too successful. I was able to cheat a bit ^^ but that's not what I'm looking out for.
Sorry for not providing a full answer, but if you can provide a custom Lua VM and change the standard libraries, you should be able to to change the luaL_openlibs method in the Lua source to provide a table with my_hack() inside of it.
Since the Lua interpreter is usually statically compiled into the host executable, modifying the interpreter in some way will probably not be possible.
I think your best bet is to find some piece of Lua code which gets called by the host, and from that file use dofile to run your own code.
I am having trouble using pthreads in my MPI program. My program runs fine without involving pthreads. But I then decided to execute a time-consuming operation in parallel and hence I create a pthread that does the following (MPI_Probe, MPI_Get_count, and MPI_Recv). My program fails at MPI_Probe and no error code is returned. This is how I initialize the MPI environment
MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided_threading_support);
The provided threading support is '3' which I assume is MPI_THREAD_SERIALIZED. Any ideas on how I can solve this problem?
The provided threading support is '3' which I assume is MPI_THREAD_SERIALIZED.
The MPI standard defines thread support levels as named constants and only requires that their values are monotonic, i.e. MPI_THREAD_SINGLE < MPI_THREAD_FUNNELED < MPI_THREAD_SERIALIZED < MPI_THREAD_MULTIPLE. The actual numeric values are implementation-specific and should never be used or compared against.
MPI communication calls by default never return error codes other than MPI_SUCCESS. The reason for that is, MPI calls the communicator's error handler before an MPI call returns and all communicators are initially created with MPI_ERRORS_ARE_FATAL installed as their error handler. That error handler terminates the program and usually prints some debugging information, e.g. the reason for the failure. Both MPICH (and its countless variants) and Open MPI produce quite elaborate reports on what led to the termination.
To enable user error handling on communicator comm, you should make the following call:
MPI_Comm_set_errhandler(comm, MPI_ERRORS_RETURN);
Watch out for the error codes returned - their numerical values are also implementation-specific.
If your MPI implementation isn't willing to give you MPI_THREAD_MULTIPLE, there's three things you can do:
Get a new MPI implementation.
Protect MPI calls with a critical section.
Cut it out with the threading thing.
I would suggest #3. The whole point of MPI is parallelism -- if you find yourself creating multiple threads for a single MPI subprocess, you should consider whether those threads should have been independent subprocesses to begin with.
Particularly with MPI_THREAD_MULTIPLE. I could maybe see a use for MPI_THREAD_SERIALIZED, if your threads are sub-subprocess workers for the main subprocess thread... but MULTIPLE implies that you're tossing data around all over the place. That loses you the primary convenience offered by MPI, namely synchronization. You'll find yourself essentially reimplementing MPI on top of MPI.
Okay, now that you've read all that, the punchline: 3 is MPI_THREAD_MULTIPLE. But seriously. Reconsider your architecture.
I'm using a game engine that allows you to program in Lua. The game engine commands are in a DLL created from C. There is a exe created in C that calls a Lua file. This Lua file is where you put all your game code including the main loop. There is no going back and forth with the exe, but you can call functions from the DLL.
So in here before the main loop I create a function which I'm going to create a coroutine from. This function iterates over a pretty big table so every n iterations I yield. This function has an infinite while loop around that because I need this stuff to run every single cycle of the main game loop, but it's OK if it's split between multiple cycles.
I then create a coroutine with this function as the parameter. In the main game loop I then resume this coroutine.
When I run my code I get the error: tempt to yield across metamethod/C-call boundary
I was reading some stuff online but not really understanding what the issue is here. Once the exe calls the Lua file it doesn't go back to the exe at all until the Lua file is finished, and since I have my main loop in the Lua file it never finishes in my test case.
What are my options with this then?
The error is telling you that you are attempting to yield from within Lua code where there is some C function between the Lua code doing the yielding and the Lua code that resumed the coroutine. To hit this error, what you have to have done is call some C function from Lua, which calls back into Lua code, which then calls coroutine.yield().
You can't do that. You must instead restructure your code to avoid this. Since you didn't provide any code, there's not much that can be suggested.
There are several things you can do if you cannot change your code to avoid the C/metamethod boundary:
If you are using standard Lua, and are compiling it yourself, try patching it with Coco — True C Coroutines for Lua.
True C coroutine semantics mean you can yield from a coroutine across a C call boundary and resume back to it.
Try using LuaJIT instead of the standard Lua interpreter. It uses a fully resumable VM meaning the boundary is not an issue.
Try using Lua 5.2. It features yieldable pcall and metamethods which means that it can handle your problem. However, there are some changes and incompatibilities between Lua 5.1 and Lua 5.2.