I'm working on a project which uses knex.js for database migration. I tried to query the data and insert it into new table in this way.
return knex.schema.raw(`select * from transactions`)
.then(result =>{
console.log('Migrating transactions')
return knex('new_transactions').insert(result['rows'])
})
.catch((err)=>{
console.error(err)
})
}
However, because of the large amount of data in this table, it throws an error like this
<--- Last few GCs --->
[8748:0x102641000] 18957 ms: Mark-sweep 1182.2 (1241.7) -> 1182.2 (1203.2) MB, 1145.3 / 0.0 ms (average mu = 0.248, current mu = 0.000) last resort GC in old space requested
[8748:0x102641000] 19877 ms: Mark-sweep 1182.2 (1203.2) -> 1182.2 (1203.2) MB, 920.2 / 0.0 ms (average mu = 0.146, current mu = 0.000) last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x3dddbe65be3d]
1: StubFrame [pc: 0x3dddbe64b476]
Security context: 0x31960631e6e1 <JSObject>
2: replace [0x3196063105e1](this=0x319706b02201 <Very long string[19409793]>,0x31964e437719 <JSRegExp <String[9]: (\\*)(\?)>>,0x31964e437751 <JSFunction (sfi = 0x3196d83747d1)>)
3: query [0x3196d83070d9] [/Users/grace/node_modules/knex/lib/runner.js:~128] [pc=0x3dddbe6ea4fc](this=0x3196737add89 <Runner map = 0x31...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: 0x10003b125 node::Abort() [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
2: 0x10003b32f node::OnFatalError(char const*, char const*) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
3: 0x1001a8e85 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
4: 0x1005742a2 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
5: 0x10057d7a4 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
6: 0x10054caa6 v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
7: 0x10080b626 v8::internal::Runtime_RegExpExecMultiple(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
8: 0x3dddbe65be3d
Abort trap: 6
How can I solve this question?
You should use INSERT INTO if you would like to insert data from other table, or for loop and process the data with limit and offset, avoid pull the whole table in node js if you are not sure how big will it be.
FYI, to solve js heap out of memory, see this post
Related
I am facing an issue while using the cJSON Library. I am assuming that there is a memory leak that is breaking the code after a certain time (40 mins to 1 hr).
I have copied my code below :
void my_work_handler_5(struct k_work *work)
{
char *ptr1[6];
int y=0;
static int counterdo = 0;
char *desc6 = "RSRP";
char *id6 = "dBm";
char *type6 = "RSRP";
char rsrp_str[100];
snprintf(rsrp_str, sizeof(rsrp_str), "%d", rsrp_current);
sensor5 = cJSON_CreateObject();
cJSON_AddItemToObject(sensor5, "description", cJSON_CreateString(desc6));
cJSON_AddItemToObject(sensor5, "Time", cJSON_CreateString(time_string));
cJSON_AddItemToObject(sensor5, "value", cJSON_CreateNumber(rsrp_current));
cJSON_AddItemToObject(sensor5, "unit", cJSON_CreateString(id6));
cJSON_AddItemToObject(sensor5, "type", cJSON_CreateString(type6));
/* print everything */
ptr1[counterdo] = cJSON_Print(sensor5);
printk("Counterdo value is : %d\n", counterdo);
cJSON_Delete(sensor5);
counterdo = counterdo + 1;
if (counterdo==6){
for(y=0;y<=counterdo;y++){
free(ptr1[y]);
}
counterdo = 0;
}
return;
}
I read some other threads regarding freeing up the memory and tried to do the same. Can anyone let me know if this is the right approach to free up the space allocated to the cJSON Object.
Regards,
Adeel.
Since cJSON is a portable library with no dependencies, this is better to look for a potential issue in your code on a PC: they are specialized tools available in this environment for facilitating the investigation. I am assuming here you have a Linux system, a Windows system with WSL or WSL2 installed, or a Linux virtual machine, available, and gcc, valgrind installed.
A minimal, self-contained, portable version of your code could be:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <cJSON.h>
static int rsrp_current = 1;
static char *time_string = NULL;
void
my_work_handler_5 ()
{
char *ptr1[6];
int y = 0;
static int counterdo = 0;
char *desc6 = "RSRP";
char *id6 = "dBm";
char *type6 = "RSRP";
char rsrp_str[100];
snprintf (rsrp_str, sizeof (rsrp_str), "%d", rsrp_current);
cJSON *sensor5 = cJSON_CreateObject ();
cJSON_AddItemToObject (sensor5, "description", cJSON_CreateString (desc6));
cJSON_AddItemToObject (sensor5, "Time", cJSON_CreateString (time_string));
cJSON_AddItemToObject (sensor5, "value", cJSON_CreateNumber (rsrp_current));
cJSON_AddItemToObject (sensor5, "unit", cJSON_CreateString (id6));
cJSON_AddItemToObject (sensor5, "type", cJSON_CreateString (type6));
/* print everything */
ptr1[counterdo] = cJSON_Print (sensor5);
printf ("Counterdo value is : %d\n", counterdo);
cJSON_Delete (sensor5);
counterdo = counterdo + 1;
if (counterdo == 6)
{
for (y = 0; y <= counterdo; y++)
{
free (ptr1[y]);
}
counterdo = 0;
}
return;
}
int
main (int argc, char **argv)
{
time_t curtime;
time (&curtime);
for (int n = 0; n < 3 * 6; n++)
{
my_work_handler_5 ();
}
}
Build procedure:
wget https://github.com/DaveGamble/cJSON/archive/v1.7.14.tar.gz
tar zxf v1.7.14.tar.gz
gcc -g -O0 -IcJSON-1.7.14 -o cjson cjson.c cJSON-1.7.14/cJSON.c
Running valgrind on the program:
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose ./cjson
..indicates some memory is being freed that was not previously allocated: Invalid free() / delete / delete[] / realloc():
==6747==
==6747== HEAP SUMMARY:
==6747== in use at exit: 0 bytes in 0 blocks
==6747== total heap usage: 271 allocs, 274 frees, 14,614 bytes allocated
==6747==
==6747== All heap blocks were freed -- no leaks are possible
==6747==
==6747== ERROR SUMMARY: 21 errors from 2 contexts (suppressed: 0 from 0)
==6747==
==6747== 3 errors in context 1 of 2:
==6747== Invalid free() / delete / delete[] / realloc()
==6747== at 0x483CA3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==6747== by 0x1094DA: my_work_handler_5 (cjson.c:42)
==6747== by 0x10955A: main (cjson.c:59)
==6747== Address 0x31 is not stack'd, malloc'd or (recently) free'd
==6747==
==6747==
==6747== 18 errors in context 2 of 2:
==6747== Conditional jump or move depends on uninitialised value(s)
==6747== at 0x483C9F5: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==6747== by 0x1094DA: my_work_handler_5 (cjson.c:42)
==6747== by 0x10955A: main (cjson.c:59)
==6747== Uninitialised value was created by a stack allocation
==6747== at 0x109312: my_work_handler_5 (cjson.c:11)
==6747==
==6747== ERROR SUMMARY: 21 errors from 2 contexts (suppressed: 0 from 0)
Replacing:
for (y = 0; y <= counterdo; y++)
{
free (ptr1[y]);
}
by:
for (y = 0; y < counterdo; y++)
{
free (ptr1[y]);
}
and executing valgrind again:
==6834==
==6834== HEAP SUMMARY:
==6834== in use at exit: 1,095 bytes in 15 blocks
==6834== total heap usage: 271 allocs, 256 frees, 14,614 bytes allocated
==6834==
==6834== Searching for pointers to 15 not-freed blocks
==6834== Checked 75,000 bytes
==6834==
==6834== 1,095 bytes in 15 blocks are definitely lost in loss record 1 of 1
==6834== at 0x483DFAF: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==6834== by 0x10B161: print (cJSON.c:1209)
==6834== by 0x10B25F: cJSON_Print (cJSON.c:1248)
==6834== by 0x1094AB: my_work_handler_5 (cjson.c:30)
==6834== by 0x10959C: main (cjson.c:59)
==6834==
==6834== LEAK SUMMARY:
==6834== definitely lost: 1,095 bytes in 15 blocks
==6834== indirectly lost: 0 bytes in 0 blocks
==6834== possibly lost: 0 bytes in 0 blocks
==6834== still reachable: 0 bytes in 0 blocks
==6834== suppressed: 0 bytes in 0 blocks
==6834==
==6834== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Some memory is definitively being leaked.
The reason is that char *ptr1[6] is not static, and is therefore created on the stack every time my_work_handler_5() is being called. The pointers that were returned are by cJSON_Print() are therefore lost between two calls, and free() is being called on arbitrary pointer values, since ptr1[] is not initialized as it could be:
char *ptr1[6] = { NULL, NULL, NULL, NULL, NULL, NULL };
Since you are freeing memory every 6 calls, this is causing the memory leak you were suspecting.
Replacing:
char *ptr1[6];
by:
static char *ptr1[6];
compiling, running valgrind again:
==6927==
==6927== HEAP SUMMARY:
==6927== in use at exit: 0 bytes in 0 blocks
==6927== total heap usage: 271 allocs, 271 frees, 14,614 bytes allocated
==6927==
==6927== All heap blocks were freed -- no leaks are possible
==6927==
==6927== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
The modified version of the program should now work on your bare-metal system.
Compiled with g++ 4.7.4 on Solaris 8. 32 bit application. Stack trace is
Core was generated by `./z3'.
Program terminated with signal 10, Bus error.
\#0 0x012656ec in vector<unsigned long long, false, unsigned int>::push_back (this=0x2336ef4 <g_prime_generator>, elem=#0xffbff1f0: 2) at ../src/util/vector.h:284
284 new (m_data + reinterpret_cast<SZ *>(m_data)[SIZE_IDX]) T(elem);
(gdb) bt
\#0 0x012656ec in vector<unsigned long long, false, unsigned int>::push_back (this=0x2336ef4 <g_prime_generator>, elem=#0xffbff1f0: 2) at ../src/util/vector.h:284
\#1 0x00ae66d4 in prime_generator::prime_generator (this=0x2336ef4 <g_prime_generator>) at ../src/util/prime_generator.cpp:24
\#2 0x00ae714c in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at ../src/util/prime_generator.cpp:99
\#3 0x00ae71c4 in _GLOBAL__sub_I_prime_generator.cpp(void) () at ../src/util/prime_generator.cpp:130
\#4 0x00b16a68 in __do_global_ctors_aux ()
\#5 0x00b16aa0 in _init ()
\#6 0x00640b10 in _start ()
(gdb) list
279
280 void push_back(T const & elem) {
281 if (m_data == 0 || reinterpret_cast<SZ *>(m_data)[SIZE_IDX] == reinterpret_cast<SZ *>(m_data)[CAPACITY_IDX]) {
282 expand_vector();
283 }
284 new (m_data + reinterpret_cast\<Z *>(m_data)[SIZE_IDX]) T(elem);
285 reinterpret_cast<SZ *>(m_data)[SIZE_IDX]++;
286 }
287
288 void insert(T const & elem) {
(gdb) ptype SZ
type = unsigned int
(gdb) ptype m_data
type = unsigned long long *
SIGBUS on Solaris is usually indicative of a misaligned access, but I am not sure if it is due to the casting going on an endianess issue
The SPARC data alignment requirements is most likely at issue.
The m_data field in the vector class is off by two fields that are used
to store the size and capacity of a vector.
You can debug this by displaying (printing or using the debugger) the pointer m_data and it's alignment.
One option is to supply a separate vector implementation
where the size and capacity fields are stored
in fields directly in the vector for porting this library utility.
Z3 interacts with memory alignment a few other places (but not overly many).
The main other potential places are in the watch lists (sat_solver and smt_context), and region memory allocators (region.h) and possibly in hash tables.
I've run into an interesting issue with the Swift compiler, which seems to be caused by a very simple generic function. I've got a workaround, but I'd really like to understand what the underlying issue is.
In my app I have a requirement to fade in some UICollectionViewCells in a given order, sequentially, with a slight overlap between the animations.
I've implemented this using the methods below. The revealCells method takes an array of cells. If the collection is empty, it simply returns. Otherwise it animates the fade-in on the first cell in the array, and then waits a certain amount of time before making a recursive call to itself, passing all the cells except the one it just animated.
Code below:
func revealCells(cells:[UICollectionViewCell],animationTime:NSTimeInterval,overlap:Double) {
if cells.count > 0 {
let firstCell = cells.first
UIView.animateWithDuration(0.3, animations: { () -> Void in
firstCell?.alpha = 1.0
let timeMinusOverlap = animationTime - overlap
let delayTime = dispatch_time(DISPATCH_TIME_NOW,Int64(timeMinusOverlap * Double(NSEC_PER_SEC)))
dispatch_after(delayTime, dispatch_get_main_queue(), { () -> Void in
self.revealCells(self.cdr(cells),animationTime: animationTime,overlap: overlap)
})
})
}
}
}
//Returns the collection passed into the function with its first element removed
//i.e the traditional LISP cdr function
func cdr<T>(input:[T]) -> [T] {
var rVal = [T]()
if input.count == 1 {
rVal.append(input.first!)
} else {
rVal = [T](input[1...input.count-1])
}
return rVal
}
All this works fine in the simulator. But when I try to archive the build, the swift compiler crashes with the message Swift Compiler Error Command failed due to signal: Segmentation fault 11. My setup is Xcode 6.3.1 (iOSSDK 8.3), and my min deployment target is 8.3.
Fixing the problem is trivial - if I just replace the code inside the dispatch_after with:
let newCells = [UICollectionViewCell](cells[1...cells.count-1])
self.revealCells(newCells,animationTime: animationTime,overlap: overlap)
the problem goes away. So it seems to be something to do with the generic function, or possibly something block related.
Stack trace is:
0 swift 0x000000010105ba18 llvm::sys::PrintStackTrace(__sFILE*) + 40
1 swift 0x000000010105bef4 SignalHandler(int) + 452
2 libsystem_platform.dylib 0x00007fff8725ef1a _sigtramp + 26
3 libsystem_platform.dylib 0x000000000000000f _sigtramp + 2027557135
4 swift 0x00000001021a98cb llvm::AsmPrinter::EmitFunctionBody() + 4379
5 swift 0x000000010116c84c llvm::ARMAsmPrinter::runOnMachineFunction(llvm::MachineFunction&) + 220
6 swift 0x0000000100d81d13 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) + 99
7 swift 0x00000001024d5edf llvm::FPPassManager::runOnFunction(llvm::Function&) + 495
8 swift 0x00000001024d60cb llvm::FPPassManager::runOnModule(llvm::Module&) + 43
9 swift 0x00000001024d658f llvm::legacy::PassManagerImpl::run(llvm::Module&) + 975
10 swift 0x0000000100a09b41 performIRGeneration(swift::IRGenOptions&, swift::Module*, swift::SILModule*, llvm::StringRef, llvm::LLVMContext&, swift::SourceFile*, unsigned int) + 4369
11 swift 0x0000000100a09cb3 swift::performIRGeneration(swift::IRGenOptions&, swift::SourceFile&, swift::SILModule*, llvm::StringRef, llvm::LLVMContext&, unsigned int) + 51
12 swift 0x0000000100945687 frontend_main(llvm::ArrayRef<char const*>, char const*, void*) + 6647
13 swift 0x0000000100943ae6 main + 1814
14 libdyld.dylib 0x00007fff8a46b5c9 start + 1
(a long list of program arguments, which are mostly the names of file being compiled)
1. Running pass 'Function Pass Manager' on module '/Users/tolleyr/Library/Developer/Xcode/DerivedData/ParseTestQuiz-fgnfjkxxlyqfnrfrfntgtsjnrcfv/Build/Intermediates/ArchiveIntermediates/ParseTestQuiz/IntermediateBuildFilesPath/ParseTestQuiz.build/Release-iphoneos/ParseTestQuiz.build/Objects-normal/armv7/QuizQuestionViewController.o'.
2. Running pass 'ARM Assembly / Object Emitter' on function '#_TFC13ParseTestQuiz26QuizQuestionViewController13viewDidAppearfS0_FSbT_'
(the last part enabled me to figure out what code was causing the problem). The command being run was CompileSwift normal armv7
I'm going to file a radar for this, since the compiler itself is crashing , but thought I'd post it here in case anyone has an idea of what might be going on, or has run into the same issue.
The problem:
I have 2 MH_EXECUTE iOS binary files (compiled, no source code).
Lets name them binary1 and binary2.
I try to switch between them before UIApplicationMain is called!
1 try
I successfully do this with binary1 and one dylib. So I try to convert MH_EXECUTE to MH_DYLIB.
step 1
creating iOS Application binary1
#import <dlfcn.h>
int main(int argc, char * argv[])
{
NSLog(#"binary1 -> Hello, World!");
void *handle = dlopen([[[NSBundle mainBundle] pathForResource:#"binary2" ofType:nil] cStringUsingEncoding:NSUTF8StringEncoding], RTLD_NOW);
if (handle)
{
NSLog(#"DLOPEN is OK!");
}
else
{
NSLog(#"!OK ... --> %s", dlerror());
}
return 0;
}
creating iOS Application binary2
int main(int argc, char * argv[])
{
NSLog(#"binary2 -> Hello, World!");
return 0;
}
When I run binary1 I get:
step 2
Lets see the difference MH_EXECUTE vs MH_DYLIB
fullscreen
as we can see the main difference here is File Type: MH_EXECUTE vs MH_DYLIB
Lets change them and run binary1 again.
After change the result was out of address space
step 3
Lets see Load Commands
fullscreen
* in dylib there is NO __PAGEZERO segment
* dylib __TEXT segment VM address == 0 but in binary2 == 0000000100000000
So lets patch them too ... (patched: __TEXT, ___DATA and __LINKEDIT)
After run binary1 i get malformed mach-o image: segment __PAGEZERO overlaps load commands
step 4
I successfully removed __PAGEZERO from Load Commands now binary looks like dylib:
fullscreen
But on start binary1 I get BAD_ACCESS
Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Subtype: KERN_PROTECTION_FAILURE at 0x00000001019e0010
Triggered by Thread: 0
Thread 0 name: Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0 dyld 0x0000000120016d78 ImageLoaderMachOCompressed::rebase(ImageLoader::LinkContext const&) + 892
1 dyld 0x0000000120016c24 ImageLoaderMachOCompressed::rebase(ImageLoader::LinkContext const&) + 552
2 dyld 0x0000000120010c8c ImageLoader::recursiveRebase(ImageLoader::LinkContext const&) + 132
3 dyld 0x000000012001039c ImageLoader::link(ImageLoader::LinkContext const&, bool, bool, bool, ImageLoader::RPathChain const&) + 176
4 dyld 0x00000001200088e0 dyld::link(ImageLoader*, bool, bool, ImageLoader::RPathChain const&) + 180
5 dyld 0x000000012000df68 dlopen + 684
6 libdyld.dylib 0x0000000194e65b94 dlopen + 68
7 binary1 0x00000001000b7e18 main (main.m:16)
8 libdyld.dylib 0x0000000194e66a04 start + 0
Any idea ???
You're getting BAD_ACCESS because you removed __PAGEZERO and thus invalidated the rebasing opcodes. Keep __PAGEZERO but nullify it. I similarly converted an executable into a shared library and it's loading fine on iOS and macOS.
I have some trouble with managing memory in my function. Valgrid says I'm having a memory leak after the convert-function. Could it be because of the data is not being properly released? I've tried to use temp-pointers, but my program either crashes or is not working properly. Have someone encountered this problem before?
this->images.push_back(new cv::Mat()); //ID
cv::threshold(*this->images[MASK], *this->images[ID], 0.0, 1.0, cv::THRESH_BINARY);
this->images[ID]->convertTo(*this->images[ID], CV_32SC1);
This is the valgrid output:
==5663== 64,339,996 bytes in 1 blocks are possibly lost in loss record 380 of 380
==5663== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5663== by 0x4E95117: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.2.4.9)
==5663== by 0x4F31F38: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.2.4.9)
==5663== by 0x4F39CF9: cv::_OutputArray::create(cv::Size_<int>, int, int, bool, int) const (in /usr/local/lib/libopencv_core.so.2.4.9)
==5663== by 0x4EB9373: cv::Mat::convertTo(cv::_OutputArray const&, int, double, double) const (in /usr/local/lib/libopencv_core.so.2.4.9)
==5663== by 0x40D168: DataFrame::init() (DataFrame.cpp:68)
==5663== by 0x40C943: DataFrame::DataFrame(char const*, LeafClassifier*) (DataFrame.cpp:31)
==5663== by 0x414A19: DataHandler::loadFrame() (DataHandler.cpp:68)
==5663== by 0x406680: main (main.cpp:58)
please don't store pointers to Mat in your vector(or anywhere else !).
those things are refcounted internally, like smartpointers, and you're wrecking that by storing/copying pointers (a vector of pointers to smartpointers would sound silly anyway, no?).
use a plain vector<Mat>, trade some additional ~56 bytes per item against sound sleep tonight ;)