Indy TFTP Server Exception EIdTFTPAllocationExceeded - c++builder

I am receiving an EIdTFTPAllocationExceeded exception when transferring a file from me (the server - using the Indy TIdTrivialFTPServer component) to a device. I cannot find any information about what that exception might mean except maybe a disk space problem on the client (which I know is not the case because if I transfer the file through a different TFTP server, there is no problem).
What is the exception trying to tell me?
How do I get around it?
Is there any code that I'm missing?
My TFTP Server code (all of it) for the server is:
__fastcall TTrivialFTPServer::TTrivialFTPServer(TComponent* Owner) : TDataModule(Owner)
{
root = IncludeTrailingPathDelimiter(GetCurrentDir());
}
// ---------------------------------------------------------------------------
void __fastcall TTrivialFTPServer::tftpReadFile(TObject *Sender, UnicodeString &FileName, const TPeerInfo &PeerInfo, bool &GrantAccess, TStream *&AStream, bool &FreeStreamOnComplete)
{
FreeStreamOnComplete = true;
FileName = StringReplace(FileName, "/", "\\", TReplaceFlags() << rfReplaceAll);
FileName = ExtractFileName(FileName);
if (FileExists(root + "files\\" + FileName, false))
{
AStream = new TFileStream(root + "files\\" + FileName, fmOpenRead | fmShareDenyWrite);
GrantAccess = true;
}
else
{
GrantAccess = false;
}
}

After much searching and head scratching, I finally opened the IdTrivialFTPServer.pas file and found the problem. The code states:
if FBlkCounter = High(UInt16) then begin
raise EIdTFTPAllocationExceeded.Create('');
end;
When I added text to the exception I received the added text, so this is where the error is occurring. I tried converting from UInt16 to UInt32, but caused many more problems, so I wanted to see what would happen if I just commented out the check and let the counter roll back to zero.
As It turns out, nothing at all bad happens and the file transfers just fine!

Related

How RichEdit can accept file drag & dropping in BCB or Delphi program?

I'm writing a program that could drag and drop text files onto the form to show and edit it by RichEdit.
I've used ChangeWindowMessageFilterEx to make sure that WM_DROPFILES and WM_COPYDATA can received by my Main Form:
ChangeWindowMessageFilterEx(Handle, WM_DROPFILES, MSGFLT_ADD, NULL);
ChangeWindowMessageFilterEx(Handle, WM_COPYDATA, MSGFLT_ADD, NULL);
ChangeWindowMessageFilter(73 , MSGFLT_ADD);
and call DragAcceptFiles(Handle, true) in the form creation function.
Now the drag operation is valid on any places of the window but except the RichEdit, the cursor shows a deny icon when dragging on the RichEdit.
Dragging on any components, eg. text editors, panels, combo boxes and buttons, on the form can lead to receive the WM_DROPFILES message, but except RichEdit.
Actually, I'm sure that it is possible to drag files on the RichEdit because I have wrote the code last year, but I have lost the source code and forgot it. I'm trying to rebuild the same one now.
Here is the google drive download link to the executable file that I have finished last year. And here is the github url to the uncompleted source code that I'm writing currently.
Thank you for your watching.
I don't know why TRichEdit does not receive WM_DROPFILES when using a message map, but you could handle the WindowProc of the TRichEdit.
A possilble implementation could look like this:
Drop a TRichEdit on your Form
Modify header file
private:
TWndMethod OldWindowProc;
void __fastcall NewWindowProc(TMessage& Msg);
Add implementation
__fastcall TForm1::TForm1(TComponent* Owner)
: TForm(Owner)
{
OldWindowProc = RichEdit1->WindowProc;
RichEdit1->WindowProc = NewWindowProc;
DragAcceptFiles(RichEdit1->Handle, true);
}
void __fastcall TForm1::NewWindowProc(TMessage& Msg)
{
switch (Msg.Msg) {
case WM_DROPFILES:
{
HDROP DropH = (HDROP)Msg.WParam;
int droppedFileCount = DragQueryFile(DropH, 0xFFFFFFFF, NULL, 0);
TStringList* Buffer = new TStringList();
for (int i = 0; i < droppedFileCount; i++) {
int fileNameLength = DragQueryFile(DropH, i, NULL, 0);
String FileName;
FileName.SetLength(fileNameLength);
DragQueryFile(DropH, i, FileName.w_str(), fileNameLength + 1);
Buffer->LoadFromFile(FileName);
RichEdit1->Lines->AddStrings(Buffer);
RichEdit1->Lines->Add("");
}
delete Buffer;
DragFinish(DropH);
Msg.Result = 0;
break;
}
case CM_RECREATEWND:
DragAcceptFiles(RichEdit1->Handle, true);
break;
default:;
}
OldWindowProc(Msg);
}

Cannot open file on Ubuntu

I'm using Ubuntu 14 and trying to create a script to write files, but I'm getting the 5004 error, every time I try to open a file.
datetime currtime;
bool newcandle;
string terminal_data_path = TerminalInfoString( TERMINAL_DATA_PATH );
string filename = terminal_data_path + "\\MQL4\\Files\\" + "data.csv";
int filehandle;
filehandle = FileOpen( filename, FILE_WRITE | FILE_CSV );
if ( filehandle < 0 ){
Print( "Failed to open the file by the absolute path " );
Print( "Error code ", GetLastError() );
}
else {
Print( "file opened with sucess" );
}
How can I solve this problem on Ubuntu?
UPDATE
I tried to change my file to the following:
string terminal_data_path = TerminalInfoString( TERMINAL_DATA_PATH );
string filename = terminal_data_path + "\\tester\\files\\data.csv";
and just for this
string filename = "\\tester\\files\\data.csv";
and for this
string filename = "\\files\\data.csv";
But I'm still getting error, but this time is 5002 not 5004.
MQL4 Permissions By Design Do Not Allow / Restrict FileIO
There are three directories (with subdirectories) where working files can be placed:
/HISTORY/<current broker> - especially for the FileOpenHistory() function;
/EXPERTS/FILES - common case;
/TESTER/FILES - especially for testing ( ref. during Strategy Tester operations ).
Working with files from other directories is prohibited.
Solution
Adapt your MQL4-code so as to meet this fact and respect pre-Build 762 and post-Build 762 differences ( a "new"-MQL4 file-localisations ).
Update
As posted, your MQL4-code ( whether you share it's updated state or not ) shall better handle exceptions. Have met several suprising artefacts with filenames. Some platform specific, causing no harm in wXP, but failing to operate (the same code) on VPS-hosted wServer2008 VM or a LinuxVM-encapsulated Wine/MT4 instance.
Carefully read MQL4-help documentation and create a few post-mortem tools to step further.
5002
ERR_FILE_WRONG_FILENAME
Wrong file name -------> pre-test + "fuse" the corner cases
5003
ERR_FILE_TOO_LONG_FILENAME
Too long file name
5004 <------ a good sign, we are on the safer side here
ERR_FILE_CANNOT_OPEN
Cannot open file
//-------------------------------------------------------------
// MT4_GUI_postMortem
//-------------------------------------------------------------
void MT4_GUI_postMortem( string aFileNAME = "caller forgot to pass aFileNAME"
){
// SYNTAX
// if ( aFileHANDLE == INVALID_HANDLE ) MT4_GUI_postMortem( filename );
//
int aLastErrorNUM = GetLastError();
Comment( "POST-MORTEM >>> [", aFileNAME, "] Threw error ", aLastErrorNUM );
Print( "POST-MORTEM >>> [", aFileNAME, "] Threw error ", aLastErrorNUM );
return;
}

Delphi and using Teamspeak SDK

I'm trying to use TeamSpeak3 SDK with my delphi my have come accross a few problems, the code compiles and appears to work, most of the code is example code from example projects, that's except the attempt to read the returned data.
1. Do I free the memory correct?
2. Do I read the returned data from the SDK correct or can it be done in a better way?
I have asked a question about this SDK in another thread, but I was obviously too quick to mark the thread as answered. :/
SDK Documentation:
To get a list of all currently visible clients on the specified virtual server:
unsigned int ts3client_getClientList(serverConnectionHandlerID, result);
uint64 serverConnectionHandlerID;
anyID** result;
Parameters
• serverConnectionHandlerID
ID of the server connection handler for which the list of clients is requested.
• result
Address of a variable that receives a NULL-termianted array of client IDs.
Unless an error occurs, the array must be released using ts3client_freeMemory.
Returns ERROR_ok on success, otherwise an error code as defined in public_errors.h. If an error has occured, the result array is uninitialized and must not be released.
A list of all channels on the specified virtual server can be queried with:
unsigned int ts3client_getChannelList(serverConnectionHandlerID, result);
uint64 serverConnectionHandlerID;
uint64** result;
Parameters
• serverConnectionHandlerID
ID of the server connection handler for which the list of channels is requested.
• result
Address of a variable that receives a NULL-termianted array of channel IDs. Unless an error occurs, the array must be released using ts3client_freeMemory.
Returns ERROR_ok on success, otherwise an error code as defined in public_errors.h. If an error has occured, the result array is uninitialized and must not be released.
unsigned int ts3client_getCaptureDeviceList (modeID, result); const char* modeID; char**** result;
Parameters
• modeID
Defines the playback/capture mode to use. For different modes there might be different device lists. Valid modes are returned ts3client_getDefaultPlayBackMode/ts3client_getDefaultCaptureMode and ts3client_getPlaybackModeList/ts3client_getCaptureModeList.
• result
Address of a variable that receives a NULL-terminated array { { char* deviceName, char* deviceID }, { char* deviceName, char* deviceID }, ... , NULL }.
Unless the function returns an error, the elements of the array and the array itself need to be freed using ts3client_freeMemory.
Returns ERROR_ok on success, otherwise an error code as defined in public_errors.h. In case of an error, the result array is uninitialized and must not be released.
Playback and capture devices available for the given mode can be listed, as well as the current operating systems default. The returned device values can be used to initialize the devices.
To query the default playback and capture device, call
To get a list of all available playback and capture devices for the specified mode, call
unsigned int ts3client_getPlaybackDeviceList(modeID, result);
const char* modeID;
char**** result;
unsigned int ts3client_getCaptureDeviceList(modeID, result);
const char* modeID;
char**** result;
Parameters
• modeID
Defines the playback/capture mode to use. For different modes there might be different device lists. Valid modes are returned by
ts3client_getDefaultPlayBackMode / s3client_getDefaultCaptureMode and ts3client_getPlaybackModeList / ts3client_getCaptureModeList.
• result
Address of a variable that receives a NULL-terminated array { { char* deviceName, char* deviceID }, { char* deviceName, char* deviceID }, ... , NULL }.
Unless the function returns an error, the elements of the array and the array itself need to be freed using ts3client_freeMemory.
Returns ERROR_ok on success, otherwise an error code as defined in public_errors.h. In case of an error, the result array is uninitialized and must not be released.
unsigned int ts3client_startConnection(serverConnectionHandlerID,identity,ip,port,nickname,defaultChannelArray,defaultChannelPassword,serverPassword);
uint64 serverConnectionHandlerID; const char* identity; const
char* ip; unsigned int port; const char* nickname; const char**
defaultChannelArray; // This the thingy I dont get const char*
defaultChannelPassword; const char* serverPassword;
Parameters
• serverConnectionHandlerID
Unique identifier for this server connection. Created with ts3client_spawnNewServerConnectionHandler
• identity
The clients identity. This string has to be created by calling ts3client_createIdentity.
Please note an application should create the identity only once, store the string locally and reuse it for future connections.
• ip
Hostname or IP of the TeamSpeak 3 server.
If you pass a hostname instead of an IP, the Client Lib will try to resolve it to an IP, but the function may block for an unusually long period of time while resolving is taking place. If you are relying on the function to return quickly, we recommend to resolve the hostname yourself (e.g. asynchronously) and then call ts3client_startConnection with the IP instead of the hostname.
• port
UDP port of the TeamSpeak 3 server, by default 9987. TeamSpeak 3 uses UDP. Support for TCP might be added in the future.
• nickname
On login, the client attempts to take this nickname on the connected server. Note this is not necessarily the actually assigned nickname, as the server can modifiy the nickname ("gandalf_1" instead the requested "gandalf") or refuse blocked names.
• defaultChannelArray
String array defining the path to a channel on the TeamSpeak 3 server. If the channel exists and the user has sufficient rights and supplies the correct password if required, the channel will be joined on login.
To define the path to a subchannel of arbitrary level, create an array of channel names detailing the position of the default channel (e.g. "grandparent", "parent", "mydefault", ""). The array is terminated with a empty string.
Pass NULL to join the servers default channel.
• defaultChannelPassword
Password for the default channel. Pass an empty string if no password is required or no default channel is specified.
• serverPassword
Password for the server. Pass an empty string if the server does not require a password.
All strings need to be encoded in UTF-8 format
Important
Client Lib functions returning C-strings or arrays dynamically allocate memory which has to be freed by the caller using ts3client_freeMemory. It is important to only access and release the memory if the function returned ERROR_ok.
Should the function return an error, the result variable is uninitialized, so freeing or accessing it
could crash the application.
See the section Calling Client Lib functions for additional notes and examples.
A printable error string for a specific error code can be queried with
unsigned int ts3client_getErrorMessage(errorCode, error);
unsigned int errorCode;
char** error;
Parameters
• errorCode
The error code returned from all Client Lib functions.
• error
Address of a variable that receives the error message string, encoded in UTF-8 format. Unless the return value of the function is not ERROR_ok, the string should be released with ts3client_freeMemory.
Example:
unsigned int error;
anyID myID;
error = ts3client_getClientID(scHandlerID, &myID); /* Calling some Client Lib function */
if(error != ERROR_ok) {
char* errorMsg;
if(ts3client_getErrorMessage(error, &errorMsg) == ERROR_ok)
{ /* Query printable error */
printf("Error querying client ID: %s\n", errorMsg);
ts3client_freeMemory(errorMsg); /* Release memory */
}
}
type
PPanyID = ^PAnyID;
PanyID = ^anyID;
anyID = word;
var
error: longword;
errormsg: PAnsiChar;
procedure TfrmMain.RequestOnlineClients;
var
ids : PanyID;
pids : PanyID;
aid : anyID;
begin
error := ts3client_getClientList(FTSServerHandlerID, #ids);
if (error <> ERROR_ok) then
begin
if (ts3client_getErrorMessage(error, #errormsg) = ERROR_ok) then
begin
LogMsg(Format('Error requesting online clients: %s', [errormsg]));
ts3client_freeMemory(errormsg);
end;
end else
begin
pids := ids;
while (pids^ <> 0) do
begin
aid := pids^;
LogMsg(format('userid %u',[aid, getUserNickNameById(aid)]));
inc(pids);
end;
ts3client_freeMemory(#pids^); // here's potiential problem
end;
end;
procedure TfrmMain.RequestChannels;
var
ids : PUint64;
pids : PUint64;
aid : uint64;
channelname : PAnsiChar;
begin
error := ts3client_getChannelList(FTSServerHandlerID, #ids);
if (error <> ERROR_ok) then
begin
if (ts3client_getErrorMessage(error, #errormsg) = ERROR_ok) then
begin
LogMsg(Format('Error requesting channels: %s', [errormsg]));
ts3client_freeMemory(errormsg);
end;
end else
begin
pids := ids;
while (pids^ <> 0) do
begin
aid := pids^;
LogMsg(format('channelid %u %s',[aid, getChannelNameById(aid)]));
inc(pids);
end;
ts3client_freeMemory(#pids^);
end;
end;
**// Added details 25-11-2014**
char* defaultMode;
if(ts3client_getDefaultPlayBackMode(&defaultMode) == ERROR_ok) {
char*** array;
if(ts3client_getPlaybackDeviceList(defaultMode, &array) == ERROR_ok) {
for(int i=0; array[i] != NULL; ++i) {
printf("Playback device name: %s\n", array[i][0]); /* First element: Device name */
printf("Playback device ID: %s\n", array[i][1]); /* Second element: Device ID */
/* Free element */
ts3client_freeMemory(array[i][0]);
ts3client_freeMemory(array[i][1]);
ts3client_freeMemory(array[i]);
}
ts3client_freeMemory(array); /* Free complete array */
} else {
printf("Error getting playback device list\n");
}
} else {
printf("Error getting default playback mode\n");
}
Example to query all available playback devices:
char* defaultMode;
if(ts3client_getDefaultPlayBackMode(&defaultMode) == ERROR_ok) {
char*** array;
if(ts3client_getPlaybackDeviceList(defaultMode, &array) == ERROR_ok) {
for(int i=0; array[i] != NULL; ++i) {
printf("Playback device name: %s\n", array[i][0]); /* First element: Device name */
printf("Playback device ID: %s\n", array[i][1]); /* Second element: Device ID */
/* Free element */
ts3client_freeMemory(array[i][0]);
ts3client_freeMemory(array[i][1]);
ts3client_freeMemory(array[i]);
}
ts3client_freeMemory(array); /* Free complete array */
} else {
printf("Error getting playback device list\n");
}
} else {
printf("Error getting default playback mode\n");
}
procedure TfrmMain.ConnectServer2;
var
version : PAnsiChar;
DefaultChannelsArr : PPAnsiChar;
begin
if Connected then Exit;
if not ClientInitialized then
InitializeClient;
// Dbl Check if we can connect
if ClientInitialized then
try
// Connect to server on localhost:9987 with nickname "client", no default channel, no default channel password and server password "secret"
// error := ts3client_startConnection(FTSServerHandlerID, identity, '127.0.0.1', 9987, 'Delphi Client', nil, '', 'secret'); // example connection setup
ts3check(ts3client_startConnection(FTSServerHandlerID, PAnsiChar(FSetup.ClientIdentity), PAnsiChar(FSetup.ServerAddress), FSetup.FServerPort, PAnsiChar(FSetup.NickName), nil, '', PAnsiChar(FSetup.ServerPassword)));
{ TODO -oMe -cImportant : Need to check how to convert ansistrings to UTF8 } // UnicodeToUtf8() // AnsiToUtf8()...
// Query and print client lib version
ts3check(ts3client_getClientLibVersion(#version));
LogMsg(Format('Client lib version: %s', [version]));
ts3client_freeMemory(version); // Release dynamically allocated memory
// Do not set connected here, wait for the callback connected state
except
on e: exception do
begin
UnInitializeClient; // clear the hole thing and start over
LogMsg(Format('Error connecting: %s',[e.Message]));
end;
end;
end;
I'd translate ts3client_getClientList like this:
function ts3client_getClientList(serverConnectionHandlerID: UInt64;
out result: PAnyID): Cardinal; cdecl; external '...';
I think that an out parameter is better than a double pointer. It makes the intent clearer.
Then to call the function I'd write it like this:
var
ids: PAnyID;
idarr: TArray<anyID>;
....
ts3check(ts3client_getClientList(serverConnectionHandlerID, ids));
try
idarr := GetIDs(ids);
finally
ts3check(ts3client_freeMemory(ids));
end;
Here, ts3check is a function that raises an exception if it is passed a return value other than ERROR_ok.
function ts3client_getErrorMessage(error: Cardinal;
out errormsg: PAnsiChar): Cardinal; cdecl; external '...';
....
procedure ts3check(error: Cardinal);
var
errormsg: PAnsiChar;
errorstr: string;
begin
if error = ERROR_ok then
exit;
if ts3client_getErrorMessage(error, #errormsg) <> ERROR_ok then
raise Ets3Error.CreateFmt('Error code %d', [error]);
errorstr := UTF8ToUnicodeString(errormsg);
ts3client_freeMemory(errormsg);
raise Ets3Error.CreateFmt('Error code %d (%s)', [error, errorstr]);
end;
And you can implement GetIDs like this:
function GetIDs(const ids: PAnyID): TArray<anyID>;
var
Count: Integer;
p: PAnyID;
begin
Count := 0;
p := ids;
while p^ <> 0 do
begin
inc(Count);
inc(p);
end;
SetLength(Result, Count);
Count := 0;
p := ids;
while p^ <> 0 do
begin
Result[Count] := p^;
inc(Count);
inc(p);
end;
end;
Now, I don't imagine that you really want an array of IDs. You'd probably be happy to process the IDs inline. I don't want to get into how to do that though because that leads me into code of yours that I cannot see. You won't write the code exactly as I have done above, but you can hopefully use the above as a source of ideas.
The main point in all of this is to try to encapsulate as much of the messy boiler plate code as possible. Wrapping the call to ts3client_getErrorMessage makes the higher level code so much easier to read. Use things like OleCheck and Win32Check as inspiration.
One point I would make is that it feels wrong for this code to live inside a form. Normally it is cleaner to keep such code removed from your UI. Make a wrapper to this library that can be consumed by your UI code. Keep that wrapper in a dedicated unit and so hide away the gnarly details.

TStringStream gets corrupted when received using (winsock's) recv?

I'm working on a fairly simple Client/Server application and have some trouble receiving a TStringStream from a client using recv provided by winsock API.
I keep getting this error: 'access violation at 0x00000000: read of address 0x00000000'.
The client only copies text into a TStringStream, gets it's length and sends it to the server. The server then receives the Stream and outputs it's text.
Below some abstract code extracts.
{ the server's part }
inBuf := TStringStream.Create;
{ MAKE THIS SOCKET A PASSIVE ONE }
listen(serversock, LISTENQ);
{ ACCEPT CONNECTION ON serversock FROM cliaddr -> CONNECTED SOCKET = connfd }
connfd := accept(serversock, #cliaddr, #len);
recv(connfd, inLen, sizeof(inLen), 0);
//up to here everything is fine with the strem:
//Size = InLen, Position = 0, all bytes are '0'
rec := recv(connfd, inBuf, inLen, 0);
//rec = inLen, which is fine
//now this: inBuf: FMemory $1, FSize 9 (no matter how long the msg is)
// FPosition 7077987 and FBytes: many many random
DebugOutput(inBuf.DataString); //the error is thrown here
where connfd is the connected socket, servsock is the listening socket, inLen is a cardinal containing the length of inBuf, inBuf is a global TStringStream. rec is a cardinal containing the # of bytes received by recv.
{ the client's send function }
function SSend(sock :TSocket; addr :sockaddr_in; msg :TStringStream) :Integer;
var
len: Cardinal;
begin
len := msg.Size;
send(sock, len, sizeof(len), 0);
msg.Seek(0,0);
send(sock, msg, sizeof(msg), 0);
Result := 0;
end;
and the client's call to SSend:
{ CREATE (OUTPUT)STREAM }
s := TStringStream.Create;
s.WriteString(_input.Text);
//_input is a TMemo with text, let's say, ´hello´
SSend(client, servaddr, s);
//client is a TSocket
Thanks for any help in advance!
p1.e
You are passing into recv a pointer to TStringStream object itself, not to its data buffer. That's why the object gets corrupted. Use Memory property: recv(connfd, inBuf.Memory^, inLen, 0).
The same goes for sending: send data from stream, not the stream object (sizeof(msg) in your SSend returns just size of a pointer).

Node.js is running out of memory on large bit-by-bit file read

I'm attempting to write a bit of JS that will read a file and write it out to a stream. The deal is that the file is extremely large, and so I have to read it bit by bit. It seems that I shouldn't be running out of memory, but I do. Here's the code:
var size = fs.statSync("tmpfile.tmp").size;
var fp = fs.openSync("tmpfile.tmp", "r");
for(var pos = 0; pos < size; pos += 50000){
var buf = new Buffer(50000),
len = fs.readSync(fp, buf, 0, 50000, (function(){
console.log(pos);
return pos;
})());
data_output.write(buf.toString("utf8", 0, len));
delete buf;
}
data_output.end();
For some reason it hits 264900000 and then throws FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory. I'd figure that the data_output.write() call would force it to write the data out to data_output, and then discard it from memory, but I could be wrong. Something is causing the data to stay in memory, and I've no idea what it would be. Any help would be greatly appreciated.
I had a very similar problem. I was reading in a very large csv file with 10M lines, and writing out its json equivalent. I saw in the windows task manager that my process was using > 2GB of memory. Eventually I figured out that the output stream was probably slower than the input stream, and that the outstream was buffering a huge amount of data. I was able to fix this by pausing the instream every 100 writes to the outstream, and waiting for the outstream to empty. This gives time for the outstream to catch up with the instream. I don't think it matters for the sake of this discussion, but I was using 'readline' to process the csv file one line at a time.
I also figured out along the way that if, instead of writing every line to the outstream, I concatenate 100 or so lines together, then write them together, this also improved the memory situation and made for faster operation.
In the end, I found that I could do the file transfer (csv -> json) using just 70M of memory.
Here's a code snippet for my write function:
var write_counter = 0;
var out_string = "";
function myWrite(inStream, outStream, string, finalWrite) {
out_string += string;
write_counter++;
if ((write_counter === 100) || (finalWrite)) {
// pause the instream until the outstream clears
inStream.pause();
outStream.write(out_string, function () {
inStream.resume();
});
write_counter = 0;
out_string = "";
}
}
You should be using pipes, such as:
var fp = fs.createReadStream("tmpfile.tmp");
fp.pipe(data_output);
For more information, check out: http://nodejs.org/docs/v0.5.10/api/streams.html#stream.pipe
EDIT: the problem in your implementation, btw, is that by doing it in chunks like that, the write buffer isn't going to get flushed, and you're going to read in the entire file before writing much of it back out.
According to the documentation, data_output.write(...) will return true if the string has been flushed, and false if it has not (due to the kernel buffer being full). What kind of stream is this?
Also, I'm (fairly) sure this isn't the problem, but: how come you allocate a new Buffer on each loop iteration? Wouldn't it make more sense to initialize buf before the loop?
I don't know how the synchronous file functions are implemented, but have you considered using the asynch ones? That would be more likely to allow garbage collection and i/o flushing to happen. So instead of a for loop, you would trigger the next read in the callback function of the previous read.
Something along these lines (note also that, per other comments, I'm reusing the Buffer):
var buf = new Buffer(50000),
var pos = 0, bytesRead;
function readNextChunk () {
fs.read(fp, buf, 0, 50000, pos,
function(err, bytesRead){
if (err) {
// handle error
}
else {
data_output.write(buf.toString("utf8", 0, bytesRead));
pos += bytesRead;
if (pos<size)
readNextChunk();
}
});
}
readNextChunk();

Resources