I am uploading very large files, which exceed the available memory and thus I am using a FileBuffer as Receiver. Unfortunately, when uploading a large file, it takes very long to save it. Instead, I'd much rather start processing the file while it's still uploading.
So is there a way to receive the upload as a stream? Or could I implement my own Receiver, but then do I need to also implement my entire processing logic in that receiver?
Yes, you can implement your own Receiver; the built-in receivers like FileBuffer and MemoryBuffer are just helpers to cover basic use cases. The Receiver interface is simple:
public interface Receiver extends Serializable {
/**
* Invoked when a new upload arrives.
*
* #param fileName
* the desired filename of the upload, usually as specified by
* the client
* #param mimeType
* the MIME type of the uploaded file
* #return stream to which the uploaded file should be written
*/
OutputStream receiveUpload(String fileName, String mimeType);
}
Your task is to provide the OutputStream for the Upload component to write on; in the case of a FileBuffer, the class is creating a File object and a FileOutputStream. If you want to process the stream yourself, you can either extend FileOutputStream or create a custom OutputStream and implement the write method. A BufferedOutputStream is also a good option to investigate.
Related
This is the code in Java to make the socket call, but I want to know how can I replicate this or something similar in iOS (Swift or Objective-C)
public String MakeSocketRequest() {
DataInputStream inputSt;
DataOutputStream outputSt;
Socket socket = new Socket(InetAddress.getByName("socketurl.io"), 40008);
String jsonStr = "{\"id\":1,\"method\":\"themethod\"}";
inputSt = new DataInputStream(socket.getInputStream());
outputSt = new DataOutputStream(socket.getOutputStream());
PrintWriter pw = new PrintWriter(outputSt);
pw.println(string);
Log.d("PrintWriter", jsonStr);
pw.flush();
BufferedReader bfr = new BufferedReader(new InputStreamReader(inputSt));
JSONObject json = new JSONObject(bfr.readLine());
Log.d("Json", json.toString());
inputSt.close();
outputSt.close();
return json.toString();}
If you want to do it natively without 3rd-party libraries,
then you can use CFStreamCreatePairWithSocketToHost function to create input and output streams (no socket object is needed).
Here's some example code to set this up
And the search shows many more
On iOS you can't write or read the streams immediately, and you have to wait until the socket is connected, and you get a permission to read/write. This is done by implementing NSStreamDelegate.
If you get NSStreamEventHasSpaceAvailable event there, you can write your string to the output stream. You don't need a PrintWriter to just write a string, because it is easy to convert NSString to NSData, and write NSData.
If you get NSStreamEventHasBytesAvailable event, means you can try to read data from the input stream to some buffer (like NSMutableData). There's no builtin BufferedReader with a readLine method, so you will have to buffer the data yourself and detect when a new line character appears there. After that you can cut a part of the buffer until the new line, and convert NSData to NSString (or a JSON object by using NSJSONSerialization).
Note: scheduleInRunLoop calls might look confusing, but they are required to start receiving events via the delegate. It kind of tells the system on which thread you want to receive them.
P.S. I agree with commenters that if you have control over the server code, it's better to use a standard protocol like Socket IO or msgpack instead of inventing your own, because they have better and nicer libraries and wider community support.
I'm converting an MP3 file into raw PCM, and I need to save it as just raw PCM. (Note, am using Java/RoboVM to port to iOS.)
I'm using the coreaudio package, and the relevant part of my code looks like this:
// Define the output PCM format.
AudioStreamBasicDescription outputFormat = new AudioStreamBasicDescription();
outputFormat.setFormat(AudioFormat.LinearPCM);
outputFormat.setFormatFlags(AudioFormatFlags.Canonical);
outputFormat.setBitsPerChannel(16);
outputFormat.setChannelsPerFrame(1);
outputFormat.setFramesPerPacket(1);
outputFormat.setBytesPerFrame(2);
outputFormat.setBytesPerPacket(2);
outputFormat.setSampleRate(22050);
// ...
outputFile = ExtAudioFile.create(outputFileURL, AudioFileType.CAF, outputFormat, null, AudioFileFlags.EraseFile);
I then run through a loop, reading from the MP3 file and writing to the output file.
Upon importing this raw file into Audacity, I notice it always has a spike at the start, indicating that it's not actually a raw PCM file but instead is inside of a wrapper with a header (whether it be WAV or CAF headers, etc).
I understand I can just take the file and strip the header off and get the raw PCM data, but in terms of space/performance of this part of my app, I'd love if I can just keep it simple and save the raw PCM data as-is without a wrapper, but I don't know how to go about doing that.
The issue arises here:
outputFile = ExtAudioFile.create(outputFileURL, AudioFileType.CAF, outputFormat, null, AudioFileFlags.EraseFile);
There aren't many choices for AudioFileType, I've tried WAVE and CAF. Ideally there would be a PCM or RAW option but there's not. Is there a specific AudioFileType I should choose, or do I need to go about this another way?
The extended audio file services framework doesn't support a "raw" PCM format.
For an application to understand a PCM format it needs to know data stuff like:
How many channels are there
Are they interleaved or not
What is the sample rate
Is the data floating point or not
What is the bit depth
etc...
In fact, on iOS and OS X the AudioStreamBasicDescription is a struct which tells you what is required to interpret a PCM stream. For this reason, a "raw PCM" format doesn't really work, it needs at least some metadata. The closest formats to raw PCM are WAV, AIFF and CAF. If these don't serve your purposes you'll have to create a custom file format. But this doesn't need to be difficult.
The extended audio file services APIs are quite configurable. After opening an audio file to read (ExtAudioFileOpenURL) you can set various properties on the ExtAudioFileRef handle.
In your case consider setting kExtAudioFileProperty_ClientDataFormat. This property controls the format of the PCM data read from the file. As ExtAudioFileRead decodes the input file, it will convert the data it sends back to the format you specify. There are some limitations to this method. IIRC, it does not support doing sample rate conversion and things like that.
As you read the properly decoded data, you can then use something like NSOutputStream to write the "raw PCM" format of your choice directly to a file with no metadata at all.
I want to update an existing Excel file using SpreadsheetGear.
I load the Excel file using:
_activeWorkbook = Factory.GetWorkbook(#"./excel/test.xls");
workbookView1.ActiveWorkbook = _activeWorkbook;
but when I want to save it:
private void menuSave_Click(object sender, EventArgs e)
{
workbookView1.GetLock();
try
{
_activeWorkbook.Save();
}
finally
{
workbookView1.ReleaseLock();
}
}
I get this error: System.IO.IOException: The process cannot access the file 'C:...\bin\Debug\excel\test.xls' because it is being used by another process.
As the exception indicates, some other process (or possibly your same application, if you utilize multiple threads) has a lock on your file and so SpreadsheetGear cannot save it back to disk. Any number of other external processes could be the culprit, such as anti-virus software scanning your file or having it open in Excel itself when you try to save it. There is no way to determine the exact cause with just the information provided above.
What I can tell you is that SpreadsheetGear does not keep any file streams open after reading and writing files. The file stream and lock on the file is only open for as long as it takes to read/write its contents, which is usually very short if the file is small. Put another way, #".excel/test.xls" should be writable immediately after your Factory.GetWorkbook(...) line executes.
I would like to add three "parts" to an NSInputStream: an NSString, an output from another stream and then another NSString. The idea is the following:
The first and last NSStrings represent the beginning and end of a SOAP request while the output from the stream is a result of loading a very large file and encoding it as Base64 string. So, in the end I would have the final NSInputStream hold the whole SOAP request like this:
< soap beginning > < Base64 encoded data > < soap ending >
The reason I want the whole request to be held in NSInputStream is two-fold:
I don't what to load the very large data file into memory
I think that this is the only way to enforce sending the final request in HTTP 1.1 chunks (which I need because otherwise, if the request becomes too big, the server won't accept it). So, I know that doing this:
NSInputStream *dataStream = ....;
[request setHTTPBodyStream:dataStream];
ensures that the request will be sent as HTTP 1.1 chunks and not as one huge raw SOAP request.
So, I wonder how this can be achieved - namely, how do I "enqueue" things into an NSInputStream? Can it be even done? Is there an alternative way?
Just for reference, in Java this can be done as follows
Vector<InputStream> streamVec = new Vector<InputStream>();
BufferedInputStream fStream = new BufferedInputStream(fileData.getInputStream());
Base64InputStream b64stream = new Base64InputStream(fStream, true);
String[] SOAPBody = GenerateSOAPBody(fileInfo).split("CUT_HERE");
streamVec.add(new ByteArrayInputStream(SOAPBody[0].getBytes()));
streamVec.add(b64stream);
streamVec.add(new ByteArrayInputStream(SOAPBody[1].getBytes()));
SequenceInputStream seqStream = new SequenceInputStream(streamVec.elements());
because Java has these objects available, but NSStreams in objective-c look like very low level objects and are very hard to work with.
Note: I completely re-wrote the original question as I asked it 2 days ago, since I think the new edit explains more clearly what the problem is. I hope it would help it be easier comprehended and maybe answered
UPDATE 2
Here is what I've been able to achieve so far: Instead of trying to enqueue into a stream, I am using a temp file to first write the < soap beginning >, then I set up an input stream to read from the file in chunks, encode each chunk as a Base64 string and write this to the same temp file, finally, when my stream closes, I write the < soap ending > to the temp file. Then I set up another input stream with the contents of this file which I pass to the NSMutableURLRequest:
NSMutableURLRequest* request = [NSMutableURLRequest requestWithURL:url];
...
NSInputStream *dataStream = [NSInputStream inputStreamWithFileAtPath:_tempFilePath];
[request setHTTPBodyStream:dataStream];
This ensures HTTP 1.1 chunked transfer of the contents of the file. After the connection finishes, delete the temp file.
This seems to work fine but of course this is an annoying work-about. I don't want to be writing to a temp file when it all could have been handled by streams (ideally.) If anybody still has better suggestions, let me know :)
UPDATE 3
OK, another update is in order. While my writing to file seems to work, I am now hitting an unexpected issue with some of my requests failing to upload to the server. Specifically, everything is going according to the plan, I am reading the contents of the temp file into a stream and set HTTP body of my request to be this stream and it starts transmitting the HTTP 1.1 chunks as I want it to - but for some reason some packets get dropped and the final request - this is my guess - gets malformed and thus fails. I think the issue of dropped packets is random, because I observe it on larger requests - that is, the issue just has more chance to show up - while my smaller requests usually go thru just fine. This is of course a separate issue from the original in this question. If anybody has a good idea what might be causing this, I asked about the problem here: Packets dropped during chunked HTTP 1.1 request sent by NSURLConnection
Your solution is an ok option, but you can do it with a stream. It means subclassing NSInputStream, and that isn't trivial because there are a bunch of methods you need to implement.
Basically your subclass would initially return the header bytes, then it would return bytes from the 'internal' stream to the file content, then when that's used up it returns the footer bytes. It means maintaining a record of how big the header and footer are and how much has been processed so far, but that isn't a big issue.
There's an example of creating a subclass here which shows the tricky hidden methods you need to implement to get the stream subclass to work properly without throwing exceptions.
I use this code to play decrypted audio on BlackBerry on the fly (for the sake of simplicity, I use TEA)
public void play(String path){
try {
FileConnection fc = (FileConnection) Connector.open(path, Connector.READ);
InputStream is = fc.openInputStream();
byte[] rawData = IOUtilities.streamToBytes(is);
processEncryptedAudio(rawData);
is.close();
fc.close();
}
catch (IOException ioex){
}
}
// TEA code is taken from http://www.winterwell.com/software/TEA.php
private void processEncryptedAudio(byte[] data) throws IOException {
TEA tea = new TEA("ABCDE ABCDE ABC A ABCDEF".getBytes());
byte[] decrypted_data = tea.decrypt(data);
ByteArrayInputStream stream = new ByteArrayInputStream(decrypted_data);
ByteArrayInputStreamDataSource source = new ByteArrayInputStreamDataSource(stream, "audio/mpeg");
try {
player = Manager.createPlayer(source);
player.start();
}
catch (MediaException me){
Dialog.alert("MediaException: "+me.getMessage());
}
}
The problem is decryption takes quite long time to finish. For example: on simulator, decrypting a 9 MB audio takes around 5 secs, but on BlackBerry Torch 9860 it takes more than 20 secs.
Is there any way to improve this? Actually the whole file doesn't neet to be encrypted, as long as it is obscured/cannot be played directly.
You could try switching from TEA to RC4, which is also very simple to implement and quite possibly faster.
Also, it looks like you're doing some unnecessary data copying: it would be slightly more efficient to make your decrypt() method modify the input byte array directly. This may require changing the calling code to skip some number of bytes at the beginning and/or end of the decrypted data, but that shouldn't be too hard. (The ByteArrayInputStream constructor can take optional offset and length arguments.)
If you want to get really fancy, you could try writing your own custom InputStream subclass that does the decryption "on the fly" while the audio is playing. If you use a block cipher in CTR, CFB or CBC mode (or ECB, but that's not secure), you can even make the stream seekable. If you want to be even fancier, make it a wrapper around the original InputStream so that you can do the loading, decryption and playing all at the same time.
Another option might be to use the RIM Crypto API, whose cipher implementations might be more efficient (possibly implemented in optimized native code) than your own. The Crypto API also already provides the DecryptorInputStream class which works in the manner I described above.
One possible down side is that the Crypto API seems to be only available to signed apps.