I am currently putting together a rails-based web application which will only serve and receive data via json and xml. However, some requirements contain the ability to upload binary data (images).
Now to my understanding JSON is not entirely meant for that... but how do you in general tackle the problem of receiving binary files/data over those two entrypoints to your application?
I suggest encoding the binary data in something like base64. This would make it safe to use in XML or JSON format.
http://en.wikipedia.org/wiki/Base64
maybe you could have a look on Base64 algorithm.
This is used to "transform" everything to ascii char.
You can code and decode it. It's used for webservices, or even on dotnet Serialization.
Hope this helps a little.
Edit: I saw "new post", while posting, someone was faster.Rails base64
If you are using Rails and json and xml than you are using HTTP. "POST" is a part of HTTP and is the best way to transform binary data. Base64 is a very inefficient way of doing this.
If your server is sending data, I would recommend putting a path to the file on the server in the XML or JSON. That way your server doesn't have to base64 encode the data and your client, which already supports HTTP GET, can pull down the data without decoding it. (GET /path/to/file)
For sending files, have your server and/or client generate a unique file name and use a two step process; the client will send the xml or json message with fileToBeUploaded: "name of file.ext" and after sending the message will POST the data with the aforementioned filename. Again, client and server won't have to encode and decode the data. This can be done with one request using a multi-part request.
Base64 is easy but will quickly chew up CPU and/or memory depending on the size of the data and frequency of requests. On the server-side, it's also not an operation which is cached whereas the operation of your web server reading the file from disk is.
If your images are not too large, putting them in the database with a RoR :binary type makes a lot of sense. If you have database replicas, the images get copied for free to the other sites, there's no concern about orphaned or widowed images, and the atomic transaction issues become far simpler.
On the other hand, Nessence is right that Base64, as with any encoding layer, does add network, memory and CPU load to the transactions. If network bandwidth is your top issue, make sure your web service accepts and offers deflate/gzip compressed connections. This will reduce the cost of the Base64 data on the network layer, albeit at the cost of even more memory and CPU load.
These are architectural issues that should be discussed with your team and/or client.
Finally, let me give you a heads up about RoR's XML REST support. The Rails :binary database type will become <object type="binary" encoding="base64">...</object> XML objects when you render to XML using code like this from the default scaffolding:
def show
#myobject = MyObject.find(:id)
respond_to do |format|
format.xml { render => #myobject }
end
end
This works great for GET operations, and the PUT and POST operations are about as easy to write. The catch is the Rails PUT and POST operations don't accept the same tags. This is because the from_xml code does not interpret the type="binary" tag, but instead looks for type="binaryBase64". There is a bug with a patch at the Rails lighthouse site to correct this.
Related
I was playing around with GZIP compression recently and the way I understand the following:
Client requests some files or data from a Web Server. Client also sends a header that says "Accept-Encoding,gzip"
Web Server retrieves the files or data, compresses them, and sends them back GZIP compressed to the client. The Web Server also sends a header saying "Content-Encoded,gzip" to note to the Client that the data is compressed.
The Client then de-compresses the data/files and loads them for the user.
I understand that this is common practice, and it makes a ton of sense when you need to load a page that requires a ton of HTML, CSS, and JavaScript, which can be relatively large, and add to your browser's loading time.
However, I was trying to look further into this and why is it not common to GZIP compress a request body when doing a POST call? Is it because usually request bodies are small so the time it takes to decompress the file on the web server is longer than it takes to simply send the request? Is there some sort of document or reference I can have about this?
Thanks!
It's uncommon because in a client - server relationship, the server sends all the data to the client, and as you mentioned, the data coming from the client tends to be small and so compression rarely brings any performance gains.
In a REST API, I would say that big request payloads were common, but apparently Spring Framework, known for their REST tools, disagree - they explicitly say in their docs here that you can set the servlet container to do response compression, with no mention of request compression. As Spring Framework's mode of operation is to provide functionality that they think lots of people will use, they obviously didn't feel it worthwhile to provide a ServletFilter implementation that we users could employ to read compressed request bodies.
It would be interesting to trawl the user mailing lists of tomcat, struts, jackson, gson etc for similar discussions.
If you want to write your own decompression filter, try reading this: How to decode Gzip compressed request body in Spring MVC
Alternatively, put your servlet container behind a web server that offers more functionality. People obviously do need request compression enough that web servers such as Apache offer it - this SO answer summarises it well already: HTTP request compression - you'll find the reference to the HTTP spec there too.
Very old question but I decided to resurrect it because it was my first google result and I feel the currently only answer is incomplete.
HTTP request compression is uncommon because the client can't be sure the server supports it.
When the server sends a response, it can use the Accept-Encoding header from the client's request to see if the client would understand a gzipped response.
When the client sends a request, it can be the first HTTP communication so there is nothing to tell the client that the server would understand a gzipped request. The client can still do so, but it's a gamble.
Although very few modern http servers would not know gzip, the configuration to apply it to request bodies is still very uncommon. At least on nginx, it looks like custom Lua scripting is required to get it working.
Don't do it, for no other reason than security. Firewalls have a hard or impossible time dealing with compressed input data.
I need to send to the customers a raw emails via my rails app.
When they click a link, a new page must open and they need to be able to see the source code of an email. I have a lot of cases where there are emails really big (even 40/50 mb), and it takes a lot of time to server to send it.
E.G.
I have an email with 3 attachments, the total size is 30mb. My
controller method it takes 700 ms to process it and to retrieve the
raw source from imap server, but in the broswer, it takes up to 5
seconds. (2.5 to the first byte, 2.5 to download it).
Right now I just send the string with the render method. Is there a better way? where I am losing all that time?
To be more clear:
With the word 'send' I mean when the server has to 'send' the source code to the browser so the suer can visualize it
What about storing attachments on your server and include in email only links to those attachments? This way your emails will be blazingly fast and clients may download attachments separately. If you don't want to store files on your server, you may use Amazon S3 or some other cloud storage (there are many in these days).
To ease file uploading, I'd recommend you carrierwave library, Amazon S3 integration goes from the box.
I would suggest you to use Delayed Job: https://github.com/collectiveidea/delayed_job or Sidkiq: https://github.com/mperham/sidekiq
When the job is in processing state, you can mark that email as Sending.... and once the background job is completed you can mark it as Sent
Hope this helps
I think you could benefit from looking into solutions for HTTP streaming, which would allow you to start sending data while still processing the request.
This is difficult with Rack based servers, so Rails might have some issues with this approach.
Another approach is to try and split the raw source into chunks and request each chunk using Ajax.
This will allow your app to be more responsive and offer a better user experience. This is also known as a perceived performance approach, since the user experiences the app as more responsive even if it takes the same amount of time to load.
If I were trying to resolve the issue, I would look into an Ajax solution that would allow me to leverage IMAP's partial fetch feature, referenced in it's RFC.
It's possible to write a simple server side API that fetches a part of an email or returns a signal when there is no more to download and than use Javascript to request the data from the server until the 'no more data' return value is received.
This would allow you to display the downloaded data as it's being received.
My company has a Phonegap application that's running into errors with UIWebView; it's hitting the 10MB memory cap. We're using .NET MVC4 as our backend.
This is due to a very large JSON response from a sync call. The JSON string (approximately 12 megs of data) is saved to memory, which causes a memory exception. Due to the way our application works, we have to send this data.
I'm considering using Oboe.JS to stream in the JSON response (to avoid the allocation of the full JSON object). I think that this may fix the issue on the frontend, but I'd like to stream the JSON response from the backend. I know that there are chunking methods, but I'd prefer not to use that option.
Currently, our controller method produces an object, serializes it to JSON, and sends the entire response. I would like to open a one-way HTTP stream, if possible, and asynchronously write JSON data to this stream (which Oboe would parse on the client side).
Are there any technologies that fit my use case, or can someone point me to some methods I may use to accomplish this? I don't need a two-way conduit - I just want to write the JSON to the HTTPstream as objects are produced.
Thank you.
I am just reading up on OData from here.
http://msopentech.com/odataorg/introduction/
Sorry, I am getting a bit impatient.
I just have a simple question for now before I go through the rest of the material. Which of the two options describe OData?
I understand it provides a protocol (much like SOAP or XML/Json over HTTP or XML-RPC) to transfer data from services over the web to clients. What I am intrigued by is that it also helps query that data, which is a great problem to solve as it help reduce payloads that you usually encounter when querying large data sets with XML/SOAP web services or other means (XML over Http, Json over Http, RPC responses, you name it).
Option A
Does oData get all the data to the client, use some client-based storage (like HTML 5 local storage for desktop browsers) to store it, and then query the data on the client using an in-process API?
Or
Option B
Does it provide an XML-based syntax for translation Linq like expressions and getting only the relevant result sets (filtered, ordered, whatever else) stuff from the server?
It's funny how when you type your thoughts, you end up solving your own problems. I think just typing the question has given me the answer. Option A sounds preposterous for so many reasons:
1) If it's a data-centric protocol, it has to not care about what type of client or consumer will want the data, so it cannot have any affinity to client or the capabilities (caching on client side) of the client.
2) It is a data-centric protocol and hence does not prescribe how data must be read or offer any tools on the client or server sides. It merely prescribes a data format, I would imagine.
It has to be Option B. Still, I just want a confirmation or correction.
Yes, it is Option B.
You could obviously write a terrible implementation of a client that would download ALL the data and then filter and show data based on client-side logic. But that would be rather silly.
The way you "write" your queries is quite well detailed in OData.org's "URL Conventions" page, typically something along the lines of: http://someserver/odata.svc/Customers(Location eq 'New York')
In one web service written in Rails, I would like to answer with a file along with additional information.
For this, I consider respond with multipart data. How can I send a multipart response with a file and json?
If there is a better way to do this, please let me know. Note that is not possible add the extra data in the file I'm sending.
Extra points for the face of the problem, that is send a file and data at same time. I already accomplished that by doing a multipart request, but if is there a better way to do this, I would like to know.
I don't know exactly what kind of front end you are using and what your browser compatibility requirements are, or you need the webservice for integration with other apps only, but assuming you are communicating with server over ajax and your app is running in modern browser (or you are allowed to use flash plugin), you can return file contents as base64 encoded string as a part of json response. So in rails controller you would have something like this:
render json: {success: true, file: {name: 'invoice.pdf', data: Base64.encode64(#file.read), extra_info: 'some info'}}
on client side you can process that json, get all the metadata you need from it and also let user save the file to their computer. For that you can use flash plugin or native api implementation
This way you can return couple files with any metadata you want with them to user and on client side user can save files if needed. Also, same webservice can be consumed by other applications as long as they can parse json.
try using gem 'curb' or 'rest-client' gem.
https://github.com/rest-client/rest-client
and
https://github.com/taf2/curb
I'm sure you have done some googling already, have you seen this already? It seems like there is a Gem for what you are trying to accomplish, the Gem however seems to be pretty old.