I'm trying to build an iPad app to download and display documents (pdf, ppt, doc, etc.) from a web server.
Currently it does this by parsing the HTML structure (using hpple) on the server.
For example, the files are held at:
Http://myserver.com/myFolders/myFiles/
The app goes to this location and traverses the tree, using an X-Path query, e.g.
"/html/body/ul/li/a"
It then downloads whatever documents it finds to the iPad for display.
So far this works quite well but the server is publicly accessable.
My question is, how would I go about doing something similar with a secure server?
e.g. is it possible to password protect the server, connect to it with username/password from the iPad and use the same system?
In the end I decided not to parse the HTML as there seemed to be no straightforward way to do so. Instead the documents are held on an ASP.Net server with authentication required for access.
It would've been nice to know how to do so by traversing HTML but no biggie.
Related
Background
I have a web application where users may upload a wide variety of files (e.g. jpg, png, r, csv, epub, pdf, docx, nb, tex, etc). Currently, we whitelist exactly which files types are a user may upload. This limitation is sometimes annoying for users (i.e. because they must zip disallowed files, then upload the zip) and for us (i.e. because users write support asking for additional file types to be whitelisted).
Ideal Solution
Ideally, I'd love to whitelist files more aggressively. Specifically, I'd would like to (1) figure out which file types may be trusted and (2) whitelist them all. Having a larger whitelist would be more convienient for users and it reduce the number of support tickets (if only very slightly).
What I Know
I've done a few hours of research and have identified common problems (e.g. path traversal, placing assets in root directory, htaccess vulnerabilities, failure to validate mime type, etc). While this research has been interesting, my understanding is that many of these issues are moot (or considerably mitigated) if your assets are stored on Amazon S3 (or a similar cloud storage service) – which is how most modern web application manage user-uploaded files.
Hasn't this question already been asked a zillion times?!
Please don't mistake this as a general "What are the security risks of user-uploaded content?" question. There are already many questions like that and I don't want to rehash that discussion here.
More specifically, my question is, "What risks, if any, exist given a conventional / modern web application setup?" In other words, I don't care about some old PHP app or vulnerabilities related to IE6. What should I be worried about assuming files are stored in a cloud service like AmazonS3?
Context about infrastructure / architecture
So... To answer that, you'll probably need more context about my setup. That said, I suspect this is a relatively common setup and therefore hope the answers will be broadly useful to anyone writing a modern web application.
My stack
Ruby on Rails application, hosted on Heroku
Users may upload a variety of files (via Paperclip)
Server validates both mime type and extension (against a whitelist)
Files are stored on Amazon S3 (with varying ACL permissions)
When a user uploads a file...
I upload the file directly on AS3 in a tmp folder (hasn't touched my server yet)
My server then downloads the file from the tmp folder on AS3.
Paperclip runs validations and executes any processing (e.g. cutting thumbnails of images)
Finally, Paperclip places the file(s) back on AS3 in their new, permanent location.
When a user downloads a file...
User clicks to download a file which sends a request to my API (e.g. /api/atricle/123/download)
Internally, my API reads the file from AS3 and then serves it to the user (as content type attachment)
Thus the file does briefly pass through my server (i.e. not merely a redirect)
From the user's perspective, the file is served from my API (i.e. the user has no idea the file live on AS3)
Questions
Given this setup, is it safe to whitelist a wide range of file types?
Are there some types of files that are always best avoided (e.g. JS files)?
Are there any glaring flaws in my setup? I suspect not, but if so, please alert me!
I am trying to create an HTTP Server inside my iOS application, to develop something like Xender application. Right now I Succeed to setup HTTP Server inside my Application and hosting any HTML file there, that can be loaded on another Device/System using IP and Port.
But, I want to Link that HTML to my application database to populate data on that HTML file, followed by making it dynamic so that It can be opened from another device or system.
Ultimately, I need to Query on SQLITE database of application from
HTML file, Is there any way to do such thing?
Can I connect SQLITE to frontend of HTML? In case of Web apps these things can be done using any server side scripting languages like PHP, by connecting with Databases like MySQL. But, Here My case is HTML and Sqlite.
EDIT
I found Is it possible to access an SQLite database from JavaScript? . but this is all about Client side local storage, but I think in my case its on Server side SQLITE.
You have to create template HTML files and provide a set of variables for it. Then, when the file is requested in your server, you load it into memory.
Now you do some RegEx magic to get the query parameters, do your SQL stuff and then replace the corresponding variables in your HTML string and finally serve it to the client.
Your would need to define your own non-logical "scripting" language that is able to tell your application what data is requested and where to output possibly returned data.
I think this is quite hard work and you should possibly try to find a better solution that is probably already done by others.
EDIT
You could use Node.js and this interpreter but it's not maintained anymore. But their might be similar projects.
EDIT II
I've found the neu.Node, which sounds quite promising. They haven't done anything in 4 months, but they seem to be well organized and documented.
I have a web application which allows users to upload files and share them with other people across the internet. Anyone who has access can download the files, but if the uploader doesn't specifically share the file with someone else, that person can't download the files.
Since the user permissions are controlled by rails, each time someone tries to download a file it sent to the user from a rails process. This is a serious bottle neck - rails is needed for the file upload and permissions but it shouldn't be in the way taking up memory just for others to download files.
I would like to split the application on different servers for the frontend, database and file server. If the user does to my site, they should have the ability to download the file directly from something like my-fileserver.domain.com/file/38183 instead of running it through rails.
What is the best option for this? I would like to control file access at the database level, not the file system - but I don't want rails taking up all of the memory on my system for such a simple process. Any ideas?
Edit:
One thing I may be able to do is load a list of files/permissions from mysql into a node.js app and give access rights to the file server as a true/false response based on what the file server sends in. This still requires the file server to run a web server, however.
May be You could generator a rand url for file, and control by center system .
My iPhone app relies on a vendor's XML feed to provide data. But that feed is not locked down. The vendor could change the format of the XML at any time, although so far they've promised not to.
Since I might want to tell my app to use a different URL for its data source, I'd like to set up a single "Command Central" Web page, on my own server, to direct the app to the correct data source.
In other words, each time my app starts, in the background and unseen by the user, it would visit "http://www.myserver.com/iphoneapp_data_sources.xml" to retrieve the URL for retrieving data from my vendor. That way, if my vendor suddenly changes the exact URL or the XML feed that the app needs, I can update that Web page and ensure that all installations of the app are using the correct XML feed.
Does anyone have any advice or examples showing this kind of approach? It seems as if this must be a common problem, but so far I haven't found a well-established design pattern that fits it.
Instead of connecting your iPhone app directly to the vendor's XML feed, you could use a page on your own server as a proxy.
The iPhone app connects with http://www.yourserver.com/proxyxmlfeed.xml, which redirects to the correct vendor URL, http://www.vendorsserver.com/realxmlfeed.xml. This way, if for some reason the real XML feed URL changes, you only have to change the URL the proxyxmlfeed.xml file redirects to, and you're done!
I'm trying to log download statistics for .pdfs and .zips (5-25MB) in a rails app that I'm currently developing and I just hit a brick wall; I found out our shared hosting provider doesn't support mod_xsendfile. The sources I've read state that without this, multiple downloads could potentially cause a DoS issue—something I'm definitely trying to avoid. I'm wondering if there are any alternatives to this method of serving files through rails?
Well, how sensitive are the files you're storing?
If you hosted these files somewhere under your app's /public directory, you could just do a meta tag or javascript redirect to the public-facing URL of these files after your users hit some sort of controller action that will update your download statistics.
In this case, your users would probably need to get one of those "Your download should commence in a few moments" pages before the browser would start the file download.
Under this scenario, your Rails application won't be streaming the file out, your web server will, which will give you the same effect as xsendfile. On the other hand, this won't work very well if you need to control access to those downloadable files.