just recently I was thinking and wondered, how does Dropbox upload my files to its S3 storage and how might that one be organized?
Let's just completely forget about the sync aspect for a second and scale the problem down to one S3 bucket.
Say, in that bucket's root directory you have lots of folders, each belonging to an arbitrary user.
Now if that user wants to upload a file to his folder... how does that happen internally? I mean, Dropbox can't just store the Amazon S3 access credentials/keys hard-coded into the application (be it on ios or windows) as it might get reverse-engineered and thus exposed.
Any thoughts on this?
Thanks!
Some guys from EADS did reengineering on Dropbox, the presentation slides are available for download: A CRITICAL ANALYSIS OF
DROPBOX SOFTWARE SECURITY
In the same way websites don't allow users to directly access their databases but rather provide interfaces that can control permissions and handle authentication, I'm sure Dropbox has some kind of application that the client on your computer interacts with. Their server daemon will have permissions to write to the disk, but your computer has to go through it (and it's security procedures) before anything your computer sends is written.
Related
I would like to use some publicly available data from a government website as a source of data in an iOS app. But I am not sure what is the best / most polite / scalable way have a large number of users request data from this website with the least impact on their servers and best reliability for me.
It is 1-50kb of static XML with a fixed URL scheme
It updates with a new XML once a day
New users would need to download past data
It has a Last-Modified header but no caching headers
It does not use compression or a CDN
It's a government website, so if someone even replies to my email I doubt they are going to change how they host it for me...
I'm thinking I could run a script on a server to download this file once a day and re-host for my app however my heart desires. But I don't currently run a server which I could use for this and it seems like a lot just for this. My knowledge of web development is not great, so am I perhaps missing something obvious and I just don't know what search terms I should be using to find the answer.
Can I point a CDN at this static data somehow and use that?
Is there something in CloudKit I could use?
Should I run a script on AWS somehow to do the rehosting without needing a full server?
Should I just not worry about it and access the data directly??
You can use the AWS S3 service (Simple Storage Service).
The flow is somewhat like this:
If the file doesn't exist on S3 yet, or, if the creation date of the file on S3 is yesterday, the iOS app downloads the XML from the gov site and stores it in S3.
If the file exists on S3 and is up to date, download it from S3.
After that, the data can be presented by the app without overloading to the site.
I think the best way for you is to create an intermediary database where you can store your data in a secure manner.
Create a pipeline that does some data transformation and store in you newly created database.
Create an api with pagination and you desired filters
Also make sure you are not violating any data policies in the process.
I hope this helps.
I'm searching a way to make a social network where the data (images, videos...) is stored on the user's computer (or client-side) and a remote server is used to access it by serving the functionnality and the design. The basic thing would be for each user to have a profile and to be able to browse other's profiles.
Note : It's not a problem if the client-side computer shuts down its machine. The user's profile would be just designed as offline. And when the user is connected, others would be able to browse his – her profile.
I searched a lot and I came up with different ideas :
→ store data with indexeddb and then stream it with webrtc. The problem : webrtc is not really made for showing a simple profile page to be browsed by multiple users.
→ make a desktop app to access local files and then try to make a communication with the server to deliver the app (with something like electron ?). The problem is I don't know if electron is well suited for this kind of project.
Any ideas, questions, suggestions, remarks or anything that helps would be greatly welcomed.
We have an application which i a Cloud Screen service/application. The purpose of the application is to store images and videos locally but also on the server. So if a computer connects to the network, it'll download the current content (json, images, videos) so it can display these even without internet because it's stored locally.
The client was made in Electron and the management application was .NET Core.
If you're thinking about creating a desktop application, then yes, it is possible, but in order to achieve this, you have to store on the server as well.
Let's assume that someone wants to check your profile. It has to load your datas which has to be stored on the server because they can't access your local files and computer. But you can store your files locally to boost up the speed.
I would only recommend this to store images/videos and bigger files, but you have to specify your requirements because in this case, every time you bump into a new content, you have to download it (if you want to store locally so it'll load faster the stored files).
The second option is to store these in the client. But be aware, desktop application's size can increase easily with lots of images and videos. If you store these in the client, you only have to store the URLs in case of dynamic datas (static URLs can be stored as static data).
I hope I could help, if you have any more questions, feel free to ask.
I'm building a streaming app similar to pandora. However right now I'm storing all my files on http and accessing them with urls. Is there an alernative to this because all the files are in the public html folder? For example how does apps like pandora or spotify pull files off their servers. I'm new to web severs and not sure where to ask this question. I have a centos server on vps hosting with apache, MySQL, http, ftp.
You just need to provide the content as a bit stream rather than a file download. The source of that data to send as a stream can be stored as binary data in a BLOB column in a database or as a regular file on a non-public part of the file system. It really does not mater which one you use.
Storing them in the database gives your app a bit easier access and makes the app more portable since it is not restricted the file system level permissions.
The fact you currently have the files in a public folder is not really that critical of an issue since you are making them available for download. You would just need to make sure you have an authentication requirement if you want to restrict who can access them.
Im struggling to find an answer to this. I have a website that is deployed in a shared hosting environment. I want to allow people to upload files to my azure blob storage account.
I have this working locally, using the storage emulator, however when I publish the site I get a Security Exception.
Is this actually possible under a shared hosting envrionment ?
Cheers
A bit more detail would help, in understanding how these uploads are taking place. That said, I'll make the assumption that people are uploading directly to Blob Storage, and not through your Website (or Web Service).
To allow direct uploads, you need to provide either a public blob or container (which everyone in the world can see), or create a temporary Shared Access Signature (SAS) on a specific blob or container, that grants access for a short time window.
If your app is Silverlight, then you are probably running into a cross-domain issue (and you'll need to correct that with an access policy).
If you provide more details around the way uploads are being sent, as well as the client and server technology, I can edit my answer to be more specific.
Imagine the following use case:
You have a basecamp style application hosting files with S3. Accounts all have their own files, but stored on S3.
How, therefore, would a developer go about securing files so users of account 1, couldn't somehow get to files of account 2?
We're talking Rails if that's a help.
S3 supports signed time expiring URLs that mean you can furnish a user with a URL that effectively lets only people with that link view the file, and only within a certain time period from issue.
http://www.miracletutorials.com/s3-amazon-expiring-urls/
If you want to restrict control of those remote resources you could proxy the files through your app. For something like S3 this may defeat the purpose of what you are trying to do, but it would still allow you to keep the data with amazon and restrict access.
You should be careful with an approach like this as it could cause your ruby thread to block while it is proxying the file, which could become a real problem with the application.
Serve the files using an EC2 Instance
If you set your S3 bucket to private, then start up an EC2 instance, you could serve your files on S3 via EC2, using the EC2 instance to verify permissions based on your application's rules. Because there is no charge for EC2 to transfer to/from S3 (within the same region), you don't have to double up your bandwidth consumption costs at Amazon.
I haven't tackled this exact issue. But that doesn't stop me from having an opinion :)
Check out cancan:
http://github.com/ryanb/cancan
http://railscasts.com/episodes/192-authorization-with-cancan
It allows custom authorization schemes, without too much hassle.