At some point a dedicated iot device and app may be created but I'm working with an app on an iPhone for now that doesn't address the requirements but still helpful.
The app can stream it's data via POST. I have a php file set up that captures the data and writes it out to a csv file.
The data is a time series with several columns of data that is sent as a POST at every second. It's about 10 minutes in total time.
Instead of writing to a csv, the data needs to be persisted to a database
What I'm unsure about...
Since this is just testing a proof of concept, it may not be an issue till later but can the high frequency of new connections inserts be expensive? Assuming that each for POST a new connection is needed. For now I have no way of authenticating the device so I'm assuming I can use a local account for all known devices.
Is there a better way of handling the data than running a web server with a php script that grabs the data? I was thinking of Kafka + a connector for a database to persist the data but I have no way of configuring the mobile app to know what it needs to do to send data to the server. Communication is not both ways. Otherwise, my experience with POST requests are the typical web form inputs
Anyone able to give some guidance?
I am working on an app that collects user data including photos. It's mandated that this app should work in offline mode - meaning that the user can complete surveys and take photos without an internet connection and that data should sync back to a remote database. How is this generally handled? Do I create a local database with Core Data and write an additional layer to manage saving/reading from a server? Are there any frameworks that help facilitate that syncing?
I have also been looking into backend services such as Firebase that include iOS SDKs that appear to handle a lot of the heavy lifting of offline support, but it does not appear to support offline syncing of image files through the Firebase Storage SDK.
Can anyone recommend the least painful way to handle this?
Couchbase Mobile / Couchbase Lite is probably the best solution I've come across so far.
It allows offline data storage including binary data, and online syncing with a CouchDB compatible server. It works best with their Couchbase Server / Sync Gateway combination, but if you don't need to use filtered replication or 'channels' (e.g. for syncing data specific to a single user with a shared database), you can use Cloudant which saves you having to set up your own server.
Its also available across most platforms.
Generally for images it is best to use NSFileManager and save your images in either the documents directory or the caches directory depending on the types of images you are storing. Core Data or Firebase are databases that are more qualified for data than images although they do support arbitrary data storage.
You can also try SDWebImage which has a lot of features around loading and storing images.
This question already has answers here:
Closed 13 years ago.
Possible Duplicate:
storing uploaded photos and documents - filesystem vs database blob
I am starting to develop a web app, the primary purpose of which is to display photos. The users will be able to upload photos as well.
The first question that came up was where to store the photos: on the file system or the database.
I will be using a Windows box to host the site. The database is MySQL and the backend code is in C# utilizing ASP.NET MVC.
Filesystem, of course, unless you're aiming for a story on thedailywtf. The easiest way is to have the photos organized by a property you can derive from the file itself, such as its SHA-1 hash. Then just store the hash in the database, attached to the photo's primary key and other attributes (who uploaded it, upload date, etc).
It's also a good idea to divvy up the photos on the filesystem, so you don't end up with millions of files in a single directory. So you'll have something like this:
storage/00/e4/f56c0de1c61fdb926e79e8a0a65bd12930c9.jpg
storage/25/9a/ec1c55bfb660548a6770238668c4b117d92f.jpg
storage/5d/d5/4b01d98f17a9ad9dd1526b49ba39b5aa37a1.jpg
storage/63/49/6f740b6c284ce6685dc17d473a7360ace249.jpg
storage/b1/75/066d178188dde110149a8422ab651b0ee615.jpg
storage/b1/20/a2b7d02b7b0c43530677ab06235382a37e20.jpg
storage/da/39/a3ee5e6b4b0d3255bfef95601890afd80709.jpg
This is also easy to port if you ever move to sharded storage.
If you are using SQL Server 2008 there's a Filestream datatype that handles most of the problems mentioned about the DB getting larger. It handles all the annoying details of synchronizing between the filesystem and the table.
Look here for a blog post about the topic: Store any data in SQL Server 2008 (Katmai)
If you're building a website around photos then forget the database. If it will become popular your database is going to be hit hard and the majority of its time will be spent delivering photos. Also databases don't scale very well. There are so much more advantages in keeping them on the file system. And you can scale very well, having static content servers, using services for content delivery.
Also, Amazon S3 or other cloud providers do have their advantages. For instance S3 + Amazon CloudFront will provide good performance. CloudFront caches your files on servers around the world so they'll be very easily/fast accessible from anywhere. BUT if we're talking pictures and the site becomes popular your bills might be quite high.
For S3 Amazon charges per storage and per transfer in/out of the cloud.
For CloudFront per transfer.
Generally, people store binary data such as images on the filesystem, not the database. They reference the filesystem path from the database. Retrieving BLOBs (binary large objects) from the database is slower than allowing the web server to serve static files from the file system.
I would use something like Amazon S3.
But, if the choice is between filesystem and database I would choose filesystem because it's faster to server images from a filesystem than a database.
The only reason I would put photos as BLOBs in a database would be if I had a cluster of servers, and I was using database replication to automatically copy the photos to every machine in the cluster.
Life is much simpler if you just store the photos as files, and store the filenames of the photos in the database. If you need to create unique filenames for the photos, you can use a primary key integer from the database as part of the filename. But you could also just use a hash of the photo itself, as suggested by John Milliken. That's simple, and simple is better.
Some people point out that it's easier to manage if everything's in the database: including making backups, and preserving referential integrity.
If you store it in db, the db will grow quickly and will be much, much larger. It is just a touch more complicated to get an image out of db for display then to it is to get it from a file system. On the other hand, you better make sure that the file names and paths do not get out of sync with what is stored in db. In the past i have chosen to store on disk instead of db. It made it easier for me do move the database to different boxes. Worked out well.
We had a similar decision to make for a project I am on. The compelling thing about jamming stuff (images and other BLOBy things) into the DB is that it is is less likely that someone might delete/alter something (either intentionally or unintentionally). But, that isn't the choice we made. Instead we have the path info stored in the DB and use that to reference the data via UNC path. Data paths are stored in two parts - a part that references the location of the data relative to which machine it resides on and a part that points to which machine that group of data is on. When we need to move data around we can update the appropriate path info.
It is certainly quick to get the data without pulling out of the DB. Ultimately that was a major deciding factor.
It makes life so easy when you have a blob database. You should forget about the nightmare that is file system management.
EDIT
ID
VARBINARY
From experience this is an efficient way to manage binary files. You have one database that has only binary files. How can this be any harder to backup?
I've seen lots of questions and answers about this topic, but unfortunately none of them seem to fit my needs closely enough. This question is about both iOS and vb2010.
I have created an app which communicates with my server (at this point, "my server" is actually my Win7 machine that I am using for development. Eventually it will move to an actual server). The back end is a VB program that interfaces between the iPhone app and a SQL Server database. A specific port is open to the computer, and everything there works great. I am able to retrieve data from the database and display it on the iPhone with the app, and also able to save records from the app to the database. It's all so very cool, and I'm really jazzed about it.
The final phase of the project requires uploading a camera image to the server. This is the part where it gets funky for me.
So far I have been able to save the image as a file on the iPhone's disk. Now I need to understand better how to transfer that file to a directory on the server.
I know that there are many techniques that use the "post" method, but as I understand it, that requires a PHP or ASP page to accomplish. FTP generally uses ports 20 and 21. What I would really like is to be able to set up the existing software to receive an upload request from the app, and then place itself in receive mode. This would not require opening another port, and I can tailor the VB code to handle the file however I like.
All I really need to understand is, how can I get the two programs to do the file transfer? And is it reasonable to assume that I can use the same port?
I wish to create an application that will hold a large amount of heavy data (images and sound files) - I wish to avoid having a heavy app. I saw that there are apps that you download from app store and only after they are on your device they download from a server the rest of the files.
I already started reading about JSON and it seems quite clear, but I have never worked with servers before.
Can anyone direct me to a good tutorial that explains how to create my database on a server?
Maybe some recommandations to which servers and tools are available? - preferably free tools...
You can use phpMyAdmin and mySql (both free).
You can find all the info you need to install phpMyAdmin here :
http://www.phpmyadmin.net/documentation/
And you will learn all you need for basic sql here:
http://www.w3schools.com/sql/default.asp