As the title says, is there any decentralized storage technology for storing sensitive data like the personal information of users? Concretely, I am developing a Dapp on blockchain, where users have to give their personal information for specific purposes. Now what I am missing is a technology to store such sensitive data in a way that users can trust me I'm not using their information for other purposes. This will work if data is stored on a decentralized storage that can interact with smartcontract, in addition, whenever users make a request, my smartcontract should be able to retrieve data from the storage (as an API). This article shows 7 decentralized data storage networks that seem to work on my Dapp. But I'm a programmer and relatively new to blockchain so I don't really know which one is the best suit for my purposes and can be easily scaled up when more and more users enter. I'd like to hear your opinions, especially from those who worked/ is working on a similar projects to mine. Also, it would be great if someone can give me any application has been released that using such technology to store sensitive data. So far, I don't see any such application.
Any helps would be appreciated! Thanks.
Related
Google just released Cloud Firestore, their new Document Database for apps.
I have been reading the documentation but I don't see a lot of differences between Firestore and Firebase DB.
The main point is that Firestore uses documents and collections which allow the easy use of querying compared to Firebase, which is a traditional noSQL database with a JSON base.
I would like to know a bit more about their differences, or usages, or whether Firestore just came to replace Firebase DB?
I wrote an entire blog post all about this very question, and I recommend you check it out (or the official documentation) for a more complete answer.
But if you want the quick(-ish) summary, here it is:
Better querying and more structured data -- While the Realtime Database is just a giant JSON tree, Cloud Firestore is a little more structured. All your data consists of documents (which are basically key-value stores) and collections (which are collections of documents). Documents will also frequently point to subcollections, which contain other documents, which themselves can contain other documents, and so on.
This structured data helps you out in two ways. First, all queries are shallow, meaning that you can request a document without grabbing all the data underneath. This means you can keep your data stored hierarchically in a way that makes more sense to you without having to worry about keeping your database shallow. Second, you have more powerful queries. For instance, you can now query across multiple fields without having to create those "combo" fields that combine (and denormalize) data from other parts of your database. In some cases, Cloud Firestore will just run those queries directly, and in other cases, it will automatically create and maintain indexes for you.
Designed to Scale -- Cloud Firestore will be able to scale better than the Realtime Database. It's important to note that your queries scale to the size of your result set, not your data set. So searching will remain fast no matter how large your data set might become.
Easier manual fetching of data -- Like the Realtime Database, you can set up listeners in Cloud Firestore to stream in changes in real-time. But if you don't want that kind of behavior, and just want a simple "fetch my data" call, Cloud Firestore has that as well, and it's built in as a primary use case. (They're much better than the once calls in Realtime Database-land)
Multi region support -- This basically means more reliability, as your data is shared across multiple data centers at once. But you still have strong consistency, meaning you can always make a query and be assured that you're getting the latest version of your data.
Different pricing model -- While the Realtime Database primarily charges based on storage or network bandwidth, Cloud Firestore primarily charges based on the number of operations you perform. Will this be better, or worse? It depends on your app.
For powering a news app, turn-based multiplayer game, or something like your own version of Stack Overflow, Cloud Firestore will probably look pretty favorable from a pricing standpoint. For something like a real-time group drawing app where you're sending across multiple updates a second to multiple people, it probably will be more expensive than the Realtime Database.
Why you still might want the to use the Realtime Database -- It comes down to a few reasons.
That whole "it'll probably be cheaper for apps that make lots of frequent updates" thing I mentioned previously,
It's been around for a long time and has been battle tested by thousands of apps,
It's got better latency and when you need something with reliably low latency for a real-timey feel, the Realtime Database might work better.
For most new apps, we recommend you check out Cloud Firestore. But if you have an app that's already on the Realtime Database, I don't really recommend switching just for the sake of switching, unless you have a compelling reason to do so.
Reasons to choose Cloud Firestore over Realtime Database
It is an improved version
Firebase database was enough for basic applications. But it was not powerful enough to handle complex requirements. That is why Cloud Firestore is introduced. Here are some major changes.
The basic file structure is improved.
Offline support for the web client.
Supports more advanced querying.
Write and transaction operations are atomic.
Reliability and performance improvements
Scaling will be automatic.
Will be more secure.
Pricing
In Cloud Firestore, rates have lowered even though it charges primarily on operations performed in your database along with bandwidth and storage. You can set a daily spending limit too. Here is the complete details about billing.
Future plans of Google
When they discovered the flaws with Real-time Database, they created another product rather than improving the old one. Even though there are no reliable details revealing their current standings on Real-time Database, it is the time to start thinking that it is likely to be abandoned.
Suggest link from google as well :
Firebase Real-time Database vs FireStore
Extracted from google docs, a small sumamry here:
FireBase Real Time DB is JSON based NO SQL DB, meant for mobile apps, regional, and used typically to store and sync data between users/devices in realtime / extremely low latency.
FireStore is JSON 'like' NOSQL DB meant for high concurrency, global, easily auto scaling persistence, designed for any clients (not only mobile apps) with typical use cases such as asset tracking, real time analytics, building retail product catalogs, social user profile, gaming leaderboards, chat based applications etc.
Cloud Firestore is Firebase's database for mobile app
development. It builds on the successes of the Realtime Database with
a new, more intuitive data model. Cloud Firestore also features
richer, faster queries and scales further than the Realtime Database.
Realtime Database is Firebase's original database. It's an efficient,
low-latency solution for mobile apps that require synced states
across clients in realtime.
To choose between Firebase Realtime database and Cloud firestore based on your application requirements, read official documentation here.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I'm developing an app for distributors which gives them a list of stores and they note in the app how many products did they sell to any store. This is the main function of my app. There are two choices: firebase realtime database or Cloud Firestore. I need only speed and less pricing. Please tell me how their pricing does work and also which one is faster? Explain shortly please.
Among other factors, the ones mentioned below are really important when choosing a Database Management System are:
Data Model, Data consistency, Data Security, Data Protection, Multi Access and Integration, Efficiency, Usability, Implementation and Service Costs.
From Firebase documentation:
Which database is right for your project?
We recommend Cloud Firestore for most developers starting a new
project. Cloud Firestore offers additional functionality, performance,
and scalability on an infrastructure designed to support more powerful
features in future releases. Expect to see new query types, more
robust security rules, and improvements to performance among the
advanced features planned for Cloud Firestore.
Using Cloud Firestore and Realtime Database
You can use both databases within the same Firebase app or project.
Both NoSQL databases can store the same types of data and the client
libraries work in a similar manner.
Please note that in the link above you shall find detailed information about the differencies between the two.
From medium.com:
Cloud Firestore vs. Firebase Realtime Database
Things they have in common:
They are both easy to integrate into a project with limited setup, and they are compatible with everything else offered by Firebase.
Administrators will be able to see the data through the Firebase
console, which uses the same scheme in both of them. What this means
is that you can scour through the nodes and collections of the top
level to find the data or information that you are looking for.
Beyond that, they do not offer any further level of exploration. If you know the keys and objects that you are looking for, this will be
useful.
Let’s take a look at the differences.
Querying Support — Firestore is more potent in this regard. Locate records that match several field comparisons. Firebase uses a
simplistic data structure, which means that you will only be able to
run queries that search for the field beginning with your query.
Importing and Exporting Data — This is a feature that Firebase provides. It comes in handy when you are migrating data or if team
members who are not developers to make some changes to the data.
Real-time updates — Firebase focuses on real-time updates, which are very useful for handling customers who are using social media or
collaborative apps. It gives developers everything they need to
determine the customers that are active users in real-time.
Costs — The costs of the Realtime Database will go up as you send more data via reading/write operations. The price of the Firestore
database will increase with every API call that you make. However, be
sure to look at the entire cost breakdown before making any decisions.
Pricing
In the following link there are detailed information about Firebase
Realtime Database Pricing and Cloud Firestore Pricing
You may also find this article useful:
Cloud Firestore vs the Realtime Database: Which one do I use?
You should think more carefully the future work load your environment will have. That will give you better ground to evaluate the pricing. In general, I would prefer to use Firestore because it is the new one and it is superior when it comes to scaling the service. It will scale automatically as your user base grows and will not need any sharding in high volumes.
Realtime Database is the Firebase’s first and original cloud-based database. For the mobile apps requiring synced states across clients in realtime, it is an efficient and low-latency solution.
Cloud Firestore is Firebase’s newest flagship database for mobile apps. It is a successor to the Realtime Database with a new and more intuitive data model. Cloud Firestore is richer, faster, and more scalable than the Realtime Database.
You should check the official docs. And get a proper idea between these two. There are things you need to consider before choosing either of them.
According to your goal, both will help you in different ways.
Realtime and offline support, Querying, Writes and transactions, Reliability and Performance, Scalability, Security, Pricing, Data Model, etc.
These are the things you need to consider while choosing either of these
I would like to suggest this link though.
Cloud Firestore vs the Realtime Database: Which one do I use?
My question might seem a bit naive, but as a beginner iOS developer, I'm starting to think that Core Data is replaceable by firebase realtime database (or firestore in the future). I used both of them in two seperate projects and after activating the offline feature in firebase, I got the same results (that is, the data was saved to the device without the need for an internet connection). I think I read something in the firebase documentation about it not being able to filter and sort at the same time which would probably mean that Core Data can be more convenient for complex queries. It would be great to have some senior developers' views on this subject.
Thanks in advance.
The question is a bit off-topic for SO (IMO) and is (kind of) asking for opinions but it may be worth a high-level answer. I use both platforms daily.
Core Data and Firebase are two unrelated platforms used to (manage and) store data; it's hard to directly compare them without understanding your use case.
CD is a framework used to model objects in your app. It's the 'front end' of data storage, where the 'back end' could be SQL, flat files, plists etc. It's more of a single user concept which stores data locally on the device (it has cloud functionality but that's a different topic).
Firebase on the other hand is a live, event driven, cloud based, multi user capable NoSQL storage. While it offers off-line persistence, that's really for situations where you need to be interacting with data when the device is temporarily disconnected from the internet.
It is not correct that:
firebase documentation about it not being able to filter and sort at
the same time
But, your Firebase structure is dependent on what you want to get out of it - if it's structured correctly, it can be filtered and sorted at the same time in a variety of very powerful (and faaast) ways.
Core Data is really an incredible technology and building relationships between objects is very straight forward and has SQL-like queries for retrieving data.
If you are looking for a database that leverages local storage - go with Core Data or another database that's really strong locally such as Realm, MySql and a number of others.
If you want to have Cloud based, multi-user, event driven storage, Firebase is a very strong contender (Realm is another option as well)
I would suggest building a very simple To-Do type app and use Firebase for storage in one and then build another using Core data. Should only be a couple of hours of work but it will really give you some great basic experience with both - you can make a more informed decision from there.
I'm building a web app that need to store some resources, including but not limited to articles, pictures and videos. My question here is how videos (mp4/ogg) are stored on web server? just as bare file or as binaries in relational or nosql db?
The question to BLOB data almost always comes down to "don't BLOB data". There are very few times that make more sense to write a database connector for your data then to just keep it on disk.
The general trend is to use an established service that employs good design patterns, such as Paperclip for ruby, and tailor it to your needs.
Using an external storage service is also a good idea, for example Amazon S3 will store all of your data for pennies on the dollar per gigabyte, and they'll do an excellent job of it.
If you do decide to cook up your own server that handles data internally, might I recommend digital ocean? I have been very happy with the SSD servers I have setup there (which are super fast).
For video you will almost certainly need a webserver that is capable of streaming the file. I think Nginx has this feature.
I think you need to elaborate a bit about the use case you wish to implement for this app. Only then you can have precise answer.
And to to help out with that, here are some questions you need to ponder:
1- You said you wanted to store videos, what are your requirements beyond storage?
2- do you wish for example to offer access to third party users to these videos and search with keywords?
3- If yes, what kind of information is available about the videos? what is the expected average size of these files?
Many database engines offer the possibility of storing big binary files, but that comes with an impact on performance. That's why most of the storage systems that deal with big files, store the files themselves on the disk and any related metadata (file name, last updated, associated keywords, etc.) are stored in the database. That makes for a scalable system.
I'll edit this answer, if you find it useful and have further related-questions.
An unlimited file storage is difficult to setup without AWS S3. S3 is cheap and scalable solution but expensive to use without proper caching, so we have Nginx S3 proxy that works well: https://stackoverflow.com/a/44749584/290338
I know that there is Core Data to store information onto your phone, but what if you are making a social media application such as Instagram. Where did they manage to store all the information for every user? My main question is when there is a large amount of data to be stored for an application where do application designers store this information? Thanks
The data is stored on their servers in the form of a database.
http://en.wikipedia.org/wiki/Database
When the application needs to retrieve some of the user's information such as likes, photos etc... a HTTP Request is made to the server to request the data from the database.
http://en.wikipedia.org/wiki/Http_request#Request_message
PHP and MySQL are standard and low-end languages and databases to use and quite easy to learn and become familiar with. More advanced databases tend to use caching layers and other types of layers over the database which prevent data loss, increase read and write speeds, cache the request data, and more. Furthermore, other companies may chose to use NoSQL databases.
There is a comparison between NoSQL and SQL databases here: http://metadata-standards.org/Document-library/Documents-by-number/WG2-N1501-N1550/WG2_N1537_SQL_Standard_and_NoSQL_Databases%202011-05.pdf
There is a comparison chart in more detail that can be found here: http://en.wikipedia.org/wiki/Comparison_of_relational_database_management_systems
One the data has been requested and delivered onto your iOS Device, it can be manipulated to be stored in many different ways.
Core Data and SQLite are the generally used methods of storing the data as they derive from eachother and it is quite familiar to the majority of SQL databases in terms of the schema.
You can use coredata to store the data. Images can be stored in document folder.