I have successfully migrated my parse db to aws but the urls of image files still like http://files.parsetfss.com/77447afb-f681-4b55-afad-6bceeb2e155a/tfss-79297c86-bd48-4d7f-87ab-c43e02d1a8f3-photo.png
it means files are still on parse s3 cloud or something their own storage so what will happen to those files after parse shutdown.
what is the way to migrate the images to new database/storage on my own AWS. I am worried because I have apprx 14.5 k images on parse.
Please provide you valuable guidance on this.
As you know, Parse Files is a feature allowing developers to upload files (up to 10 megabytes each) and reference those files from objects in Parse Core data classes or directly by URL provided in the API response from Parse.
Behind-the-scenes, Parse is uploading your files to a Parse-owned S3 bucket (files.parsetfss.com) and prefixing the file objects with your application “File key”.
To answer your questions directly, there's active solutions in-the-works and
here’s the latest addressing the migration and optional Parse File storage options post migration.
How do I migrated my legacy Parse Files over to Parse Sever?
Migrating legacy Parse Files from Parse-owned S3 bucket to developer-owned Parse Sever: https://github.com/ParsePlatform/parse-server/issues/8
What NON database options do I have for storing my Parse Files after migrating to Parse Server?
Add support to upload Parse Files directly to Amazon Simple Storage (S3) via S3 adapter running Parse Server: https://github.com/ParsePlatform/parse-server/pull/113
Migration Considerations for Parse Files:
When a user first uploads a file, Parse service uploads it to files.parsetfss.com and responds with a link directly to the file. At this point, there is NO POINTER or METADATA referencing this file object in Parse Core or other data classes. The developer would need to keep a reference to this file in their own data class OR make another API call to create an object or update an existing object and associate the Parse File with that object. Otherwise, the file is orphaned. Parse does allow you to "Clean Up Files" in the App Settings of your application. This option will delete any files that are not referenced by any objects. Orphaned files can only be deleted by using the Master Key and there is currently no way to search ALL your uploaded Parse Files per account or application unless it’s associated with a class object.
What happens to EXISTING Parse Files during the migration to Parse Server?
During the migration, the files stay on Parse's S3 bucket, but the newly migrated Parse Server knows how to continue serving them up post migration. NO FILES HAVE BEEN MIGRATED! Only the pointers to the S3 bucket owned by Parse AND only if those files are associated with an object. So, if the developer DOES NOT MIGRATE the “legacy” pre-migration Parse Files from Parse prior to Parse shutdown in 2017, they could lose access to these files.
Parse and the open source Parse Server community is ACTIVELY working on providing migration solutions. See here.
What happens to NEW Parse Files uploaded after the migration to Parse Server?
New Parse Files uploaded to a Parse Server after migration are hosted in MongoDB GridStore(Mongo). Only files uploaded through api.parse.com API endpoint are hosted by Parse. In other words, if you migrated your app to Parse Server but have not updated the clients to use the new Parse Server API endpoint, those Parse Files will still get uploaded to the Parse owned S3 bucket. For those clients that upload Parse Files using their new Parse Server API endpoint, the files will be stored directly into the developers MongoDB database.
I hope you found this information useful.
Related
I have CSV files in a directory of an S3 bucket. I would like to use all of the files as a single table in Dremio, I think this is possible as long as each file has the same header/columns as the others.
Do I need to first add an Amazon S3 data source using the UI or can I somehow add one as a Source using the Catalog API? (I'd prefer the latter.) The REST API documentation doesn't provide a clear example of how to do this (or I just didn't get it), and I have been unable to find how to get the "New Amazon S3 Source" configuration screen as shown in the documentation, perhaps because I've not logged in as an administrator?
For example, let's say I have a dataset split over two CSV files in an S3 bucket named examplebucket within a directory named datadir:
s3://examplebucket/datadir/part_0.csv
s3://examplebucket/datadir/part_1.csv
Do I somehow set the S3 bucket/path s3://examplebucket/datadir as a data source and then promote each of the files contained therein (part_0.csv and part_1.csv) as a Dataset? Is that sufficient to allow all the files to be used as a single table?
It turns out that this is only possible for admin users, normal users can't add a source. To do what I have proposed above you put the files into an S3 bucket which has already been configured as a Dremio source by an admin user. Then you promote the files or folder as a data source using the Dremio Catalog API.
Referring to the link here:
https://learn.microsoft.com/en-us/vsts/extend/develop/data-storage.
Does the "documents" for the data storage refers to any file types?
Can we upload files via VSTS extension?
Ie. Is it possible to invoke a server side implementation using aspx or php to store a file inside my extension?
As Jazimov said that you can’t store the files in VSTS extension data storage.
I recommend that you can upload the files to a repository of VSTS through REST API (e.g. Add a binary file) in your VSTS extension, then store the necessary information (e.g. server path, file name, objectId etc) in data storage.
The Documents object is a collection of Document objects. Document objects are deserialized as JSON objects.
When you ask if "documents" can refer to any file type, the answer is "No". Documents are not files. They start as C# objects that are serialized then persisted to a data store. When retrieved, they are returned as JSON strings.
You can encode a file into your data structure before storing and then the returned JSON will contain your deserialized file information. See Binary Data in JSON String. Something better than Base64 for more details.
The last part of your question: Of course your can invoke a service that uploads and downloads files. You would have to write that code logic on your own--it's not part of an VSTS extension's data-storage subsystem.
I need help moving the images I have from Parse to S3 on AWS. I have viewed numerous supposed guides and GitHub projects, but everything stops short at giving you all the information. One even says, you need GCS bucket set up, but gives no details on how to set up one. Just someone please help me with this. I have the S3 File Adapter in my index.js all set up for the app, but none of the images are there, they are still hosted in parse.
If you are referring to old images that where hosted with parse.com that you want to move across to your own environment then it can be done with the utility tool.
Get all files across all classess in a Parse database. Print file URLs
to console OR transfer to S3, GCS, or filesystem. Rename files so that
Parse Server no longer detects that they are hosted by Parse. Update
MongoDB with new file names.
https://github.com/parse-server-modules/parse-files-utils
Moving forward if you have setup your S3 bucket correctly all new images from your app will be stored there.
https://github.com/ParsePlatform/parse-server/wiki/Configuring-File-Adapters
I am using the aws-sdk gem to read a CSV file stored in AWS S3.
Referencing the AWS doc. So far I have:
Aws::S3::Resource.new.bucket(ENV['AWS_BUCKET_NAME']).object(s3_key).get({ response_target: "#{Rails.root}/tmp/items.csv" })
In Pry, this returns:
output error: #<IOError: closed stream>
However, navigating to tmp/. I can see the items.csv file and it contains the right content. I am not certain wether the return value is an actual error.
My second concern. Is it fine to store temporary files in "#{Rails.root}/tmp/"?
Or should I consider another approach?
I can load the file in memory and then CSV.parse. Will this have implications if the CSV file is huge?
I'm not sure how to synchronously return a file object using the aws gem.
But I can offer some advice on the other topics you mentioned.
First of all, /tmp - I've found that saving files here is a working approach. On AWS, I've used this directory to create a local LRU cache for S3-stored images. The key thing is to preemp the situation where the file has been automatically deleted. The file needs to be refetched if this happens. By the way, Heroku has a 'read-only filesystem' but still permits you to write into /tmp.
The second part is the question of synchronously returning a file object.
While it may be possible to do this using the S3 gem, I've found success fetching it over HTTP using something like open-uri or mechanize. If it's not supposed to be a publically-available asset, you can change the permissions on S3 to restrict access to your server.
I am trying to create a dashboard using CSV files, Highcharts.js, and HTML5. In a local development environment I can render the charts using CSVs both on my file system and hosted on the web. The current goal is to deploy the dashboard live on Heroku.
The CSVs will be updated manually - for now - once per day in a consistent format as required by Highcharts. The web application should be able to render the charts with these new, "standardized" CSVs whenever the dashboard page is requested. My question is: where do I host these CSVs? Do I use S3? Do I keep them on my local file system and manually push the updates to heroku daily? If the CSVs are hosted on another machine, is there a way for my application (and only my application) to access them securely?
Thanks!
Use the gem carrierwave direct to upload the file directly from the client to an Amazon S3 bucket.
https://github.com/dwilkie/carrierwave_direct
You basically give the trusted logged in client a temporary key to upload the file, and nothing else, and then the client returns information about the uploaded file to your web app. Make sure you have set the upload to be private to prevent any third parties from trying to brut force find the CSV. You will then need to create a background worker to do the actually work on the CVS file. The gem has some good docs on how to do this.
https://github.com/dwilkie/carrierwave_direct#processing-and-referencing-files-in-a-background-process
In short in the background process you will download the file temporarily to heroku, parse it out, get the data you need and then discard the copy on heroku, and if you want the copy on S3. This way you get around the heroku issue of permanent file storage, and the issue of tied up dynos with direct uploads, because there is nothing like NGINX for file uploads on heroku.
Also make sure that the file size does not exceed the available memory of your worker dyno, otherwise you will crash. Sense you don't seem to need to worry about concurrency I would suggest https://github.com/resque/resque.