I saw N number of tutorials on uploading a file to active storage locally, to S3 etc using rails view. But I cannot find a legit source on how to upload a file to an active storage through API Say via postman, like what are prerequisites required to pass an attachment through API via postman?
Let me detail it.
Step 1: Through React or any frontend framework, I will choose a file to upload(a .PDF file), ultimately it should be saved somewhere.
Step 2: This chosen file should be passed through as API to the backend service where the storage of the chosen file to be saved in a storage like AWS S3.
How will the API request be sent to store the file? Can someone help me out with it?
its Very Simple to Do
Open the Postman
Go to Body -> Form-data
after that take a mouse to the key filed you will find Option for File
Then you have to select the file and send the request
Or send the File Path in the raw JSON object as well like this
and then Create the Object at backend so place check on file object if it blank then send path like this
// For Temporary Use Over Back End Only NOT NEED To Send These
"file_path" : "/home/jolly/Downloads/pdf/pdf-5.pdf",
"file_name" : "pdf-5.pdf"
and then in the code use it to create the file object which u pass where u needed to save s3 or rails storage
file_path = params[:file_path]
file_name = params[:file_name]
image_path = Rails.root+file_path
image_file = File.new(image_path)
Related
So, I'm creating an app that works like a bot, it makes a call to an API from time to time, then it receives a response in a json-like format and saves it like this:
finalResult = RestClient.get( apiUrl, headers = apiHeaders )
jsonData = JSON.parse(ActiveSupport::Gzip.decompress(finalResult))
time = Time.now
File.write("public/#{time}.json", jsonData)
I'm using ActiveSupport to be able to parse this Gzip compressed data, since it's a lot of data, otherwise it takes forever to get it. Then I get the time the data was received, basically, and I use it to name the file so that I can keep good track of it.
What I need to do now is compress this .json file, if possible, into a .zip file(it can be .rar, or .7z, or .tz, whatever) before I upload it to my storage so it takes less space. Is there anyway that I can do something similar to File.write but to save it as a zipped json file? I already checked stuff like zlib and ruby-zip, but they only let me zip files that already "exist", so I can't save it as a zipped .json directly, I'd need to take the .json file and then zip it, but how could I do that if the name of the file is a Time.now and it always change?
I'd appreciate any help, thanks in advance :)
EDIT¹
Giving some more details that may help you to help me:
I created a controller and model to handle this, since I'll be using ActiveStorage. It's ResponsesController and Response model, and the only parameter that the Response model has is has_one_attached :json_file. I intend to use Heroku to handle the CRON job of calling the API and I'll upload the .json files(or .zip files) to an AWS storage.
I'm processing an XML file added to S3 and writing the results to a firehose, and storing the results on the same S3 bucket, but the destination filename has to be in a specific format. I've examing the documentation and I can't see any way to set the format of the filename.
The closest I can find is in the firehose FAQ
Q: What is the naming pattern of the Amazon S3 objects delivered by Amazon Kinesis Data Firehose?
The Amazon S3 object name follows the pattern DeliveryStreamName-DeliveryStreamVersion-YYYY-MM-DD-HH-MM-SS-RandomString, where DeliveryStreamVersion begins with 1 and increases by 1 for every configuration change of the delivery stream. You can change delivery stream configurations (for example, the name of the S3 bucket, buffering hints, compression, and encryption) with the Firehose Console or the UpdateDestination operation.
If you are using static naming, you can specify it through Firehose Console or the UpdateDestination operation.
But if you are looking for some dynamic naming, unfortunately, currently it is not possible. Refer to this question for detail answering -
Storing Firehose transfered files in S3 under custom directory names
I too wasn't happy with that I couldn't specify the name of my files dynamically, so I made a lambda function to rename the files that my Kinesis stream outputs. These were the steps I took
I included the filename I wanted in my Kinesis data.
I created a new lambda function, set up to run whenever kinesis outputs a file.
My lambda function:
opens my file
grabs the new file name
creates the new file
deletes the badly named old file.
import boto3
import json
def lambda_handler(event, context):
key = event["Records"][0]["s3"]["object"]["key"]
bucket=event["Records"][0]["s3"]["bucket"]["name"]
s3resource = boto3.resource('s3')
obj = s3resource.Object(bucket, key)
body = obj.get()['Body'].read()
dic = json.loads(body)
my_new_file_name= dic["my_new_file_name"]
s3resource.Object(bucket, str(my_new_file_name).copy_from(CopySource=f'{bucket}/{key}')
s3resource.Object(bucket, key).delete()
Is it possible to use AWSS3TransferManager or an alternative class to determine if a file has changed on the s3 server before downloading it?
Ideally I'd like to have some sort of automatic function that takes into account automatically the following:
File size
File type
File name
or:
File content data analysis (some sort of signature based on bit content)
Alternatively:
I see that it is possible to get the modification date for an s3 bucket, however I am not sure how this works in iOS.
If you want to download the file like the AWSS3TransferManager download method, you can use the alternative class AWSS3GetObjectRequest / AWSS3GetObjectOutput.
If you do not want to download the file, you can use AWSS3HeadObjectRequest / AWSS3HeadObjectOutput that will return the object's metadata or "head" of the file.
The value: lastModified will be in the AWSS3...ObjectOutput value for both class responses.
The flow is:
The user selects an image on the client.
Only filename, content-type and size are sent to the server. (E.g. "file.png", "image/png", "123123")
The response are fields and policies for upload directly to S3. (E.g. "key: xxx, "alc": ...)
The case is that if I change the extension of "file.pdf" to "file.png" and then uploads it, the data sent to the server before uploads to S3 are:
"file.png"
"image/png"
The servers says "ok" and return the S3 fields for upload .
But the content type sent is not a real content type. But how I can validate this on the server?
Thanks!
Example:
Testing Redactorjs server side code (https://github.com/dybskiy/redactor-js/blob/master/demo/scripts/image_upload.php) it checks the file content type. But trying upload fake image (test here: http://imperavi.com/redactor/), it not allows the fake image. Like I want!
But how it's possible? Look at the request params: (It sends as image/jpeg, that should be valid)
When I was dealing with this question at work I found a solution using Mechanize.
Say you have an image url, url = "http://my.image.com"
Then you can use img = Mechanize.new.get(url)[:body]
The way to test whether img is really an image is by issuing the following test:
img.is_a?(Mechanize::Image)
If the image is not legitimate, this will return false.
There may be a way to load the image from file instead of URL, I am not sure, but I recommend looking at the mechanize docs to check.
With older browsers there's nothing you can do, since there is no way for you to access the file contents or any metadata beyond its name.
With the HTML5 file api you can do better. For example,
document.getElementById("uploadInput").files[0].type
Returns the mime type of the first file. I don't believe that the method used to perform this identification is mandated by the standard.
If this is insufficient then you could read the file locally with the FileReader apis and do whatever tests you require. This could be as simple as checking for the magic bytes present at the start of various file formats to fully validating that the file conforms to the relevant specification. MDN has a great article that shows how to use various bits of these apis.
Ultimately none of this would stop a malicious attempt.
I am using paperclip to attach an excel file to a mode.
The purpose is to import the data from the excel file to the database.
Model: Import
has_attached_file: spreadsheet
For the import process, I want to access the file in my model as following.
path = "#{Rails.root}/public/#{spreadsheet.url}"
This doesn't work. I guess because the url has the timestamp at the end.
In general what is the best way to access the attachments on server side ?
I think you're looking for the to_file method. You should be able to do something like this:
excel_file = self.spreadsheet.to_file
which will either return the uploaded file from the server (if you're using s3 or remote storage), or if it has been assigned to the model but not actually stored yet (if you haven't called model.save since it was uploaded), it returns the temp file stored on disk.
From there you should be able to use an excel gem or library to parse the contents.
Alternatively, you can use spreadsheet.url(nil, false) - the second parameter denotes whether or not to append a timestamp.
#spreadsheet.path
http://rdoc.info/gems/paperclip/2.3.8/Paperclip/Attachment#path-instance_method