Save large text file without using too much memory - ruby-on-rails

Question

I have a model that creates a KML file. I treat that KML as a string and then forward that to mailer which then delivers it:
def write_kml(coords3d, time)
kml = String.new
kml << header
coords3d.each do |coords|
coordinates = String.new
coords.each do |coord|
lat = coord[0].to_f
lng = coord[1].to_f
coordinates << "#{lng}" + "," "#{lat}" + ",0 "
kml << polygon(coordinates)
end
end
kml << footer
kml
end
This gets used here:
CsvMailer.kml_send(kml,time, mode, email).deliver
Mailer:
def kml_send(kml, time, mode, email)
#time = (time / 60).to_i
#mode = mode
gen_time = Time.now
file_name = gen_time.strftime('%Y-%m-%d %H:%M:%S') + " #{#mode.to_s}" + " #{#time.to_s}(mins)"
attachments[file_name + '(KML).kml'] = { mime_type: 'text/kml', content: kml}
mail to: email, subject: ' KML Filem'
end
It takes up a huge amount of memory. Some of these files are quite large (200MB), so on Heroku for example, they take up too much space.
I had some ideas using S3, but I would need to create this file to begin with, so it would still use the memory. Can I write straight to S3 without using memory?

Answer 1

You can do this with s3 multipart uploads, since they don't require you to know the file size upfront.
Parts have to be at least 5MB in size, so the easiest way to use this is to write your data to an in memory buffer and upload the part to s3 every time you get past 5MB. There's a limit of 10000 parts for an upload, so if your file size is going to be > 50GB then you'd need to know that ahead of time so that you can make the parts bigger.
Using the fog library, that would look a little like
def upload_chunk connection, upload_id, chunk, index
md5 = Base64.encode64(Digest::MD5.digest(chunk)).strip
connection.upload_part('bucket', 'a_key', upload_id, chunk_index, chunk, 'Content-MD5' => md5 )
end
connection = Fog::Storage::AWS.new(:aws_access_key_id => '...', :region => '...', :aws_secret_access_key => '...'
upload_id = connection.initiate_multipart_upload('bucket', 'a_key').body['UploadId']
chunk_index = 1
kml = String.new
kml << header
coords3d.each do |coords|
#append to kml
if kml.bytesize > 5 * 1024 * 1024
upload_chunk connection, upload_id, kml, chunk_index
chunk_index += 1
kml = ''
end
end
upload_chunk connection, upload_id, kml, chunk_index
#when you've uploaded all the chunks
connection.complete_multipart_upload('bucket', 'a_key', upload_id)
You could probably come up with something neater if you created an uploader class to wrap the buffer and stuck all the s3 logic in there. Then your kml code doesn't have to know whether it has an actual string or a string that flushes to s3 periodically