I made a very big script to feel my initial datas into my rails app. I have about 3000 lines in my CSV and 10000 images.
After maybe 300 upload i got this message :
/usr/local/lib/ruby/gems/1.9.1/gems/activesupport-3.0.9/lib/active_support/core_ext/kernel/agnostics.rb:7:in ``': Cannot allocate memory - identify -format %wx%h '/tmp/stream20111104-14788-1hsumv7.jpg[0]' (Errno::ENOMEM)
My upload script :
if (row[28] != nil)
hotelalbum = HotelAlbumPhoto.find_or_create_by_title(h.title)
hotelalbum.description = "Album photo de l'hotel " + h.title.capitalize
hotelalbum.hotel_id = h.id
hotelalbum.save
files = Dir.glob('IMAGES/' + row[28].gsub(/\\/,'/') + '/*.jpg')
i =0
for file in files
i += 1
photopath = File.expand_path('../../import', __FILE__) + '/' + file
photoname = file.split('/').last
if (i==1)
hotelalbum.thumbnail = open(photopath)
hotelalbum.save
end
if (i==1)
h.thumbnail = open(photopath)
end
photo = HotelImage.find_or_create_by_image_file_name_and_hotel_album_photo_id(photoname,hotelalbum.id)
if (photo.image_file_size == nil || photo.image_file_name != photoname)
photo.image = open(photopath)
photo.activated = true
photo.alt = "Photo de l'hotel " + h.title
photo.save
else
puts photopath + ' already updated'
end
end
end
When i check my memory with top command, i see ruby process use more memory on each upload. How can i manage this?
Thank you for help
ps : My server is a virtual machine with 512Mb memory, one solution is to inscrease this memory but i hope to find an other.
I don't know where the open function is defined, but I'm suspicious that I don't see a corresponding close...
update Better idea, change photo.image = open(photopath) to photo.image = File.read(photopath)
According to the docs, read:
Opens the file, optionally seeks to the given offset, then
returns length bytes (defaulting to the rest of the file).
read ensures the file is closed before returning.
Looks like a memory leak problem of ImageMagick? Maybe it will help to process the list in big blocks or chunks with in_groups_of and force garbage collection with GC.start after each chunk:
files.in_groups_of(100) {|chunk|
# process chunk
GC.start
}
Related
Would it be possible to access the ActiveStorageBlob or ActiveStorageAttachment like it would be a native model ?
E.g.
I want to do ActiveStorageBlob.first to access the first record of this model/table.
or. ActiveStorageAttachment.all.as_json to generate json formated print.
The background idea is to find a way how to dump the content of these ActiveStorage related tables as json formated files. Then change simething on these files, and load it back.
----Extending this text after got correct answer-----
Thank you very much Sarah Marie.
And I hope you know how to load the JSON data back into these tables ?
I have tried this :
dump_file_path = File.join(Rails.root, "backup", active_storage_blobs_file)
load_json = JSON.parse(File.read(dump_file_path))
load_json.each do |j|
ActiveStorage::Blob.create(j)
end
But thats not working.
ActiveModel::UnknownAttributeError (unknown attribute
'attachable_sgid' for ActiveStorage::Blob.)
ActiveStorage::Blob.first
ActiveStorage::Attachment.all.as_json
---- For second extended question ----
ActiveStorage::Blob.create_before_direct_upload!(
filename: j[:filename],
content_type: j[:content_type],
byte_size: j[:byte_size],
checksum: j[:checksum]
)
# or
ActiveStorage::Blob.create_before_direct_upload!(**j.symbolize_keys)
Reference: https://github.com/rails/rails/blob/5f3ff60084ab5d5921ca3499814e4697f8350ee7/activestorage/app/controllers/active_storage/direct_uploads_controller.rb#L8-L9
https://github.com/rails/rails/blob/098fd7f9b3d5c6f540911bc0c17207d6b48d5bb3/activestorage/app/models/active_storage/blob.rb#L113-L120
Now I have a complete solution, how to dump and load the ActiveStorage tables as JSON files.
...dump it
active_storage_blobs_file = "active_storage_blob.json"
active_storage_attachments_file = "active_storage_attachment.json"
puts("...dump active_storage_blob")
dump_file_path = File.join(Rails.root, "backup",active_storage_blobs_file)
dump_file = File.open(dump_file_path, "w")
dump_file.write(JSON.pretty_generate(ActiveStorage::Blob.all.as_json))
dump_file.close()
puts("...dump active_storage_attachment")
dump_file_path = File.join(Rails.root, "backup",
active_storage_attachments_file)
dump_file = File.open(dump_file_path, "w")
dump_file.write(JSON.pretty_generate(ActiveStorage::Attachment.all.as_json))
dump_file.close()
...load it back
puts("...load active_storage_blob")
dump_file_path = File.join(Rails.root, "backup", active_storage_blobs_file)
abort("File does not exist (" + dump_file_path + ") > abort <") unless File.exist?(dump_file_path)
load_json = JSON.parse(File.read(dump_file_path))
load_json.each do |j|
j = j.except("attachable_sgid")
result = ActiveStorage::Blob.create(j)
if (not result.errors.empty?)
puts(result.errors.full_messages.to_s)
puts(j.inspect)
exit(1)
end
end
puts("...load active_storage_attachment")
dump_file_path = File.join(Rails.root, "backup", active_storage_attachments_file)
abort("File does not exist (" + dump_file_path + ") > abort <") unless File.exist?(dump_file_path)
load_json = JSON.parse(File.read(dump_file_path))
load_json.each do |j|
result = ActiveStorage::Attachment.create(j)
if (not result.errors.empty?)
puts(result.errors.full_messages.to_s)
puts(j.inspect)
exit(1)
end
end
I'm new to Rails.
I have a method that downloads a file from google drive and saves it in the local disc. When the file is downloaded, the console returns nil, but the file is in the folder.
I need to use this file in my controller, but if the download method is returning nil I can't pass it around as an object.
download method:
def download
found = google_files.select { |f| f.id == file_id }
file_title = found.first.title
file = session.file_by_title(file_title)
path = File.join(Backup.base_directory, file_title)
file.download_to_file("#{path}")
end
Controller:
def create
# file_id = params.fetch(:file_id)
file_id = "0Byyflt8z3jarbm5DZGNNVXZSWjg"
#backup = DiscourseDownloadFromDrive::DriveDownloader.new(file_id).download
end
console output after executing the download method:
[...]
Writing chunk (1397 bytes)
Writing chunk (1397 bytes)
Writing chunk (1397 bytes)
Writing chunk (619 bytes)
Success - nil
=> nil
[4] pry(main)>
Logger:
Rails.logger.debug(">>> #BACKUP >>>: #{#backup.inspect}")
D, [2017-09-07T20:21:24.835450 #7755] DEBUG -- : >>> #BACKUP >>>: nil
Any hint on how to proceed with this would be very much appreciated!
Your download method always returns nothing but nil. That's because the gem's download_to_file always returns nil.
You shoud change your download method for it to return something, that you can use to get the file. I think this method should return the path to the downloaded file.
def download
found = google_files.select { |f| f.id == file_id }
file_title = found.first.title
file = session.file_by_title(file_title)
path = File.join(Backup.base_directory, file_title)
file.download_to_file("#{path}")
path
end
Now you can use it in the controller:
def create
# file_id = params.fetch(:file_id)
file_id = "0Byyflt8z3jarbm5DZGNNVXZSWjg"
file_path = DiscourseDownloadFromDrive::DriveDownloader.new(file_id).download
#backup = File.open(file_path) # do whatever you want with the file, since now you know how to get it
end
I have a ruby controller
def new
counter = 1
fileW = File.new("query_output.txt", "w")
file = File.new("query_data.txt", "r")
while (line = file.gets)
puts "#{counter}: #{line}"
query = "select name,highway from planet_osm_line where name ilike '" +line+"'"
#output = PlanetOsmLine.connection.execute(query)
#output.each do |output|
fileW.write(output['highway'] + "\n")
end
counter = counter + 1
end
file.close
query = ""
#output = PlanetOsmLine.connection.execute(query)
end
So in this I am reading from a file like
%12th%main%
%100 feet%
%12th%main%
%12th%main%
%12th%main%
%100 feet%
In the ruby console I can see all the queries getting executed but in query_output.txt I only get the output of last query. What am I doing wrong here?
You use filemode w which will re-create the output file every time (so you will write into an empty file). Instead open your file as follows:
fileW = File.new("query_output.txt", "a")
a stands for append. It will open or create the file, and append at the back.
For more info concerning file-modes: http://pubs.opengroup.org/onlinepubs/009695399/functions/fopen.html
I am comparing many images based on surf features. I am able to compare only 2 images, but I need to do it for multiple images in two folders; can someone help me out?
functionname='OpenSurf.m';
functiondir=which(functionname);
functiondir=functiondir(1:end-length(functionname));
addpath([functiondir '/SubFunctions'])
defaultoptions=struct('tresh',0.0002,'octaves',5,'init_sample',2,'upright',false,'extended',false,'verbose',false);
if(~exist('Options','var')),
Options=defaultoptions;
else
tags = fieldnames(defaultoptions);
for i=1:length(tags)
if(~isfield(Options,tags{i})), Options.(tags{i})=defaultoptions.(tags{i}); end
end
if(length(tags)~=length(fieldnames(Options))),
warning('register_volumes:unknownoption','unknown options found');
end
end
iimg=IntegralImage_IntegralImage(img);
FastHessianData.thresh = Options.tresh;
FastHessianData.octaves = Options.octaves;
FastHessianData.init_sample = Options.init_sample;
FastHessianData.img = iimg;
ipts = FastHessian_getIpoints(FastHessianData,Options.verbose);
if(~isempty(ipts))
ipts = SurfDescriptor_DecribeInterestPoints(ipts,Options.upright, Options.extended, iimg, Options.verbose);
end
When I copy a YouTube link, like this one, for example (http://www.youtube.com/watch?v=_Ka2kpzTAL8), and paste it into a Facebook status update, Facebook somehow grabs a still of the video. I'd like to build a feature that does the same thing, but I don't really know how to find the image.
Can anyone point me in the right direction? I'm open to PHP, Python, Ruby or Perl.
Alright, found it !
Take this link: http://i4.ytimg.com/vi/something/hqdefault.jpg and replace something with the video's ID on Youtube. For a smaller image, just remove the hq part of the URL.
I can also confirm this implementation is the way two go. I have used it a Ruby screen scraper ( search application ) # http://stripsearch.me
# search10b Youtube crawl9
#search10a1 = search_array[1]
if search_array[1].present?
#results_9b = []
params_9b = search_array[1].split.map(&:downcase).join('+')
search_url_yt_9b = url_9b << params_9b
doc_9b = Nokogiri::HTML(open(search_url_yt_9b).read, nil, 'utf-8')
entries_9b = doc_9b.css('.yt-ui-ellipsis-2.spf-link')
if entries_9b.size != 0
entries_9b.each do |entry_9b|
title = entry_9b['title']
li = "https://www.youtube.com"
nk = entry_9b['href']
link = li << nk
img_src = entry_9b['href']
uid = img_src.split('=').last
precede_url = "http://i4.ytimg.com/vi/"
append_url = "/hqdefault.jpg"
img_url = precede_url << uid << append_url
precede_tag = "<img src="
append_tag = ">"
img = precede_tag << img_url << append_tag
#results_9b.push(title, link, img)
end
else
#error_9b = "no results found here for " + params_9b
end
end