I have a ruby controller
def new
counter = 1
fileW = File.new("query_output.txt", "w")
file = File.new("query_data.txt", "r")
while (line = file.gets)
puts "#{counter}: #{line}"
query = "select name,highway from planet_osm_line where name ilike '" +line+"'"
#output = PlanetOsmLine.connection.execute(query)
#output.each do |output|
fileW.write(output['highway'] + "\n")
end
counter = counter + 1
end
file.close
query = ""
#output = PlanetOsmLine.connection.execute(query)
end
So in this I am reading from a file like
%12th%main%
%100 feet%
%12th%main%
%12th%main%
%12th%main%
%100 feet%
In the ruby console I can see all the queries getting executed but in query_output.txt I only get the output of last query. What am I doing wrong here?
You use filemode w which will re-create the output file every time (so you will write into an empty file). Instead open your file as follows:
fileW = File.new("query_output.txt", "a")
a stands for append. It will open or create the file, and append at the back.
For more info concerning file-modes: http://pubs.opengroup.org/onlinepubs/009695399/functions/fopen.html
Related
Would it be possible to access the ActiveStorageBlob or ActiveStorageAttachment like it would be a native model ?
E.g.
I want to do ActiveStorageBlob.first to access the first record of this model/table.
or. ActiveStorageAttachment.all.as_json to generate json formated print.
The background idea is to find a way how to dump the content of these ActiveStorage related tables as json formated files. Then change simething on these files, and load it back.
----Extending this text after got correct answer-----
Thank you very much Sarah Marie.
And I hope you know how to load the JSON data back into these tables ?
I have tried this :
dump_file_path = File.join(Rails.root, "backup", active_storage_blobs_file)
load_json = JSON.parse(File.read(dump_file_path))
load_json.each do |j|
ActiveStorage::Blob.create(j)
end
But thats not working.
ActiveModel::UnknownAttributeError (unknown attribute
'attachable_sgid' for ActiveStorage::Blob.)
ActiveStorage::Blob.first
ActiveStorage::Attachment.all.as_json
---- For second extended question ----
ActiveStorage::Blob.create_before_direct_upload!(
filename: j[:filename],
content_type: j[:content_type],
byte_size: j[:byte_size],
checksum: j[:checksum]
)
# or
ActiveStorage::Blob.create_before_direct_upload!(**j.symbolize_keys)
Reference: https://github.com/rails/rails/blob/5f3ff60084ab5d5921ca3499814e4697f8350ee7/activestorage/app/controllers/active_storage/direct_uploads_controller.rb#L8-L9
https://github.com/rails/rails/blob/098fd7f9b3d5c6f540911bc0c17207d6b48d5bb3/activestorage/app/models/active_storage/blob.rb#L113-L120
Now I have a complete solution, how to dump and load the ActiveStorage tables as JSON files.
...dump it
active_storage_blobs_file = "active_storage_blob.json"
active_storage_attachments_file = "active_storage_attachment.json"
puts("...dump active_storage_blob")
dump_file_path = File.join(Rails.root, "backup",active_storage_blobs_file)
dump_file = File.open(dump_file_path, "w")
dump_file.write(JSON.pretty_generate(ActiveStorage::Blob.all.as_json))
dump_file.close()
puts("...dump active_storage_attachment")
dump_file_path = File.join(Rails.root, "backup",
active_storage_attachments_file)
dump_file = File.open(dump_file_path, "w")
dump_file.write(JSON.pretty_generate(ActiveStorage::Attachment.all.as_json))
dump_file.close()
...load it back
puts("...load active_storage_blob")
dump_file_path = File.join(Rails.root, "backup", active_storage_blobs_file)
abort("File does not exist (" + dump_file_path + ") > abort <") unless File.exist?(dump_file_path)
load_json = JSON.parse(File.read(dump_file_path))
load_json.each do |j|
j = j.except("attachable_sgid")
result = ActiveStorage::Blob.create(j)
if (not result.errors.empty?)
puts(result.errors.full_messages.to_s)
puts(j.inspect)
exit(1)
end
end
puts("...load active_storage_attachment")
dump_file_path = File.join(Rails.root, "backup", active_storage_attachments_file)
abort("File does not exist (" + dump_file_path + ") > abort <") unless File.exist?(dump_file_path)
load_json = JSON.parse(File.read(dump_file_path))
load_json.each do |j|
result = ActiveStorage::Attachment.create(j)
if (not result.errors.empty?)
puts(result.errors.full_messages.to_s)
puts(j.inspect)
exit(1)
end
end
What I want to solve
I want to change the name of the file after downloading the zip. I was trying to figure out how to remove the directory name in the middle of this.
In this case, the intermediate directory name is "output".
I would like to know if there is a better way.
2_J000001719_1202テスト/output/WNS_UP用データ.txt
2_J000001719_1202テスト/output/Images/1051687701.jpg
↓
2_J000001719_1202テスト/WNS_UP用データ.txt
2_J000001719_1202テスト/Images/1051687701.jpg
Original Source Code
add_dir_name = "#{order.priority}_#{order.orderno}_#{company_name[0, 15]}"
Zip::File.open_buffer(obj) do |zip|
zip.each do |entry|
ext = File.extname(entry.name)
file_name = File.basename(entry.name)
dir_name = File.dirname(entry.name)
next if ext.blank? || file_name.count(".") > 1
dir = File.join(add_dir_name, dir_name)
FileUtils.mkpath(dir.to_s)
zip.extract(entry, dir + ext) {true}
file_name.force_encoding("UTF-8")
new_file_name = "#{dir}/#{file_name}"
new_file_name.force_encoding("UTF-8")
File.rename(dir + ext, new_file_name)
#input_dir << new_file_name
end
end
What I tried
I added this method to the source code, but it did not work.
new_dir = Dir.glob(dir_name+"/*").last
demdem = new_dir.split('/').last
new_file_name = dir.sub("/" + demdem, "")
Here's a generic function to move all files (recursively) from one dir to another:
orig_dir = "2_J000001719_1202テスト/output"
new_dir = "2_J000001719_1202テスト"
def move_all_files(orig_dir, new_dir)
Dir.glob("#{orig_dir}/**/*").each do |path|
File.move(path, path.gsub(orig_dir, new_dir))
end
end
I am writing my Hashmap into a csv file with headers using the below piece of code. However, what I notice is that the first data row is not available in the file. I can the headers and all other rows accurately
def self.execute_auto_pg_debtors pg_debtors_dt_list
partition_key = Date.today.prev_day.prev_day.strftime "%Y%m%d"
csvfilename = "PG_Debtors_" + partition_key + ".CSV"
pg_debtors_dt_batch = Array.new
rowid = 0
pg_debtors_dt_list.each { |x|
pg_debtors_details = Hash.new
pg_debtors_details["Store_Order_Id"] = x['Store_Order_Id']
pg_debtors_details["Transaction_Id"] = x['Transaction_Id']
pg_debtors_details["Gateway_Payment_Id"] = x['Gateway_Payment_Id']
pg_debtors_details["PPS_Id"] = x['PPS_Id']
pg_debtors_details["Event_Type"] = x['Event_Type']
pg_debtors_details["Event_Date"] = x['Event_Date']
pg_debtors_details["Gateway_Name"] = x['Gateway_Name']
pg_debtors_details["Open_Amount"] = "%f" % x['Open_Amount']
pg_debtors_details["Invoice_No"] = x['Invoice_No']
pg_debtors_dt_batch << pg_debtors_details
rowid += 1
if rowid == 1
CSV.open(csvfilename, "w") do |csv|
csv << pg_debtors_details.keys# adding header row (column labels)
end
else
CSV.open(csvfilename, "a") do |csv|
csv << pg_debtors_details.values# of if/else inside hsh
end# of hsh's (rows)
end# of csv open
}
return pg_debtors_dt_batch
end
Please help.
You are writing the headers instead of the first row!
I recommend, that you open the file and iterate through your hash inside the CSV.open do ... end
AND do not use a else after your if rowid == 1. Just execute that for EVERY values, so you do not skip data row 1
Even if you check for rowid, the .each loop is still not aware of it. So, for rowid == 1, it will write the headers, but in the next iteration, x will point to the second item in pg_debtors_dt_list.
To solve it, write your code in the following order:
Open the file, and write the headers first.
Loop through pg_debtors_dt_list, and write subsequent data to the file.
Hope it helps.
CSV parsing of the file was very slow so I was trying to load the file directly in to some temp table in database directly and then doing the computation as below :
Earlier it was like this, took 13 mins to add the entries using below method :
CSV.foreach(fileName) do |line|
completePath = line[0]
num_of_bps = line[1]
completePath = cluster_path+ '/' + completePath
inode = FileOrFolder.find_by_fullpath(completePath, :select=>"id")
metric_instance = MetricInstance.find(:first, :conditions=>["file_or_folder_id = ? AND dataset_id = ?", inode.id, dataset_id])
add_entry(metric_instance.id, num_of_bps, num_of_bp_tests)
end
def self.add_entry(metaid, num_of_bps, num_of_bp_tests)
entry = Bp.new
entry.metric_instance_id = metaid
entry.num_of_bps = num_of_bps
entry.num_of_bp_tests = num_of_bp_tests
entry.save
return entry
end
now I changed the method to this, now takes 52 mins :(
#bps = TempTable.all
#bps.each do |bp|
completePath = bp.first_column
num_of_bps = bp.second_column
num_of_bps3 = bp.third_column
completePath = cluster_path+ '/' + completePath
inode = FileOrFolder.find_by_fullpath(completePath, :select=>"id")
num_of_bp_tests = 0
if(inode.nil?)
else
if(num_of_bps !='0')
num_of_bp_tests = 1
end
metric_instance = MetricInstance.find(:first, :conditions=>["file_or_folder_id = ? AND dataset_id = ?", inode.id, dataset_id])
add_entry(metric_instance.id, num_of_bps, num_of_bp_tests)
end
end
Please help me optimize this code or let me know if you think CSV.each is faster than database read !
When you load csv into database you do:
load N csv lines
insert N records int DB
select and instantiate N active record models
iterate over its
When you work with raw csv you only
load N csv lines
iterate over its
Of course it's faster.
Please help me in solving this error.
I am getting this error while loading records from text files in to database using ruby scripts.
It just works fine if I use small number of records to load in to the database.But fails if there are large number of records.
CSV.foreach(fileName) do |line|
completePath = line[0]
num_of_bps = line[1]
completePath = cluster_path+ '/' + completePath
inode = FileOrFolder.find_by_fullpath(completePath, :select=>"id")
metric_instance = MetricInstance.find(:first, :conditions=>["file_or_folder_id = ? AND dataset_id = ?", inode.id, dataset_id])
add_entry(metric_instance.id, num_of_bps, num_of_bp_tests)
end
def self.add_entry(metaid, num_of_bps, num_of_bp_tests)
entry = Bp.new
entry.metric_instance_id = metaid
entry.num_of_bps = num_of_bps
entry.num_of_bp_tests = num_of_bp_tests
entry.save
return entry
end
Try something like this:
File.open(fileName) do |csv|
csv.each_line do |line|
CSV.parse(line) do |values|
# Here you can do your manipulation
end
end
end
This way is slower, but it should ensure you don't get out of memory.