I'm using Grit in my rails app and I'm creating a commit which i know works:
repo = Repo.new(full_path, {:is_bare => true})
fname = "snippet"
File.open("#{full_path}/#{fname}", 'w') {|f| f.puts(data)}
Dir.chdir("#{full_path}") {repo.add(fname)}
if repo.commit_index('his amazing commit')
logger.info "commit succeeded"
else
logger.info "commit failed"
end
then, im trying to get the blobs which is showing up empty:
tree = Tree.construct(repo, 'master')
data = tree.blobs.map {|b| repo.blob(b.id).data}
logger.info "data.first = #{data.first}"
data.first
What am I doing wrong here?
I guess you have no file in the root level in your repository.
tree.blobs returns files of the root level, and tree.trees returns directories. To get all files in the repository, you need to traverse the tree recursively.
I wrote some example:
require 'grit'
def traverse(tree, basename)
tree.blobs.each do |blob|
puts "#{basename}/#{blob.basename}"
end
tree.trees.each do |subtree|
traverse(subtree, "#{basename}/#{subtree.basename}")
end
end
repo = Grit::Repo.new('.')
root = Grit::Tree.construct(repo, 'master')
traverse(root, '')
Related
I have a data backup system for customers of my app. I gather up all associated csv files and zip them. Once that zip file is complete, I attach it in an email. This process breaks on heroku due to their file system. I thought since heroku-16 we could write to the app/tmp directory and that this process might occur within the same transaction and the files would be fine, but that doesn't seem to be the case. I don't even seem to be writing the files to the tmp directory in production (in Dev I am).
So, what I would like to do instead is just write the csv files directly to S3, then Zip those files and also save the .zip to S3...then, pull that file as an email attachment. To do this, I need to generate the csv files and write them to S3 from inside ActiveJob. I use S3 already as part of ActiveStorage, but this process will not utilize ActiveStorage.
Is there's a command for me to manually direct upload to an S3 bucket. I've been digging around in the docs, etc but don't see what I'm after.
The Job (using /tmp)
def perform(company_id, recipient_id)
company = Company.find(company_id)
source_folder = "#{ Rails.root }/tmp"
zipfile_name = "company_#{ company.id }_archive.zip"
zipfile_path = "#{ Rails.root }/tmp/#{ zipfile_name }"
input_filenames = []
# USERS: create a new empty csv file,
# ... then add rows to it
# ... and, add the file name to the list of files array
users_file_name = "#{ company.name.parameterize.underscore }_users_list.csv"
input_filenames << users_file_name
users_csv_file = File.new("#{ Rails.root.join('tmp') }/#{ users_file_name }", 'w')
users_csv_file << company.users.to_csv
users_csv_file.close
...
# gather up the created files and zip them
Zip::File.open(zipfile_path, create: true) do |zipfile|
input_filenames.uniq.each do |filename|
zipfile.add(filename, File.join(source_folder, filename))
end
end
puts "attaching data_export".colorize(:red)
company.data_exports.attach(
io: StringIO.new("#{ Rails.root }/tmp/company_14_#{ Time.current.to_date.to_s }_archive.zip"),
filename: 'company_14_archive.zip',
content_type: 'application/zip'
)
last_id = company.data_exports.last.id
puts "sending mail using company.id: #{ company.id }, recipient_id: #{ recipient_id }, company.data_exports.last.id: #{ last_id }".colorize(:red)
CompanyMailer.mail_data_export(
company.id,
recipient_id,
last_id
)
end
You can upload file like this on S3
key = "file_name.zip"
file_path = "tmp/file_name.zip"
new_s3_client = Aws::S3::Resource.new(region: 'eu-west-1', access_key_id: '123', secret_access_key: '456')
new_bucket = new_s3_client.bucket('public')
obj = new_bucket.object(key)
obj.upload_file(file_path)
For a variety of reasons I am migrating my uploads from ActiveStorage (AS) to CarrierWave (CW).
I am making rake task and have the logic sorted out - I am stumped at how to feed the AS blob into the CW file.
I am trying something like ths:
#files.each.with_index(1) do | a, index |
if a.attachment.attached?
a.attachment.download do |file|
a.file = file
end
a.save!
end
end
This is based on these two links:
https://edgeguides.rubyonrails.org/active_storage_overview.html#downloading-files
message.video.open do |file|
system '/path/to/virus/scanner', file.path
# ...
end
and
https://github.com/carrierwaveuploader/carrierwave#activerecord
# like this
File.open('somewhere') do |f|
u.avatar = f
end
I tested this locally and the files are not mounted via the uploader. My question(s) would be:
am I missing something obvious here?
is my approach wrong and needs a new one?
Bonus Karma Question:
I can't seem to see a clear path to set the CW filename when I do this?
Here is my final rack task (based on the accepted answer) - open to tweaks. Does the job for me:
namespace :carrierwave do
desc "Import the old AS files into CW"
task import: :environment do
#files = Attachment.all
puts "#{#files.count} files to be processed"
puts "+" * 50
#files.each.with_index(1) do | a, index |
if a.attachment.attached?
puts "Attachment #{index}: Key: #{a.attachment.blob.key} ID: #{a.id} Filename: #{a.attachment.blob.filename}"
class FileIO < StringIO
def initialize(stream, filename)
super(stream)
#original_filename = filename
end
attr_reader :original_filename
end
a.attachment.download do |file|
a.file = FileIO.new(file, a.attachment.blob.filename.to_s)
end
a.save!
puts "-" * 50
end
end
end
desc "Purge the old AS files"
task purge: :environment do
#files = Attachment.all
puts "#{#files.count} files to be processed"
puts "+" * 50
#files.each.with_index(1) do | a, index |
if a.attachment.attached?
puts "Attachment #{index}: Key: #{a.attachment.blob.key} ID: #{a.id} Filename: #{a.attachment.blob.filename}"
a.attachment.purge
puts "-" * 50
#count = index
end
end
puts "#{#count} files purged"
end
end
Now in my case I am doing this in steps - I have branched my master with this rake task and the associated MCV updates. If my site was in true production would probably run the import rake task first then confirm all went well THEN purge the old AS files.
The file object you get from the attachment.download block is a string. More precisely, the response from .download is the file, "streamed and yielded in chunks" (see documentation). I validated this by calling file.class to make sure the class is what I expected.
So, to solve your issue, you need to provide an object on which .read can be called. Commonly that is done using the Ruby StringIO class.
However, considering Carrierwave also expects a filename, you can solve it using a helper model that inherits StringIO (from blogpost linked above):
class FileIO < StringIO
def initialize(stream, filename)
super(stream)
#original_filename = filename
end
attr_reader :original_filename
end
And then you can replace a.file = file with a.file = FileIO.new(file, 'new_filename')
I have researched this for quite some time, and have yet to solve my issue. Here is the error that I am receiving:
C:/Ruby23/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- nexpose-runner/constants (LoadError)
from C:/Ruby23/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in require'
from scan.rb:4:in `<main>'
Here is my code:
require 'nexpose'
require 'csv'
require 'json'
require 'nexpose-runner/constants'
require 'nexpose-runner/scan_run_description'
module NexposeRunner
module Scan
def Scan.start(options)
run_details = ScanRunDescription.new(options)
run_details.verify
nsc = get_new_nexpose_connection(run_details)
site = create_site(run_details, nsc)
start_scan(nsc, site, run_details)
reports = generate_reports(nsc, site, run_details)
verify_run(reports[0])
end
def self.generate_reports(nsc, site, run_details)
puts "Scan complete for #{run_details.site_name}, Generating Vulnerability Report"
vulnerbilities = generate_report(CONSTANTS::VULNERABILITY_REPORT_QUERY, site.id, nsc)
generate_csv(vulnerbilities, CONSTANTS::VULNERABILITY_REPORT_NAME)
puts "Scan complete for #{run_details.site_name}, Generating Vulnerability Detail Report"
vuln_details = generate_report(CONSTANTS:: VULNERABILITY_DETAIL_REPORT_QUERY, site.id, nsc)
generate_csv(vuln_details, CONSTANTS::VULNERABILITY_DETAIL_REPORT_NAME)
puts "Scan complete for #{run_details.site_name}, Generating Software Report"
software = generate_report(CONSTANTS::SOFTWARE_REPORT_QUERY, site.id, nsc)
generate_csv(software, CONSTANTS::SOFTWARE_REPORT_NAME)
puts "Scan complete for #{run_details.site_name}, Generating Policy Report"
policies = generate_report(CONSTANTS::POLICY_REPORT_QUERY, site.id, nsc)
generate_csv(policies, CONSTANTS::POLICY_REPORT_NAME)
puts "Scan complete for #{run_details.site_name}, Generating Audit Report"
generate_template_report(nsc, site.id, CONSTANTS::AUDIT_REPORT_FILE_NAME, CONSTANTS::AUDIT_REPORT_NAME, CONSTANTS::AUDIT_REPORT_FORMAT)
puts "Scan complete for #{run_details.site_name}, Generating Xml Report"
generate_template_report(nsc, site.id, CONSTANTS::XML_REPORT_FILE_NAME, CONSTANTS::XML_REPORT_NAME, CONSTANTS::XML_REPORT_FORMAT)
[vulnerbilities, software, policies]
end
def self.verify_run(vulnerabilities)
raise StandardError, CONSTANTS::VULNERABILITY_FOUND_MESSAGE if vulnerabilities.count > 0
end
def self.start_scan(nsc, site, run_details)
puts "Starting scan for #{run_details.site_name} using the #{run_details.scan_template} scan template"
scan = site.scan nsc
begin
sleep(3)
stats = nsc.scan_statistics(scan.id)
status = stats.status
puts "Current #{run_details.site_name} scan status: #{status.to_s} -- PENDING: #{stats.tasks.pending.to_s} ACTIVE: #{stats.tasks.active.to_s} COMPLETED #{stats.tasks.completed.to_s}"
end while status == Nexpose::Scan::Status::RUNNING
end
def self.create_site(run_details, nsc)
puts "Creating a nexpose site named #{run_details.site_name}"
site = Nexpose::Site.new run_details.site_name, run_details.scan_template
run_details.ip_addresses.each { |address|
site.add_ip address
}
if run_details.engine
site.engine = run_details.engine
end
site.save nsc
puts "Created site #{run_details.site_name} successfully with the following host(s) #{run_details.ip_addresses.join(', ')}"
site
end
def self.get_new_nexpose_connection(run_details)
nsc = Nexpose::Connection.new run_details.connection_url, run_details.username, run_details.password, run_details.port
nsc.login
puts 'Successfully logged into the Nexpose Server'
nsc
end
def self.generate_report(sql, site, nsc)
report = Nexpose::AdhocReportConfig.new(nil, 'sql')
report.add_filter('version', '1.3.0')
report.add_filter('query', sql)
report.add_filter('site', site)
report_output = report.generate(nsc)
CSV.parse(report_output.chomp, {:headers => :first_row})
end
def self.generate_template_report(nsc, site, file_name, report_name, report_format)
adhoc = Nexpose::AdhocReportConfig.new(report_name, report_format, site)
data = adhoc.generate(nsc)
File.open(file_name, 'w') { |file| file.write(data) }
end
def self.generate_csv(csv_output, name)
CSV.open(name, 'w') do |csv_file|
csv_file << csv_output.headers
csv_output.each do |row|
csv_file << row
if name == CONSTANTS::VULNERABILITY_REPORT_NAME
puts '--------------------------------------'
puts "IP: #{row[0]}"
puts "Vulnerability: #{row[1]}"
puts "Date Vulnerability was Published: #{row[2]}"
puts "Severity: #{row[3]}"
puts "Summary: #{row[4]}"
puts '--------------------------------------'
end
end
end
end
end
end
In the command prompt, I am entering in the following code to run it (this file is called scan.rb):
ruby scan.rb "http://localhost:3780" "username" "password" "3780" "webpage" "ip-address" "full-audit-widget-corp"
So far, I've tried changing require to require_relative, as well as re-arranging the paths (like putting the whole path, for example). Neither has worked.
I also made sure to have the Ruby Development Kit installed.
Thanks!
please check the local gem list: gem list --local
I cant figure out how to update/rename a file uploaded/managed with Carrierwave-mongoid in rails 3.2.6. I want to rename the file in the db as well as on the filesystem.
Something like this maybe...
def rename( id , new_name )
f = UploadedFile.find(id)
if f.update_attributes({ f.file.original_filename: new_name }) # this is WRONG, what is right???
new_path = File.join( File.dirname( f.file.current_path ) , new_name ))
FileUtils.mv( f.file.current_path , new_path )
end
return f
end
Let me add this is after it has been uploaded already.
I was able to get the following working, although I'm sure there is a more elegant way. I'd appreciate any comments on the following
*add this to app/uploaders/file_uploader.rb
def rename(new_name)
sf = model.file.file
new_path = File.join( File.dirname( sf.file ) , "#{new_name}#{File.extname( sf.file )}")
new_sf = CarrierWave::SanitizedFile.new sf.move_to(new_path)
model.file.cache!(new_sf)
model.save!
return model
end
Thanks!
The most efficient way to do this is to just move the existing S3 object (assuming your storage layer is S3):
def rename(new_name)
bucket_name = "yourapp-#{Rails.env}"
resource = Aws::S3::Resource.new
bucket = resource.bucket(bucket_name)
object = bucket.object(path)
new_filename = "#{new_name}#{File.extname(path)}"
new_path = File.join(File.dirname(path), new_filename)
object.move_to(bucket: bucket_name, key: new_path)
model.update_column(mounted_as, new_filename)
model.reload
# Now call `recreate_versions!(*versions.keys)`
# if you want versions updated. Explicitly passing
# versions will prevent the base version getting
# reuploaded.
model
end
This is using the aws-sdk-s3 gem.
I store image files -- and derivative versions -- in an S3-compatible solution. I use Carrierwave (1.2.2) with the "fog-aws" gem (3.0.0) on Rails 5.1. The following public method works for me when added to the "uploader" file (eg, app/uploaders/example_uploader.rb):
class ExampleUploader < CarrierWave::Uploader::Base
<snip>
# Renames original file and versions to match given filename
#
# Options:
# * +:keep_original+ - Do not remove original file and versions (ie, copy only)
def rename(new_filename, options = {})
return if !file || new_filename == file.filename
target = File.join(store_path, new_filename)
file.copy_to(target)
versions.keys.each do |k|
target = File.join(store_path, "#{k}_#{new_filename}")
version = send(k).file
version.copy_to(target)
end
remove! unless options[:keep_original]
model.update_column(mounted_as, new_filename) && model.reload
end
<snip>
end
I used this rake task for reprocessing uploaded images after modifying version settings (filename and image size) in my uploader file:
# Usage: rake carrierwave:reprocess class=Model
namespace :carrierwave do
task :reprocess => :environment do
CLASS = ENV['class'].capitalize
MODEL = Kernel.const_get(CLASS)
records = MODEL.all
records.each do |record|
record.photo.recreate_versions! if record.photo?
end
end
end
Notes:
Replace "photo" with whatever you named your uploader.
Rake tasks go in the lib/tasks folder.
This is using Active Record, not sure if Mongoid needs something
different.
Based on #user892583, I worked on it and came up with a simpler solution:
def rename!(new_name)
new_path = File.join(File.dirname(file.file), new_name)
file.move_to(new_path)
end
I did this with this way:
def filename
if !cached? && file.present?
new_filename = 'foobar'
new_path = File.join(File.dirname(file.path), new_filename)
file.move_to(new_path)
recreate_versions!
new_filename
else
super
end
end
I think this is only right way to rename file.
I have a small Rails 3.2.1 app that uses CarrierWave 0.5.8 for file uploads to S3 (using Fog)
I want users to be able to select some images that they'd like to download, then zip them up and send them a zip. Here is what I've come up with:
def generate_zip
#A collection of Photo objects. The Photo object has a PhotoUploader mounted.
photos = Photo.all
tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
zip.close
photos.each do |photo|
file_to_add = photo.photo.file
zip = Zip::ZipFile.open(tmp_filename)
zip.add("tmp/", file_to_add.path)
zip.close
end
#do the rest.. like send zip or upload file and e-mail link
end
This doesn't work because photo.photo.file returns an instance of CarrierWave::Storage::Fog::File instead of a regular file.
EDIT: The error this leads to:
Errno::ENOENT: No such file or directory - uploads/photos/name.jpg
I also tried the following:
tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
zip.close
photos.each do |photo|
processed_uri = URI.parse(URI.escape(URI.unescape(photo.photo.file.authenticated_url)).gsub("[", "%5B").gsub("]", "%5D"))
file_to_add = CarrierWave::Uploader::Download::RemoteFile.new(processed_uri)
zip = Zip::ZipFile.open(tmp_filename)
zip.add("tmp/", file_to_add.path)
zip.close
end
But this gives me a 403. Some help would be greatly appreciated.. It probably is not that hard I'm just Doing it Wrong™
I've managed to solve the problem with help from #ffoeg
The solution offered by #ffoeg didn't work quite so well for me since I was dealing with zip files > 500 MB which caused me problems on Heroku. I've therefor moved the zipping to a background process using resque:
app/workers/photo_zipper.rb:
require 'zip/zip'
require 'zip/zipfilesystem'
require 'open-uri'
class PhotoZipper
#queue = :photozip_queue
#I pass
def self.perform(id_of_object_with_images, id_of_user_to_be_notified)
user_mail = User.where(:id => id_of_user_to_be_notified).pluck(:email)
export = PhotoZipper.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
Notifications.zip_ready(export.archive_url, user_mail).deliver
end
# Zipfile generator
def self.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
object = ObjectWithImages.find(id_of_object_with_images)
photos = object.images
# base temp dir
temp_dir = Dir.mktmpdir
# path for zip we are about to create, I find that ruby zip needs to write to a real file
# This assumes the ObjectWithImages object has an attribute title which is a string.
zip_path = File.join(temp_dir, "#{object.title}_#{Date.today.to_s}.zip")
Zip::ZipOutputStream.open(zip_path) do |zos|
photos.each do |photo|
path = photo.photo.path
zos.put_next_entry(path)
zos.write photo.photo.file.read
end
end
#Find the user that made the request
user = User.find(id_of_user_to_be_notified)
#Create an export object associated to the user
export = user.exports.build
#Associate the created zip to the export
export.archive = File.open(zip_path)
#Upload the archive
export.save!
#return the export object
export
ensure
# clean up the tempdir now!
FileUtils.rm_rf temp_dir if temp_dir
end
end
app/controllers/photos_controller.rb:
format.zip do
#pick the last ObjectWithImages.. ofcourse you should include your own logic here
id_of_object_with_images = ObjectWithImages.last.id
#enqueue the Photozipper task
Resque.enqueue(PhotoZipper, id_of_object_with_images, current_user.id)
#don't keep the user waiting and flash a message with information about what's happening behind the scenes
redirect_to some_path, :notice => "Your zip is being created, you will receive an e-mail once this process is complete"
end
Many thanks to #ffoeg for helping me out. If your zips are smaller you could try #ffoeg's solution.
Here is my take. There could be typos but I think this is the gist of it :)
# action method, stream the zip
def download_photos_as_zip # silly name but you get the idea
generate_zip do |zipname, zip_path|
File.open(zip_path, 'rb') do |zf|
# you may need to set these to get the file to stream (if you care about that)
# self.last_modified
# self.etag
# self.response.headers['Content-Length']
self.response.headers['Content-Type'] = "application/zip"
self.response.headers['Content-Disposition'] = "attachment; filename=#{zipname}"
self.response.body = Enumerator.new do |out| # Enumerator is ruby 1.9
while !zf.eof? do
out << zf.read(4096)
end
end
end
end
end
# Zipfile generator
def generate_zip(&block)
photos = Photo.all
# base temp dir
temp_dir = Dir.mktempdir
# path for zip we are about to create, I find that ruby zip needs to write to a real file
zip_path = File.join(temp_dir, 'export.zip')
Zip::ZipFile::open(zip_path, true) do |zipfile|
photos.each do |photo|
zipfile.get_output_stream(photo.photo.identifier) do |io|
io.write photo.photo.file.read
end
end
end
# yield the zipfile to the action
block.call 'export.zip', zip_path
ensure
# clean up the tempdir now!
FileUtils.rm_rf temp_dir if temp_dir
end