Ruby LoadError issue with 'require' - ruby-on-rails

I have researched this for quite some time, and have yet to solve my issue. Here is the error that I am receiving:
C:/Ruby23/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- nexpose-runner/constants (LoadError)
from C:/Ruby23/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in require'
from scan.rb:4:in `<main>'
Here is my code:
require 'nexpose'
require 'csv'
require 'json'
require 'nexpose-runner/constants'
require 'nexpose-runner/scan_run_description'
module NexposeRunner
module Scan
def Scan.start(options)
run_details = ScanRunDescription.new(options)
run_details.verify
nsc = get_new_nexpose_connection(run_details)
site = create_site(run_details, nsc)
start_scan(nsc, site, run_details)
reports = generate_reports(nsc, site, run_details)
verify_run(reports[0])
end
def self.generate_reports(nsc, site, run_details)
puts "Scan complete for #{run_details.site_name}, Generating Vulnerability Report"
vulnerbilities = generate_report(CONSTANTS::VULNERABILITY_REPORT_QUERY, site.id, nsc)
generate_csv(vulnerbilities, CONSTANTS::VULNERABILITY_REPORT_NAME)
puts "Scan complete for #{run_details.site_name}, Generating Vulnerability Detail Report"
vuln_details = generate_report(CONSTANTS:: VULNERABILITY_DETAIL_REPORT_QUERY, site.id, nsc)
generate_csv(vuln_details, CONSTANTS::VULNERABILITY_DETAIL_REPORT_NAME)
puts "Scan complete for #{run_details.site_name}, Generating Software Report"
software = generate_report(CONSTANTS::SOFTWARE_REPORT_QUERY, site.id, nsc)
generate_csv(software, CONSTANTS::SOFTWARE_REPORT_NAME)
puts "Scan complete for #{run_details.site_name}, Generating Policy Report"
policies = generate_report(CONSTANTS::POLICY_REPORT_QUERY, site.id, nsc)
generate_csv(policies, CONSTANTS::POLICY_REPORT_NAME)
puts "Scan complete for #{run_details.site_name}, Generating Audit Report"
generate_template_report(nsc, site.id, CONSTANTS::AUDIT_REPORT_FILE_NAME, CONSTANTS::AUDIT_REPORT_NAME, CONSTANTS::AUDIT_REPORT_FORMAT)
puts "Scan complete for #{run_details.site_name}, Generating Xml Report"
generate_template_report(nsc, site.id, CONSTANTS::XML_REPORT_FILE_NAME, CONSTANTS::XML_REPORT_NAME, CONSTANTS::XML_REPORT_FORMAT)
[vulnerbilities, software, policies]
end
def self.verify_run(vulnerabilities)
raise StandardError, CONSTANTS::VULNERABILITY_FOUND_MESSAGE if vulnerabilities.count > 0
end
def self.start_scan(nsc, site, run_details)
puts "Starting scan for #{run_details.site_name} using the #{run_details.scan_template} scan template"
scan = site.scan nsc
begin
sleep(3)
stats = nsc.scan_statistics(scan.id)
status = stats.status
puts "Current #{run_details.site_name} scan status: #{status.to_s} -- PENDING: #{stats.tasks.pending.to_s} ACTIVE: #{stats.tasks.active.to_s} COMPLETED #{stats.tasks.completed.to_s}"
end while status == Nexpose::Scan::Status::RUNNING
end
def self.create_site(run_details, nsc)
puts "Creating a nexpose site named #{run_details.site_name}"
site = Nexpose::Site.new run_details.site_name, run_details.scan_template
run_details.ip_addresses.each { |address|
site.add_ip address
}
if run_details.engine
site.engine = run_details.engine
end
site.save nsc
puts "Created site #{run_details.site_name} successfully with the following host(s) #{run_details.ip_addresses.join(', ')}"
site
end
def self.get_new_nexpose_connection(run_details)
nsc = Nexpose::Connection.new run_details.connection_url, run_details.username, run_details.password, run_details.port
nsc.login
puts 'Successfully logged into the Nexpose Server'
nsc
end
def self.generate_report(sql, site, nsc)
report = Nexpose::AdhocReportConfig.new(nil, 'sql')
report.add_filter('version', '1.3.0')
report.add_filter('query', sql)
report.add_filter('site', site)
report_output = report.generate(nsc)
CSV.parse(report_output.chomp, {:headers => :first_row})
end
def self.generate_template_report(nsc, site, file_name, report_name, report_format)
adhoc = Nexpose::AdhocReportConfig.new(report_name, report_format, site)
data = adhoc.generate(nsc)
File.open(file_name, 'w') { |file| file.write(data) }
end
def self.generate_csv(csv_output, name)
CSV.open(name, 'w') do |csv_file|
csv_file << csv_output.headers
csv_output.each do |row|
csv_file << row
if name == CONSTANTS::VULNERABILITY_REPORT_NAME
puts '--------------------------------------'
puts "IP: #{row[0]}"
puts "Vulnerability: #{row[1]}"
puts "Date Vulnerability was Published: #{row[2]}"
puts "Severity: #{row[3]}"
puts "Summary: #{row[4]}"
puts '--------------------------------------'
end
end
end
end
end
end
In the command prompt, I am entering in the following code to run it (this file is called scan.rb):
ruby scan.rb "http://localhost:3780" "username" "password" "3780" "webpage" "ip-address" "full-audit-widget-corp"
So far, I've tried changing require to require_relative, as well as re-arranging the paths (like putting the whole path, for example). Neither has worked.
I also made sure to have the Ruby Development Kit installed.
Thanks!

please check the local gem list: gem list --local

Related

Pivotal tracker story is not getting updated after code push

We had pivotal tracker integrated with GitHub and when we used to commit with story id, it used to update the story with a commit message update story status.
We recently migrated to Team Foundation Server from GitHub and that integration is not working anymore.
Looks like there is no integration App exists yet.
Is there a programmatic way of doing it?
Create file named pre-push (without any extension) and put it under ./.git/hooks/ folder inside your repo. This folder should already exist if it's a valid repo. Copy paste following code in the file. Don't forget to replace the API token value in the following code -
#!/usr/bin/env ruby
# encoding: UTF-8
require 'net/https'
require 'json'
class GitLogs
attr_accessor :raw_log, :commit_data, :commit, :author, :message, :refs
def initialize
self.commit = 'Commit: ' + `git log -1 --pretty=format:%h`.force_encoding('utf-8')
self.author = 'Author: ' + `git log -1 --pretty=format:%aN`.force_encoding('utf-8')
self.message = 'Message: ' + `git log -1 --pretty=format:%s`.force_encoding('utf-8')
self.refs = 'Refs: ' + `git log -1 --pretty=format:%d`.force_encoding('utf-8')
# Example git formatted log output:
#
# Commit: 8872e8fe03a10238d7be84d78813874d79ce0c3d
# Author: John Doe <john.doe#unknown.com>
# Message: [#90743834] test new v5 hook addition
# Refs: (HEAD, feature/hook-test)
parse!
self
end
def parse!
self.commit_data = GitLog.new(self.commit, self.author, self.message, self.refs)
end
def pivotal_sync!
Pivotal.new(commit_data).send! if commit_has_story_id?
end
def commit_has_story_id?
# somewhere between square brackets there has to be a hash followed by multiple digits
!commit_data.message.scan(/\[*+\#(\d+)\D?(.*)\]/).empty?
end
end
class GitLog
attr_accessor :hash, :author, :message, :refs
def initialize hash, author, message, refs
self.hash = hash
self.author = author
self.refs = refs
self.message = message
updated_message
end
def updated_message
return message
end
def to_json
{ source_commit:
{ commit_id: self.hash,
author: self.author,
message: self.message,
}
}.to_json
end
end
class Pivotal
attr_accessor :git_log, :tracker_token
BASE_URI = URI('https://www.pivotaltracker.com/')
def initialize git_log
self.git_log = git_log
self.tracker_token = get_token
end
def get_token
'YOUR APT TOKEN GOES HERE. CAN GET IT FROM https://www.pivotaltracker.com/profile'
end
def send!
https = Net::HTTP.start(BASE_URI.host, 443, {
use_ssl: true,
verify_mode: OpenSSL::SSL::VERIFY_NONE
})
request = Net::HTTP::Post.new('/services/v5/source_commits')
request['X-TrackerToken'] = tracker_token
request['Content-type'] = 'application/json'
request.body = git_log.to_json
response = https.request(request)
end
end
GitLogs.new().pivotal_sync!

Is it possible to access the AdSense API using ruby?

I am trying to access the AdSense Management API using ruby. They recommend using their generic Google-API client library:
http://code.google.com/p/google-api-ruby-client/#Google_AdSense_Management_API
This hasn't been very helpful and I have run into errors:
Faraday conflicts in google_drive and google-api-client
Where should I start in order to get access to my AdSense data?
Thanks in advance.
Unfortunately, we haven't prepared any sample code for the AdSense Management API... yet! As you point out, though, the client library is generic, and should work with any of the newer Google APIs, so some of the other samples may help.
If you're running into any specific issues, please create a question focused on those and point me to it, and I'll do my best to help.
If you want a quick sample to get started, I can cook that up for you, but we should make sure the issues you're running into have to do with the AdSense Management API itself, and not just the client library, as the one you were linking to.
[Edit]
Here's a quick sample, based on Sinatra:
#!/usr/bin/ruby
require 'rubygems'
require 'sinatra'
require 'google/api_client'
FILENAME = 'auth.obj'
OAUTH_CLIENT_ID = 'INSERT_OAUTH2_CLIENT_ID_HERE'
OAUTH_CLIENT_SECRET = 'INSERT_OAUTH2_CLIENT_SECRET_HERE'
before do
#client = Google::APIClient.new
#client.authorization.client_id = OAUTH_CLIENT_ID
#client.authorization.client_secret = OAUTH_CLIENT_SECRET
#client.authorization.scope = 'https://www.googleapis.com/auth/adsense'
#client.authorization.redirect_uri = to('/oauth2callback')
#client.authorization.code = params[:code] if params[:code]
# Load the access token here if it's available
if File.exist?(FILENAME)
serialized_auth = IO.read(FILENAME)
#client.authorization = Marshal::load(serialized_auth)
end
if #client.authorization.refresh_token && #client.authorization.expired?
#client.authorization.fetch_access_token!
end
#adsense = #client.discovered_api('adsense', 'v1.1')
unless #client.authorization.access_token || request.path_info =~ /^\/oauth2/
redirect to('/oauth2authorize')
end
end
get '/oauth2authorize' do
redirect #client.authorization.authorization_uri.to_s, 303
end
get '/oauth2callback' do
#client.authorization.fetch_access_token!
# Persist the token here
serialized_auth = Marshal::dump(#client.authorization)
File.open(FILENAME, 'w') do |f|
f.write(serialized_auth)
end
redirect to('/')
end
get '/' do
call = {
:api_method => #adsense.reports.generate,
:parameters => {
'startDate' => '2011-01-01',
'endDate' => '2011-08-31',
'dimension' => ['MONTH', 'CUSTOM_CHANNEL_NAME'],
'metric' => ['EARNINGS', 'TOTAL_EARNINGS']
}
}
response = #client.execute(call)
output = ''
if response && response.data && response.data['rows'] &&
!response.data['rows'].empty?
result = response.data
output << '<table><tr>'
result['headers'].each do |header|
output << '<td>%s</td>' % header['name']
end
output << '</tr>'
result['rows'].each do |row|
output << '<tr>'
row.each do |column|
output << '<td>%s</td>' % column
end
output << '</tr>'
end
output << '</table>'
else
output << 'No rows returned'
end
output
end

Downloading and zipping files that were uploaded to S3 with CarrierWave

I have a small Rails 3.2.1 app that uses CarrierWave 0.5.8 for file uploads to S3 (using Fog)
I want users to be able to select some images that they'd like to download, then zip them up and send them a zip. Here is what I've come up with:
def generate_zip
#A collection of Photo objects. The Photo object has a PhotoUploader mounted.
photos = Photo.all
tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
zip.close
photos.each do |photo|
file_to_add = photo.photo.file
zip = Zip::ZipFile.open(tmp_filename)
zip.add("tmp/", file_to_add.path)
zip.close
end
#do the rest.. like send zip or upload file and e-mail link
end
This doesn't work because photo.photo.file returns an instance of CarrierWave::Storage::Fog::File instead of a regular file.
EDIT: The error this leads to:
Errno::ENOENT: No such file or directory - uploads/photos/name.jpg
I also tried the following:
tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
zip.close
photos.each do |photo|
processed_uri = URI.parse(URI.escape(URI.unescape(photo.photo.file.authenticated_url)).gsub("[", "%5B").gsub("]", "%5D"))
file_to_add = CarrierWave::Uploader::Download::RemoteFile.new(processed_uri)
zip = Zip::ZipFile.open(tmp_filename)
zip.add("tmp/", file_to_add.path)
zip.close
end
But this gives me a 403. Some help would be greatly appreciated.. It probably is not that hard I'm just Doing it Wrong™
I've managed to solve the problem with help from #ffoeg
The solution offered by #ffoeg didn't work quite so well for me since I was dealing with zip files > 500 MB which caused me problems on Heroku. I've therefor moved the zipping to a background process using resque:
app/workers/photo_zipper.rb:
require 'zip/zip'
require 'zip/zipfilesystem'
require 'open-uri'
class PhotoZipper
#queue = :photozip_queue
#I pass
def self.perform(id_of_object_with_images, id_of_user_to_be_notified)
user_mail = User.where(:id => id_of_user_to_be_notified).pluck(:email)
export = PhotoZipper.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
Notifications.zip_ready(export.archive_url, user_mail).deliver
end
# Zipfile generator
def self.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
object = ObjectWithImages.find(id_of_object_with_images)
photos = object.images
# base temp dir
temp_dir = Dir.mktmpdir
# path for zip we are about to create, I find that ruby zip needs to write to a real file
# This assumes the ObjectWithImages object has an attribute title which is a string.
zip_path = File.join(temp_dir, "#{object.title}_#{Date.today.to_s}.zip")
Zip::ZipOutputStream.open(zip_path) do |zos|
photos.each do |photo|
path = photo.photo.path
zos.put_next_entry(path)
zos.write photo.photo.file.read
end
end
#Find the user that made the request
user = User.find(id_of_user_to_be_notified)
#Create an export object associated to the user
export = user.exports.build
#Associate the created zip to the export
export.archive = File.open(zip_path)
#Upload the archive
export.save!
#return the export object
export
ensure
# clean up the tempdir now!
FileUtils.rm_rf temp_dir if temp_dir
end
end
app/controllers/photos_controller.rb:
format.zip do
#pick the last ObjectWithImages.. ofcourse you should include your own logic here
id_of_object_with_images = ObjectWithImages.last.id
#enqueue the Photozipper task
Resque.enqueue(PhotoZipper, id_of_object_with_images, current_user.id)
#don't keep the user waiting and flash a message with information about what's happening behind the scenes
redirect_to some_path, :notice => "Your zip is being created, you will receive an e-mail once this process is complete"
end
Many thanks to #ffoeg for helping me out. If your zips are smaller you could try #ffoeg's solution.
Here is my take. There could be typos but I think this is the gist of it :)
# action method, stream the zip
def download_photos_as_zip # silly name but you get the idea
generate_zip do |zipname, zip_path|
File.open(zip_path, 'rb') do |zf|
# you may need to set these to get the file to stream (if you care about that)
# self.last_modified
# self.etag
# self.response.headers['Content-Length']
self.response.headers['Content-Type'] = "application/zip"
self.response.headers['Content-Disposition'] = "attachment; filename=#{zipname}"
self.response.body = Enumerator.new do |out| # Enumerator is ruby 1.9
while !zf.eof? do
out << zf.read(4096)
end
end
end
end
end
# Zipfile generator
def generate_zip(&block)
photos = Photo.all
# base temp dir
temp_dir = Dir.mktempdir
# path for zip we are about to create, I find that ruby zip needs to write to a real file
zip_path = File.join(temp_dir, 'export.zip')
Zip::ZipFile::open(zip_path, true) do |zipfile|
photos.each do |photo|
zipfile.get_output_stream(photo.photo.identifier) do |io|
io.write photo.photo.file.read
end
end
end
# yield the zipfile to the action
block.call 'export.zip', zip_path
ensure
# clean up the tempdir now!
FileUtils.rm_rf temp_dir if temp_dir
end

Getting blobs with Grit

I'm using Grit in my rails app and I'm creating a commit which i know works:
repo = Repo.new(full_path, {:is_bare => true})
fname = "snippet"
File.open("#{full_path}/#{fname}", 'w') {|f| f.puts(data)}
Dir.chdir("#{full_path}") {repo.add(fname)}
if repo.commit_index('his amazing commit')
logger.info "commit succeeded"
else
logger.info "commit failed"
end
then, im trying to get the blobs which is showing up empty:
tree = Tree.construct(repo, 'master')
data = tree.blobs.map {|b| repo.blob(b.id).data}
logger.info "data.first = #{data.first}"
data.first
What am I doing wrong here?
I guess you have no file in the root level in your repository.
tree.blobs returns files of the root level, and tree.trees returns directories. To get all files in the repository, you need to traverse the tree recursively.
I wrote some example:
require 'grit'
def traverse(tree, basename)
tree.blobs.each do |blob|
puts "#{basename}/#{blob.basename}"
end
tree.trees.each do |subtree|
traverse(subtree, "#{basename}/#{subtree.basename}")
end
end
repo = Grit::Repo.new('.')
root = Grit::Tree.construct(repo, 'master')
traverse(root, '')

How to grep file name and extensions in webpage using nokogiri/hpricot and other gem?

I am working on an application where I have to
1) get all the links of website
2) and then get the list of all the files and file extensions in each
of the web page/link.
I am done with the first part of it :)
I get all the links of website by below code..
require 'rubygems'
require 'spidr'
require 'uri'
Spidr.site('http://testasp.vulnweb.com/') do |spider|
spider.every_url { |url|
puts url
}
end
now I have to get the all the files/file-extensions in each of the
page so I tried the below code
require 'rubygems'
require 'nokogiri'
require 'open-uri'
require 'spidr'
site = 'http://testasp.vulnweb.com'
in1=[]
Spidr.site(site) do |spider|
spider.every_url { |url| in1.push url }
end
in1.each do |input1|
input1 = input1.to_s
#puts input1
begin
doc = Nokogiri::HTML(open(input1))
doc.traverse do |el|
[el[:src], el[:href]].grep(/\.(txt|css|gif|jpg|png|pdf)$/i).map{|l| URI.join(input1, l).to_s}.each do |link|
puts link
end
end
rescue => e
puts "errrooooooooor"
end
end
but Can anybody guide me how to parse the links/webpage and get the file-
extensions in the page?
You might want to take a look at URI#parse. The URI module is a part of the Ruby standard library and is a dependency of the spidr gem. Example implementation with a spec for good measure.
require 'rspec'
require 'uri'
class ExtensionExtractor
def extract(uri)
/\A.*\/(?<file>.*\.(?<extension>txt|css|gif|jpg|png|pdf))\z/i =~ URI.parse(uri).path
{:path => uri, :file => file, :extension => extension}
end
end
describe ExtensionExtractor do
before(:all) do
#css_uri = "http://testasp.vulnweb.com/styles.css"
#gif_uri = "http://testasp.vulnweb.com/Images/logo.gif"
#gif_uri_with_param = "http://testasp.vulnweb.com/Images/logo.gif?size=350x350"
end
describe "Common Extensions" do
it "should extract CSS files from URIs" do
file = subject.extract(#css_uri)
file[:path].should eq #css_uri
file[:file].should eq "styles.css"
file[:extension].should eq "css"
end
it "should extract GIF files from URIs" do
file = subject.extract(#gif_uri)
file[:path].should eq #gif_uri
file[:file].should eq "logo.gif"
file[:extension].should eq "gif"
end
it "should properly extract extensions even when URIs have parameters" do
file = subject.extract(#gif_uri_with_param)
file[:path].should eq #gif_uri_with_param
file[:file].should eq "logo.gif"
file[:extension].should eq "gif"
end
end
end

Resources