I am using barby gem to generate the bar codes. I am getting this error on this type of strings i-e "105811/18". when I use this string like "105811_18" or "105811-18" it works fine and generate the barcode successfully but generating error on slash keyword, and pointing error on second last line of my method. "File.open(file_path, 'wb') { |f| f.write output }" Thanks for your help in advance.
require 'barby'
require 'barby/barcode/code_128'
require 'barby/outputter/png_outputter'
def self.get_bar_code(medical_record_no)
barcode_digits = medical_record_no.rjust(13, '0')
barcode = Barby::Code128.new(barcode_digits)
output = Barby::PngOutputter.new(barcode).to_png
file_path = File.join(Rails.root, 'public', "#{medical_record_no}_barcode.png")
File.open(file_path, 'wb') { |f| f.write output }
return barcode_digits
end
I want to download pictures from a URL, like: http://trinity.e-stile.ru/ and save images to a directory like "C:\pickaxe\pictures". It is important to use Nokogiri.
I read similar questions on this site, but I didn't find how it works and I didn't understand the algorithm.
I wrote the code where I parse the URL and put parts of the webpage source code with "img" tag into a links object:
require 'nokogiri'
require 'open-uri'
PAGE_URL="http://trinity.e-stile.ru/"
page=Nokogiri::HTML(open(PAGE_URL)) #parsing into object
links=page.css("img") #object with html code with img tag
puts links.length # it is 24 images on this url
puts
links.each{|i| puts i } #it looks like: <img border="0" alt="" src="/images/kroliku.jpg">
puts
puts
links.each{|link| puts link['src'] } #/images/kroliku.jpg
What method is used to save pictures after grabbing the HTML code?
How can I put the images into a directory on my disk?
I changed the code, but it has an error:
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/net/http.rb:879:in `initialize': getaddrinfo: Name or service not known (SocketError)
This is the code now:
require 'nokogiri'
require 'open-uri'
require 'net/http'
LOCATION = 'pics'
if !File.exist? LOCATION # create folder if it is not exist
require 'fileutils'
FileUtils.mkpath LOCATION
end
#PAGE_URL = "http://ruby.bastardsbook.com/files/hello-webpage.html"
#PAGE_URL="http://trinity.e-stile.ru/"
PAGE_URL="http://www.youtube.com/"
page=Nokogiri::HTML(open(PAGE_URL))
links=page.css("img")
links.each{|link|
Net::HTTP.start(PAGE_URL) do |http|
localname = link.gsub /.*\//, '' # left the filename only
resp = http.get link['src']
open("#{LOCATION}/#{localname}", "wb") do |file|
file.write resp.body
end
end
}
You are almost done. The only thing left is to store files. Let’s do it.
LOCATION = 'C:\pickaxe\pictures'
if !File.exist? LOCATION # create folder if it is not exist
require 'fileutils'
FileUtils.mkpath LOCATION
end
require 'net/http'
.... # your code with nokogiri etc.
links.each{|link|
Net::HTTP.start(PAGE_URL) do |http|
localname = link.gsub /.*\//, '' # left the filename only
resp = http.get link['src']
open("#{LOCATION}/#{localname}", "wb") do |file|
file.write resp.body
end
end
end
That’s it.
The correct version:
require 'nokogiri'
require 'open-uri'
LOCATION = 'pics'
if !File.exist? LOCATION # create folder if it is not exist
require 'fileutils'
FileUtils.mkpath LOCATION
end
#PAGE_URL="http://trinity.e-stile.ru/"
PAGE_URL="http://www.youtube.com/"
page=Nokogiri::HTML(open(PAGE_URL))
links=page.css("img")
links.each{|link|
uri = URI.join(PAGE_URL, link['src'] ).to_s # make absolute uri
localname=File.basename(link['src'])
File.open("#{LOCATION}/#{localname}",'wb') { |f| f.write(open(uri).read) }
}
I'm trying to scrape the "alt" tags from several hundred images on a webpage, then output them to a CSV file. This is essentially the entire lump of HTML I'm looking to scrape:
<div class="product-card"
id="product-35492907"
data-element="product-card"
data-owner="some-data-owner"
data-product-slug="some-data-product-slug"
data-product_id="35492907"
data-stock-status="available"
data-icon-enabled="false"
data-retailer-id="2248">
<a class="product-card-image-link"
href="some href"
data-lead-popup
data-lead-popup-url="/track/lead/21716944/?ctx=2383"
>
<img class="product-card-image draggable"
data-pin-no-hover="true"
src="some src"
data-height="250" data-width="200"
height="250" width="200"
alt="SCRAPE ME" # <<<<< here's the guy I'm after
data-product_id="35492907"
/>
</a>
Below is some code I have been using to scrape elements:
require 'rubygems'
require 'nokogiri'
require 'open-uri'
require 'csv'
url = "http://www.example.com/page"
page = Nokogiri::HTML(open(url))
CSV.open("productResults.csv", "wb") do |csv|
page.css('.product-card-image draggable').each do |scrape| #???
alt_name = scrape.at_css('alt').text #???
scrapedProducts = "#{alt_name}"
csv << [scrapedProducts]
end
end
Start simple and get more complex if necessary:
require 'nokogiri'
require 'csv'
page = Nokogiri::HTML::DocumentFragment.parse(<<EOT)
<div class="product-card"
id="product-35492907"
data-element="product-card"
data-owner="some-data-owner"
data-product-slug="some-data-product-slug"
data-product_id="35492907"
data-stock-status="available"
data-icon-enabled="false"
data-retailer-id="2248">
<a class="product-card-image-link"
href="some href"
data-lead-popup
data-lead-popup-url="/track/lead/21716944/?ctx=2383"
>
<img class="product-card-image draggable"
data-pin-no-hover="true"
src="some src"
data-height="250" data-width="200"
height="250" width="200"
alt="SCRAPE ME" # <<<<< here's the guy I'm after
data-product_id="35492907"
/>
</a>
EOT
Search for the appropriate <img> tag and output its 'alt' parameter's value:
page.css('img.product-card-image').each do |img|
puts img['alt']
end
# >> SCRAPE ME
Modifying it to output to the CSV file:
CSV.open("productResults.csv", "wb") do |csv|
page.css('img.product-card-image').each do |img|
csv << [img['alt']]
end
end
I know that this question has been asked many times before, but due to my inexperience I am unable to correct the problem using those responses. I had it working in a previous Rails project and cannot see what has changed. I started a new project using PostgreSQL rather than MySQL and I am now using Rails 3.2.8 (was 3.2.1).
Firebug showing:
TypeError: aData is undefined at:
var aData = _fnGetObjectDataFn( oSettings.sAjaxDataProp )( json );
for ( var i=0, iLen=aData.length ; i<iLen ; i++ )
I gather that this is probably caused by Datatables not getting the correct JSON. A snippet shows below, which validates in JSONLint.
[{"code":"STATE 30","created_at":"2012-12-10T06:01:34Z","id":1,"name":"ALBANY HIGHWAY","rank":null,"state":"WA","updated_at":"2012-12-10T06:01:34Z"},{"code":"ANNE BEADELL HIGHWAY","created_at":"2012-12-10T06:01:34Z","id":2,"name":"ANNE BEADELL HIGHWAY","rank":null,"state":"SA","updated_at":"2012-12-10T06:01:34Z"},...
I see that it doesn't include iTotalRecords, iTotalDisplayRecords and aaData. Is this the problem? I presume the JSON options are set by the def as_json, buy I don't explicity reference that in my render ??. The http get (as seen in Firebug) includes sEcho but not iTotalRecords.
My code is based on RailsCasts #340:
Controller: ( see the list action).
class HighwaysController < ApplicationController
respond_to :html, :json
def index
#highways = Highway.all
respond_to do |format|
format.html # index.html.erb
format.json { render json: #highways }
end
end
def list
#highways = Highway.all
logger.debug "Highway Count: #{#highways.size}"
respond_to do |format|
format.html # index.html.erb
format.json { render json: HighwaysDatatable.new(view_context)}
end
end
Associated Ruby code:
class HighwaysDatatable
delegate :params, :h,:link_to,:number_to_currency,to: :#view
def initialize(view)
#view = view
end
def as_json(options = {})
{
sEcho: params[:sEcho].to_i,
iTotalRecords: Highway.count,
iTotalDisplayRecords: highways.total_entries,
aaData: data
}
end
private
def data
highways.map do |highway|
[
h(highway.id),
h(highway.name),
h(highway.state),
h(highway.code),
h(highway.rank),
h(highway.created_at),
h(highway.updated_at)
]
end
end
def highways
#highways ||= fetch_highways
end
def fetch_highways
... (as per RailsCasts code - code wrapped and did not show properly)
end
highways
end
def page
params[:iDisplayStart].to_i/per_page + 1
end
def per_page
params[:iDisplayLength].to_i > 0 ? params[:iDisplayLength].to_i : 10
end
def sort_column
columns = %w[name state code rank]
columns[params[:iSortCol_0].to_i]
end
def sort_direction
params[:sSortDir_0] == "desc" ? "desc" : "asc"
end
end
and the list view:
<% #page_title = "Highways List" %>
<h2>Highways</h2>
<%= javascript_tag do %>
window.highwaysURL = '<%= j highways_url %>';
<% end %>
<div id="EditLink">
<%= link_to "Edit Highway", :controller => :highways, :action => :edit, :id => 1 %>
</div>
<div>
</br>
<%= link_to 'Neighbours', '/neighbours/' %>
<%= link_to 'Localities', '/localities/' %>
</br>
</div>
<div id="tools">
</div>
<table id="highways" class="display" data-source="<%= highways_url(format: "json") %>">
<thead>
<tr>
<th>Id</th>
<th>Highway</th>
<th>State</th>
<th>Code</th>
<th>Rank</th>
<th>Created At</th>
<th>Updated At</th>
</tr>
</thead>
<tbody>
</tbody>
</table>
<div id=editarea class='result' ></div>
My gemfile (with comments removed):
source 'https://rubygems.org'
gem 'rails', '3.2.8'
gem 'pg'
group :assets do
gem 'sass-rails', '~> 3.2.3'
gem 'coffee-rails', '~> 3.2.1'
gem 'uglifier', '>= 1.0.3'
end
gem 'libv8', '~> 3.11.8'
gem 'therubyracer', require: 'v8'
gem 'jquery-rails'
gem 'will_paginate'
gem 'debugger'
and application.js for Asset Pipeline:
//= require jquery
//= require jquery-ui
//= require jquery.dataTables
//= require TableTools
//= require ZeroClipboard
//= require datatable_highways
//= require datatable_localities
//= require datatable_neighbours
//= require_tree .
It's probably something simple, but it has been holding me up for some time. Thanks in anticipation.
UPDATE
The DataTables debugger bookmarklet indicates that the 'list' view is rendering the following which does not include the json parameters expected by DataTables.
http://localhost:3000/highways.json
and I believe it should be:
http://localhost:3000/highways/list.json.
I am not sure if this is a controller or a view problem.
UPDATE
I have made various changes but still have the same problem. BUT I see two things in the log which are significant:
Entering localhost:3000/highways/list shows that 2 views are rendered.
Processing by HighwaysController#list as HTML [1m[36mHighway Load (4.8ms)[0m [1mSELECT "highways".* FROM "highways" ORDER BY name
Highway Count: 229
Rendered highways/list.html.erb within layouts/highways (32.6ms)
then
Processing by HighwaysController#index as JSON
Parameters: {"sEcho"=>"1", "iColumns"=>"5", "sColumns"=>"", "iDisplayStart"=>"0",...
The first is HTML (via action 'list') the second as JSON (via action 'index').
So there are two problems:
The first render has an associated data request which causes an
'already initialised' warning from DataTables, and
My routes.rb explicit match statement is catching the HTML to render
the view, but the JSON response (from the AJAX call for the data rows) falls through
and is handled by resources :highways and it then uses the 'index'
action in the controller
routes.rb
match 'highways/list'=> 'highways#list'
...
resources :highways
How do I stop the data request when I just want the view (column headings etc) rendered?
What route do I need to send the JSON response to the 'list' action in the controller?
Resolved. Went back to a minimal set up, 1 table using Ryan Bates #340 code - so using index action/view to get around the 2 render issue above. Still not working. Removed gem to rweng github and loaded DataTables files to ./vendor/assets. Now works. The index or list action can probably be fixed with an explicit route, but I could not get past 'missing template' when I tried to coerce it through the list action.
I am working on an application where I have to
1) get all the links of website
2) and then get the list of all the files and file extensions in each
of the web page/link.
I am done with the first part of it :)
I get all the links of website by below code..
require 'rubygems'
require 'spidr'
require 'uri'
Spidr.site('http://testasp.vulnweb.com/') do |spider|
spider.every_url { |url|
puts url
}
end
now I have to get the all the files/file-extensions in each of the
page so I tried the below code
require 'rubygems'
require 'nokogiri'
require 'open-uri'
require 'spidr'
site = 'http://testasp.vulnweb.com'
in1=[]
Spidr.site(site) do |spider|
spider.every_url { |url| in1.push url }
end
in1.each do |input1|
input1 = input1.to_s
#puts input1
begin
doc = Nokogiri::HTML(open(input1))
doc.traverse do |el|
[el[:src], el[:href]].grep(/\.(txt|css|gif|jpg|png|pdf)$/i).map{|l| URI.join(input1, l).to_s}.each do |link|
puts link
end
end
rescue => e
puts "errrooooooooor"
end
end
but Can anybody guide me how to parse the links/webpage and get the file-
extensions in the page?
You might want to take a look at URI#parse. The URI module is a part of the Ruby standard library and is a dependency of the spidr gem. Example implementation with a spec for good measure.
require 'rspec'
require 'uri'
class ExtensionExtractor
def extract(uri)
/\A.*\/(?<file>.*\.(?<extension>txt|css|gif|jpg|png|pdf))\z/i =~ URI.parse(uri).path
{:path => uri, :file => file, :extension => extension}
end
end
describe ExtensionExtractor do
before(:all) do
#css_uri = "http://testasp.vulnweb.com/styles.css"
#gif_uri = "http://testasp.vulnweb.com/Images/logo.gif"
#gif_uri_with_param = "http://testasp.vulnweb.com/Images/logo.gif?size=350x350"
end
describe "Common Extensions" do
it "should extract CSS files from URIs" do
file = subject.extract(#css_uri)
file[:path].should eq #css_uri
file[:file].should eq "styles.css"
file[:extension].should eq "css"
end
it "should extract GIF files from URIs" do
file = subject.extract(#gif_uri)
file[:path].should eq #gif_uri
file[:file].should eq "logo.gif"
file[:extension].should eq "gif"
end
it "should properly extract extensions even when URIs have parameters" do
file = subject.extract(#gif_uri_with_param)
file[:path].should eq #gif_uri_with_param
file[:file].should eq "logo.gif"
file[:extension].should eq "gif"
end
end
end