I am trying to scrape a website using this tutorial:
https://towardsdatascience.com/job-board-scraping-with-rails-872c432ed2c8
Error: https://i.stack.imgur.com/XZ3T9.jpg
Did you have the line:
require 'open-uri'
before the doc = Nokogiri::HTML(open(URL))?
open-uri enhances the Kernel.open method, which normally only reads from a local file, with a http option. Your error looks like, open-uri was not loaded.
doc = Nokogiri::HTML(URI.open(link))
Added URI.
This post helped me
Related
I'm using Selenium-Webdriver, OpenUri and Nokogiri to scrape a website. I want to download a particular image from said website to my Ubuntu computer. I tried a few different methods but each of them gives a different error message.
Here's my base code, which opens the website and gets the image url (everything after this I ran in my pry console):
require 'open-url'
require 'selenium-webdriver'
require 'nokogiri'
require 'uri'
url = "https://www.google.com/"
browser = Selenium::WebDriver.for :chrome
document = open(url).read
parsed_content = Nokogiri::HTML(content)
image = "https://www.google.com" + parsed_content.css('#hplogo').attr('src').value
binding.pry
1) Here's the first thing I tried to download the image:
download = open(image)
IO.copy_stream(download, '~/image.png')
For this, I got the following error:
Errno::ENOENT: No such file or directory # rb_sysopen - ~/image.png from (pry):44:in 'initialize'
As per this question, I tried adding a directory in the code:
FileUtils.mkdir_p(image) unless File.exist?(image)
But I got the same error.
2) Next I tried this:
open('image.png', 'wb') do |file|
file << open(image).read
end
and this returns
#<File:image.png (closed)
but the file isn't anywhere on my computer and I can't figure out what that message means.
3) Next I tried
IO.copy_stream(open(image), 'image.png')
which simply returned this:
5482
but again, I have no idea what that means and the file isn't anywhere.
4) Finally I tried
read_image = open(image).read
File.open(image, 'image.png') do |file|
file.puts read_image
end
which outputs
ArgumentError: invalid access mode image.png
from (pry):53:in 'initialize
What am I doing wrong? Was I close with any of my approaches?
File open second argument is mode for file openning.
read_image = open(image).read
File.open('image.png', 'w+') do |file|
file.write read_image
end
Your third variant works good.
5482 - length of file. File 'image.png' in same directory as your .rb file.
I am trying to open a profile picture uploaded to S3 through Paperclip, in order to submit it as an online attachment to an email.
Though I get error :
No such file or directory # rb_sysopen
Here is my bit of code in question :
attachments.inline['profilepic'] = File.read(profilepic)
profilepic being an absolute URL (starting with //mybucket.S3-eu-west..... ) to the image at S3 (when pasted onto the navbar, it just shows the image perfectly)
I have tried the following using open-uri, but same error
require 'open-uri'
attachments.inline['profilepic'] = open(profilepic)
Like you did, you need to first:
require 'open-uri'
and then do:
uri = URI("http:"+profilepic.to_s)
attachments["profilepic"] = open(uri).read
I'm trying to attach a file to an email. The file is in assets/downloads/product.pdf
In the mailer, I have:
attachments["product.pdf"] = File.read(ActionController::Base.helpers.asset_path("product.pdf"))
I've tried:
attachments["product.pdf"] = File.read(ActionController::Base.helpers.asset_url("product.pdf"))
...and even:
attachments["product.pdf"] = File.read(ActionController::Base.helpers.compute_asset_host("product.pdf") + ActionController::Base.helpers.compute_asset_path("product.pdf"))
I always get the same error:
EmailJob crashed!
Errno::ENOENT: No such file or directory - //localhost:3000/assets/product.pdf
...or a variation on the theme. But even when I try using asset_url in the view or just put the url in the browser it works:
http://localhost:3000/assets/product.pdf
I've also tried using straight up:
File.read("app/assets/downloads/product.pdf")
File.read("downloads/product.pdf")
...which works in dev environment but not on staging server (heroku). Error is still:
Errno::ENOENT: No such file or directory - downloads/product-market-fit-storyboard.pdf
Also tried:
File.read("/downloads/product.pdf")
File.read("http://lvh.me:3000/assets/product.pdf")
...don't work at all.
Ideas?
you should use syntex like this.it is work for me may be will work for you also.
File.open(Dir.glob("#{Rails.root}/app/assets/downloads/product.pdf"), "r")
When using mailer, you shouldn't use assets pipline. Asset pipeline would be useful if you wanted to have link to a file inside your email. When rendering an email, action mailer has access to files in app directory.
Please read about attachments in action mailer guide. As you can see, you just need to pass path to a file, not url:
attachments['filename.jpg'] = File.read('/path/to/filename.jpg')
I want to read a file from github repository in my ruby script. Say I want to read Gemfile from my repo on github, URL for which would be like: "http://www.github.com/myrepo/blob/master/Gemfile".
I tried using File.readLink("http://www.github.com/myrepo/blob/master/Gemfile") but this gives me error saying "'readlink': No such file or directory # rb_readlink".
How do I read a file using the github URL?
You should try to fetch raw content from github files like:
require 'net/http'
uri = "https://raw.githubusercontent.com/username/myrepo/master/Gemfile"
uri = URI(uri)
file = Net::HTTP.get(uri)
With the below code, I was able to read the content of the file.
require 'open-uri'
raw_url = "https://raw.githubusercontent.com/username/myrepo/master/Gemfile"
open(raw_url) {|f|
f.each_line {|line| p line}
}
I'm having some problems reading a file from S3. I want to be able to load the ID3 tags remotely, but using open-URI doesn't work, it gives me the following error:
ruby-1.8.7-p302 > c=TagLib2::File.new(open(URI.parse("http://recordtemple.com.s3.amazonaws.com/music/745/original/The%20Stranger.mp3?1292096514")))
TypeError: can't convert Tempfile into String
from (irb):8:in `initialize'
from (irb):8:in `new'
from (irb):8
However, if i download the same file and put it on my desktop (ie no need for open-URI), it works just fine.
c=TagLib2::File.new("/Users/momofwombie/Desktop/blah.mp3")
is there something else I should be doing to read a remote file?
UPDATE: I just found this link, which may explain a little bit, but surely there must be some way to do this...
Read header data from files on remote server
Might want to check out AWS::S3, a Ruby Library for Amazon's Simple Storage Service
Do an AWS::S3:S3Object.find for the file and then an use about to retrieve the metadata
This solution assumes you have the AWS credentials and permission to access the S3 bucket that contains the files in question.
TagLib2::File.new doesn't take a file handle, which is what you are passing to it when you use open without a read.
Add on read and you'll get the contents of the URL, but TagLib2::File doesn't know what to do with that either, so you are forced to read the contents of the URL, and save it.
I also noticed you are unnecessarily complicating your use of OpenURI. You don't have to parse the URL using URI before passing it to open. Just pass the URL string.
require 'open-uri'
fname = File.basename($0) << '.' << $$.to_s
File.open(fname, 'wb') do |fo|
fo.print open("http://recordtemple.com.s3.amazonaws.com/music/745/original/The%20Stranger.mp3?1292096514").read
end
c = TagLib2::File.new(fname)
# do more processing...
File.delete(fname)
I don't have TagLib2 installed but I ran the rest of the code and the mp3 file downloaded to my disk and is playable. The File.delete would clean up afterwards, which should put you in the state you want to be in.
This solution isn't going to work much longer. Paperclip > 3.0.0 has removed to_file. I'm using S3 & Heroku. What I ended up doing was copying the file to a temporary location and parsing it from there. Here is my code:
dest = Tempfile.new(upload.spreadsheet_file_name)
dest.binmode
upload.spreadsheet.copy_to_local_file(:default_style, dest.path)
file_loc = dest.path
...
CSV.foreach(file_loc, :headers => true, :skip_blanks => true) do |row|}
This seems to work instead of open-URI:
Mp3Info.open(mp3.to_file.path) do |mp3info|
puts mp3info.tag.artist
end
Paperclip has a to_file method that downloads the file from S3.