Slowly getting there with what i am trying to achieve. I am grabbing data via screen grab and want to save the data to my model, i have two columns, home_team and away_team. So far i grab the data.
FIXTURE_URL = "http://www.bbc.co.uk/sport/football/premier-league/fixtures"
def get_fixtures # Get me all Home and away Teams
doc = Nokogiri::HTML(open(FIXTURE_URL))
home_team = doc.css(".team-home.teams").map {|h| h.text.strip }
away_team = doc.css(".team-away.teams").map {|a| a.text.strip }
#team_clean = Hash[:home_team => home_team, :away_team => away_team]
#team_clean = Hash[:team_clean => [Hash[:home_team => home_team, :away_team => away_team]]]
end
I have hashed out the two ways of getting the data into a hash, one is a hash and the other is a hash within a hash, I am not sure which one i need (if any?)
So if i want to save the data received from my home_team i run a rake task to do this
def update_fixtures #rake task method
Fixture.destroy_all
get_fixtures.each {|home| Fixture.create(:home_team => home )}
end
What i want to achieve is to be able to save home_team and away_team at the same time. Do i need to access the data within the hash, if so how? Bit lost here, but this is the first time i am attempting this
any help appreciated
Try this,
FIXTURE_URL = "http://www.bbc.co.uk/sport/football/premier-league/fixtures"
def get_fixtures # Get me all Home and away Teams
doc = Nokogiri::HTML(open(FIXTURE_URL))
matches = doc.css('tr.preview')
matches.each do |match|
home_team = match.css('.team-home').text.strip
away_team = match.css('.team-away').text.strip
Fixture.create!(home_team: home_team, away_team: away_team)
end
end
This will loop through the matches and create a new Fixture with away and home teams for each match.
Edit:
Added .text.strip
Edit 2:
This should get you the dates too,
FIXTURE_URL = "http://www.bbc.co.uk/sport/football/premier-league/fixtures"
def get_fixtures # Get me all Home and away Teams
doc = Nokogiri::HTML(open(FIXTURE_URL))
days = doc.css('#fixtures-data h2').each do |h2_tag|
date = Date.parse(h2_tag.text.strip)
matches = h2_tag.xpath('following-sibling::*[1]').css('tr.preview')
matches.each do |match|
home_team = match.css('.team-home').text.strip
away_team = match.css('.team-away').text.strip
Fixture.create!(home_team: home_team, away_team: away_team, date: date)
end
end
end
It's a bit more complicated than the previous code because it has to use some XPath to call the next HTML element after the h2 tag containing the date.
It loops through all the h2 html tags in the div#fixtures-data HTML then grabs the table tag directly below/after each h2.
Related
Could someone take a look at my code and let me know if there is a better way to do this, or even correct where I'm going wrong please? I am trying to create a new row for each venue and variant.
Example:
venue_ids => ["1","2"], variant_ids=>["10"]
So, I would want to add in a row which has a venue_id of 1, with variant_id of 10. And a venue_id of 2, with variant_id of 10
I got this working, and it's now passing in my two arrays. I think I am almost there I'm not sure the .each is the right way to do it, but I think that I'm on the right track haha. I have it submitting, however, where would I put my #back_bar.save? because this might cause issues as it won't redirect
Thanks in advance.
def create
#back_bar = BackBar.new
#venues = params[:venue_ids]
#productid = params[:product_id]
#variants = params[:variant_ids]
# For each venue we have in the array, grab the ID.
#venues.each do |v|
#back_bar.venue_id = v
# Then for each variant we associate the variant ID with that venue.
#variants.each do |pv|
#back_bar.product_variant_id = pv
# Add in our product_id
#back_bar.product_id = #productid
# Save the venue and variant to the DB.
if #back_bar.save
flash[:success] = "#{#back_bar.product.name} has been added to #{#back_bar.venue.name}'s back bar."
# Redirect to the back bar page
redirect_to back_bars_path
else
flash[:alert] = "A selected variant for #{#back_bar.product.name} is already in #{#back_bar.venue.name}'s back bar."
# Redirect to the product page
redirect_to discoveries_product_path(#back_bar.product_id)
end
end # Variants end
end # Venues end
end
private
def back_bar_params
params.require(:back_bar).permit(:venue_id,
:product_id,
:product_variant_id)
end
as i said in comments
this is untested code and just showing you how it's possible to do with ease.
class BackBar
def self.add_set(vanue_ids, variant_ids)
values = vanue_ids.map{|ven|
variant_ids.map{|var|
"(#{ven},#{var})"
}
}.flatten.join(",")
ActiveRecord::Base.connection.execute("INSERT INTO back_bars VALUES #{values}")
end
end
def create
# use in controller
BackBar.add_set(params[:venue_ids], params[:variant_ids])
# ...
end
I am trying to learn how to get data via a screen scrape and then save it to a model. So far I can grab the data. I say this as if I do:
puts home_team
I get all the home teams returned
get_match.rb #grabbing the data
require 'open-uri'
require 'nokogiri'
module MatchGrabber::GetMatch
FIXTURE_URL = "http://www.bbc.co.uk/sport/football/premier-league/fixtures"
def get_fixtures
doc = Nokogiri::HTML(open(FIXTURE_URL))
home_team = doc.css(".team-home.teams").text
end
end
Then i want to update my model
match_fixtures.rb
module MatchFixtures
class MatchFixtures
include MatchGrabber::GetMatch
def perform
update_fixtures
end
private
def update_fixtures
Fixture.destroy_all
fixtures = get_fixtures
end
def update_db(matches)
matches.each do |match|
fixture = Fixture.new(
home_team: match.first
)
fixture.save
end
end
end
end
So the next step is where I am getting stuck. First of all I need to put the home_team results into an array?
Second part is I am passing matches through my update_db method but that's not correct, what do I pass through here, the results of the home_team from my update_fixtures method or the method itself?
To run the task I do:
namespace :grab do
task :fixtures => :environment do
MatchFixtures::MatchFixtures.new.perform
end
end
But nothing is saved, but that is to be expected.
Steep learning curve here and would appreciate a push in the right direction.
Calling css(".team-home.teams").text does not return the matching DOM elements as an array, but as a single string.
In order to obtain an array of elements, refactor get fixture into something like this:
get_teams
doc = Nokogiri::HTML(open(FIXTURE_URL))
doc.css(".team-home.teams").map { |el| el.text.strip }
end
This will return an array containing the text of the elements matching your selector, stripped out of blank and new line characters. At this point you can loop over the returned array and pass each team as an argument to your model's create method:
get_teams.each { |team| Fixture.create(home_team: team) }
You could just pass the array directly to the update method:
def update_fixtures
Fixture.destroy_all
update_db(get_fixtures)
end
def update_db(matches)
matches.each {|match| Fixture.create(home_team: match.first) }
end
Or do away with the method all together:
def update_fixtures
Fixture.destroy_all
get_fixtures.each {|match| Fixture.create(home_team: match.first) }
end
I am using Nokogiri to grab data from a webpage, so far i can save to one column in the model
def update_fixtures #rake task method
Fixture.destroy_all
get_fixtures.each {|match| Fixture.create(home_team: match )}
end
def get_fixtures # Get me all Home Teams
doc = Nokogiri::HTML(open(FIXTURE_URL))
home_team = doc.css(".team-home.teams").map {|h| h.text.strip }
end
What I am wondering is the most efficient way to save to 2, 3 or 4 columms at the same time
So as an example I have another column called away_team and I would grad that data in the same way as the home team
away_team = doc.css(".team-away.teams").map {|a| a.text.strip }
is it advisable to put this within the get_fixtures method? and then add to the update_fixtures with something like
def update_fixtures #rake task method
Fixture.destroy_all
get_fixtures.each {|match| Fixture.create(home_team: match, away_team: match )}
end
After trying this the same data gets posted to the home and away columns.Which after reading back i can see why (I think its because match is only grabbing the home_team data?). How can i pass the attributes of the away team along with the home team?
This is all very new so any help provided is appreciated
This isn't the right approach because the variables home_team and away_team both are using the same common match and thus you are getting the same data for both.
Do the following:
UPDATE:
Your model:
attr_accessible :home_team, :away_team
def update_fixtures #rake task method
Fixture.destroy_all
doc = Nokogiri::HTML(open(FIXTURE_URL))
home_team = doc.css(".team-home.teams").map {|h| h.text.strip }
away_team = doc.css(".team-away.teams").map {|a| a.text.strip }
Fixture.create(home_team: home_team, away_team: away_team)
end
This works on the rails console (rails c) but when seen from browser it does not, it just not shows the results, no errors.
on controler i have:
def weekly_user_stats
#created_users = User.where("username IS NOT NULL")
#created_users_by_week = Hash.new{ |h,k| h[k] = [] }
#created_users.each do |u|
unless u.nil?
#created_users_by_week[u.created_at.beginning_of_week.strftime("%Y-%m-%d")].push("#{u.username}")
end
end
end
on views I have:
%ul
= #created_users_by_week.keys.sort.each do |k|
%li
= puts "#{k} : #{#created_users_by_week[k]}"
on the page results I only see the keys but not the values inside the keys that shoul be an array of users..
on the console works just fine..
what im doing wrong?
Shouldn't the view look more like this?
%ul
- #created_users_by_week.keys.sort.each do |k|
%li #{k} : #{#created_users_by_week[k]}
See: http://haml-lang.com/docs/yardoc/file.HAML_REFERENCE.html
I'm not familiar with haml so I don't know about your specific issue. However I would suggest you to store a different date in your database in order to retreive quickly your created users by week (or even month if you wanted).
You could do that by storing an array as an attribute containing this things:
dates = ['20120307', '201203', '2012', '201210']
With
dates[0] # (yyyymmdd)
dates[1] # (yyyymm)
dates[2] # (yyyy)
dates[3] # (yyyyww) ww = week number
Then you could retreive users by querying on the dates attributes that includes '201210'.
I have a csv importing system on my app (used locally only) which parses the csv file line by line and adds the data to the database table. This is based on a tutorial here.
require 'csv'
def csv_import
#parsed_file=CSV::Reader.parse(params[:dump][:file])
n = 0
#parsed_file.each_with_index do |row, i|
next if i == 0 #ignore the first row
course = Course.new
course.title = row[0]
course.unit_code = row[1]
course.course_type = row[2]
course.value = row[3]
course.pass_mark = row[4]
if course.save
n = n+1
GC.start if n%50==0
end
flash.now[:message] = "CSV Import Successful, #{n} new courses added to the database."
end
redirect_to(courses_url)
end
This is all in the courses controller and works fine. There is a relationship that courses HABTM years and years HABTM courses. In the csv file (effectively in row[5] to row[8]) are the year_id s. Is there a way that I can add this within the method above. I am confused as to how to loop over the 4 items and add them to the courses_years table.
Thank you
Jack
You can do this by adding a simple loop after your "normal" data is added to the model, and using the << method to append to the years association.
...
course.value = row[3]
course.pass_mark = row[4]
5.upto(8).each do |i|
one_year = Year.find(row[i])
course.years << one_year if one_year
end
if course.save
n = n+1
...
You can add more checks in the loop if you want to make sure that the values are valid, and/or change the find to locate your year in another way. Another way when the related data is "trailing off the end" like this is to keep adding until there is nothing left to add, and also to add the years themselves if they don't exist yet:
...
course.value = row[3]
course.pass_mark = row[4]
row[5..-1].each do |year_id|
one_year = Year.find_or_create_by_id(year_id)
course.years << one_year
end
if course.save
n = n+1
...
There are a lot of different ways to do this, and the way which is right is really dependent on your actual data, but this is the basic method.
Have you tried to put either one of these before you save the course:
course.years.push(row[5])
course.years.push(row[6])
course.years.push(row[7])
course.years.push(row[8])
OR
course.years = [ row[5], row[6], row[7], row[8] ]
Place it before you save the course. It will fill the joint table courses_years.
EDIT
The error that you get seems to be because we are trying to put id's instead of objects, we should do this instead:
.....
year_array = Year.find(row[5], row[6], row[7], row[8])
course.years << year_array
.....
After we get the year objects, then we put it inside the association. You can save the course object after that.