I have the following script running in my rails app. When it runs locally, it parses all of the courses correctly (9673 results) but when I push it to Heroku, it returns exactly 2x this number (19346 results). Any ideas? Are there possible Heroku performance limitations that could be causing the cycle to repeat itself?
# Pulls all course data for specified year, based on subject list
subjectdoc= Nokogiri.XML(open("https://courseroster.reg.cornell.edu/courses/roster/SP15/xml/"))
# Reads each subject and stores it in local variable prefix
subjectdoc.xpath("//subject/#subject").each do |prefix|
# Link to course pages, substituting in prefix in URL
classdoc= Nokogiri.XML(open("https://courseroster.reg.cornell.edu/courses/roster/SP15/#{prefix}/xml/"))
# Reads each course and stores listed vars
classdoc.xpath("/courses/course").each do |course|
num = course["catalog_nbr"] || "Not provided"
subj = course["subject"] || "Not provided"
title = (course.at("course_title/text()") || "Not provided").to_s
cid = (course.at("sections/section/#class_number") || "Not provided").to_s
inst = (course.at("sections/section/meeting/instructors/instructor/text()") || "Not provided").to_s
# Creates a cornell class in Cornellclasses table
Cornellclass.create(:prefix => subj, :coursenumber => num, :instructor => inst, :title => title, :courseid => cid)
end
end
Related
I want to be able to determine if a user is a memberOf a specific group as well as a couple other attributes available on the active directory. I know that the authentication and binding is working as intended, however, I keep getting two warnings:
warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
warning: The called method tcp is defined here
Since these are warnings, I wouldn't expect them to affect the output that I'm looking for.
I want to list out the attributes and the values for the search query, but when I run the code, I get the warnings, followed by "Bind succeeded!", and then the program finishes executing. I've tried changing the query, changing the attributes that I'm searching for, and ensuring that they were named correctly. I have posted the code below:
ldap = Net::LDAP.new :host => HOST, # your LDAP host name or IP goes here,
:port => PORT, # your LDAP host port goes here,
:encryption => :simple_tls,
:base => BASE# the base of your AD tree goes here,
:auth => {
:method => :simple,
:username => "#{login}##{myDomain}", # a user w/sufficient privileges to read from AD goes here,
:password => PASSWORD # the user's password goes here
}
search_param = #username
result_attrs = ["sAMAccountName", "displayName", "mail", "memberOf"] # Whatever you want to bring back in your result set goes here
# Build filter
search_filter = Net::LDAP::Filter.eq("sAMAccountName", search_param)
# Execute search
if ldap.bind
puts "Bind succeeded!"
ldap.search(:filter => search_filter, :attributes => result_attrs) do |entry|
puts "DN: #{entry.dn}"
entry.each do |attr, values|
puts ".......#{attr}:"
values.each do |value|
puts "#{value}\n"
end
end
end
else
puts "Failure to bind"
end
A couple problems I feel could be causing this issue is that I don't have the required authentication to search for users in the directory, or the users don't exist (I am searching for my own username, so it should exist, therefore I'm leaning to first theory). Thank you!
I have a rails app. I have a table User and a column Number which is a string. Some users saved their phone number with spaces (for example 1234 1234) and now I want to remove the space from their phone numbers.
I tried this but it didn't work:
space = " "
phones = User.where("number like ?", "%#{space}%").pluck(:number)
phones.each do |phone|
phone = phone.gsub(/\s+/, "")
phone.save
end
I got the error NoMethodError: undefined method 'save' How can I do this properly?
You need to have the user object to save it. Read inline comments below
space = " "
users = User.where("number like ?", "%#{space}%") # collect users with number having space character here.
# then iterate on those users
users.each do |user|
user.number = user.number.gsub(/\s+/, "") # notice here, changing the phone number of that user
user.save # and saving that user with the updated `number`
end
You pluck data from User table. Thus, phones variable contains a number array not USER objects. You can't use save on a array element. That's why the error occurs.
You can do the following:
space = " "
phones = User.where("number like ?", "%#{space}%")
phones.each do |phone|
phone.number = phone.number.gsub(/\s+/, "")
phone.save
end
The way you could do is create a rake task to update the existing records on the system.
namespace :update do
desc 'Strip space from existing numbers from Users'
task(:number => ::environment) do
space = ' '
numbers_with_space = User.where("number like ?", "%#{space}%")
numbers_with_space.each do |a|
a.number = a.number.gsub!(/\s+/, '')
a.save(validate: false) # You would like to use
# validate false in order
# to stop other validation from updating the record.
end
end
Then execute the rake task.
bundle exec rake update:number
Another way to handle this beforehand can be through reformatting number during validation. This way you'll not need to run the rake task or code to reformat and save when new data are entered in app.
class User < ApplicationRecord
before_validation :reformat_number, on: [:create, :update]
private
def reformat_number
self.number.gsub!(/\s+/, '')
end
end
I am building a daily deal Rails app
I am displaying to the user only the deals of the country i associate him with thanks to geocoder gem.
I wonder what would happen if geocoder fails (for any reason) to retrieve the country and sends an empty array, as i think it does when it fails to send ip( see https://github.com/alexreisner/geocoder#error-handling)
class StaticPagesController < ApplicationController
def home
#deals = deal_in_user_country.featured_on_hp
respond_to do |format|
format.html # home.html.erb
end
end
# create a scope to filter the deals that meet the user's country
def deal_in_user_country
Deal.where(country: set_location_by_ip_lookup.country || 'United States') # if geocoder gets pb then default = US version
end
end
end
As you see, I tried to use || and puts 'United States' but I don't think it will work. i think that if geocoder sends empty array , set_location_by_ip_lookup=[] and then set_location_by_ip_lookup.country will generate an error, am I right ?
How should i avoid bugs when geocoder sends an empty array ?
For info if it helps, here is how I set country in concerns/CountrySetter
module CountrySetter
extend ActiveSupport::Concern
included do
before_filter :set_location_by_ip_lookup
end
def set_location_by_ip_lookup
if Rails.env.development? or Rails.env.test?
Geocoder.search(request.remote_ip).first
else #in production
request.location
end
end
end
Your code should be fine, if geocoder allways returns at least an empty array (except I would not name this mehtod set_ because it's not setting anything)
Try out on irb
{a: [:d].first || :b}
=> {:a=>:d}
{a: [].first || :b}
=> {:a=>:b}
However i would put this in paranethesis to make it clear
Deal.where(country: (set_location_by_ip_lookup.country || 'United States'))
Gecodoer.search shouldn't be throwing exceptions, otherwise it would be named Geocoder.search! according to an unwritten ruby convention. I would double check this by plugin out the internet connection and see what happens
So I have some code here I need to modify regarding a Rails Sweeper:
class UserTrackingSweeper < ActionController::Caching::Sweeper
observe User
def after_update(user)
return if user.nil? || user.created_at.nil? #fix weird bug complaining about to_date on nil class
return if user.created_at.to_date < Date.today || user.email.blank?
user.send_welcome_email if user.email_was.blank?
end
#use sweeper as a way to ingest metadata from the user access to the site automatically
def after_create(user)
begin
if !cookies[:user_tracking_meta].nil?
full_traffic_source = cookies[:user_tracking_meta]
else
if !session.empty? && !session[:session_id].blank?
user_tracking_meta = Rails.cache.read("user_meta_data#{session[:session_id]}")
full_traffic_source = CGI::unescape(user_tracking_meta[:traffic_source])
end
end
traffic_source = URI::parse(full_traffic_source).host || "direct"
rescue Exception => e
Rails.logger.info "ERROR tracking ref link. #{e.message}"
traffic_source = "unknown"
full_traffic_source = "unknown"
end
# if I am registered from already, than use that for now (false or null use site)
registered_from = user.registered_from || "site"
if params && params[:controller]
registered_from = "quiz" if params[:controller].match(/quiz/i)
# registered_from = "api" if params[:controller].match(/api/i)
end
meta = {
:traffic_source => user.traffic_source || traffic_source,
:full_traffic_source => full_traffic_source,
:registered_from => registered_from,
:id_hash => user.get_id_hash
}
user.update_attributes(meta)
end
end
The problem is I've noticed that it dosen't seem possible to access the cookies and parameters hash within a sweeper yet it appears fine in some of our company's integration environments. It does not work in my local machine though. So my questions are:
How is it possible to access params / cookies within a Sweeper?
If it's not possible, what would you do instead?
Thanks
I'm sure you can use session variables in a Cache Sweeper so if anything put whatever you need there and you're set
As part of my Rails application, I've written a little importer that sucks in data from our LDAP system and crams it into a User table. Unfortunately, the LDAP-related code leaks huge amounts of memory while iterating over our 32K users, and I haven't been able to figure out how to fix the issue.
The problem seems to be related to the LDAP library in some way, as when I remove the calls to the LDAP stuff, memory usage stabilizes nicely. Further, the objects that are proliferating are Net::BER::BerIdentifiedString and Net::BER::BerIdentifiedArray, both part of the LDAP library.
When I run the import, memory usage eventually peaks at over 1GB. I need to find some way to correct my code if the problem is there, or to work around the LDAP memory issues if that's where the problem lies. (Or if there's a better LDAP library for large imports for Ruby, I'm open to that as well.)
Here's the pertinent bit of our my code:
require 'net/ldap'
require 'pp'
class User < ActiveRecord::Base
validates_presence_of :name, :login, :email
# This method is resonsible for populating the User table with the
# login, name, and email of anybody who might be using the system.
def self.import_all
# initialization stuff. set bind_dn, bind_pass, ldap_host, base_dn and filter
ldap = Net::LDAP.new
ldap.host = ldap_host
ldap.auth bind_dn, bind_pass
ldap.bind
begin
# Build the list
records = records_updated = new_records = 0
ldap.search(:base => base_dn, :filter => filter ) do |entry|
name = entry.givenName.to_s.strip + " " + entry.sn.to_s.strip
login = entry.name.to_s.strip
email = login + "#txstate.edu"
user = User.find_or_initialize_by_login :name => name, :login => login, :email => email
if user.name != name
user.name = name
user.save
logger.info( "Updated: " + email )
records_updated = records_updated + 1
elsif user.new_record?
user.save
new_records = new_records + 1
else
# update timestamp so that we can delete old records later
user.touch
end
records = records + 1
end
# delete records that haven't been updated for 7 days
records_deleted = User.destroy_all( ["updated_at < ?", Date.today - 7 ] ).size
logger.info( "LDAP Import Complete: " + Time.now.to_s )
logger.info( "Total Records Processed: " + records.to_s )
logger.info( "New Records: " + new_records.to_s )
logger.info( "Updated Records: " + records_updated.to_s )
logger.info( "Deleted Records: " + records_deleted.to_s )
end
end
end
Thanks in advance for any help/pointers!
By the way, I did ask about this in the net/ldap support forum as well, but didn't get any useful pointers there.
One very important thing to note is that you never use the result of the method call. That means that you should pass :return_result => false to ldap.search:
ldap.search(:base => base_dn, :filter => filter, :return_result => false ) do |entry|
From the docs: "When :return_result => false, #search will return only a Boolean, to indicate whether the operation succeeded. This can improve performance with very large result sets, because the library can discard each entry from memory after your block processes it."
In other words, if you don't use this flag, all entries will be stored in memory, even if you do not need them outside the block! So, use this option.