I have the following query hooked up to a rails app. It only seems to be returning 30 projects at a time, even with an increased per_page count. However, if I make the same request via the browser, I get the expected number of projects.
https://api.github.com/search/repositories?q=license:gpl+license:lgpl+license:gpl+license:mit+forks:0+stars:1..2000+pushed:%3E2019-12-11+language:ruby+language:javascript&per_page=100
controller:
def request_projects
api_key = Rails.application.credentials.dig(:github)
query = "#{Project::LICENSES}+#{Project::FORKS}+#{Project::STARS}+#{Project::DATE}+#{Project::LANGUAGES}"
uri = URI.parse("#{Project::GITHUB_BASE_URL}?q=#{query}&client_secret=#{api_key[:secret]}&per_page=100")
res = Net::HTTP.get_response(uri)
Project.create_from_request(res.body)
end
Model:
def self.create_from_request(request_body)
data = JSON.parse(request_body, symbolize_names: true)
data[:items].each do |item|
name = item[:name]
owner = item[:owner][:login]
url = item[:html_url]
stars = item[:stargazers_count]
Project.create(name: name, owner: owner, url: url, stars: stars) if !Project.exists?(name: name, owner: owner)
end
end
Related
Im trying to use Kimurai to scrape a website. Im running into this error when I want to do /scrape.
def scrape
url = "https://www.tripadvisor.com/Restaurants-g31892-Rogers_Arkansas.html"
response = RestaurantsScraper.parse!(response, url, data: {})
if response[status] == :completed && response[error].nil?
flash.now[notice] = "Successfully scraped url"
else
flash.now[alert] = response[error]
end
end
Here is my scraper class
class RestaurantsScraper < Kimurai::Base
#name = "restaurants_scraper"
#driver = :selenium_chrome
#start_urls = ["https://www.tripadvisor.com/Restaurants-g31892-Rogers_Arkansas.html"]
def parse(response, url:, data: {})
response.xpath("//div[#class=_1llCuDZj]").each do |a|
request_to :parse_repo_page, url: absolute_url(a[:href], base: url)
end
end
def parse_repo_page(response, url:, data: {})
item = {}
item["title"] = t.css('a._15_ydu6b')&.text&.squish&.gsub('[^0-9].', '')
item["type"] = t.css('span._1p0FLy4t')&.text&.squish
item["reviews"] = t.css('span.w726Ki5B').text&.squish
item["top_reviews"] = t.css('a._2uEVo25r _3mPt7dFq').text&.squish
Restaurant.where(item).first_or_create
end
end
Here is the error im getting
It's because response from RestaurantsScraper.parse!(response, url, data: {}) isn't defined.
From the kimurai docs it says you need to pass a Nokogiri::HTML::Document object.
I haven't used Kimurai and it feels like there is definitely a better way to do this, but something like the following may be enough to get you to the next step:
def scrape
require 'open-uri'
url = "https://www.tripadvisor.com/Restaurants-g31892-Rogers_Arkansas.html"
html = Nokogiri.parse open(url)
response = RestaurantsScraper.parse!(html, url, data: {})
if response[status] == :completed && response[error].nil?
flash.now[notice] = "Successfully scraped url"
else
flash.now[alert] = response[error]
end
end
I'm new to Rails and I'm trying to make a simple weather API to get weather by zipcode
is there a way to get the zipcode from user input from a simple form, this will be just for learning so I'm not trying to make users devise, or users model
require 'net/http'
require 'json'
#url = 'http://api.openweathermap.org/data/2.5/weather?zip=#{zipcode}&appid=APIKEY'
#uri = URI(#url)
#response = Net::HTTP.get(#uri)
#output = JSON.parse(#response)
actually I figured it out, i needed to add
def zipcode
#zip_query = params[:zipcode]
if params[:zipcode] == ""
#zip_query = "Hey you forgot to enter a zipcode!"
elsif params[:zipcode]
# Do Api stuff
require 'net/http'
require 'json'
#url = 'http://api.openweathermap.org/data/2.5/weather?zip='+ #zip_query +'&appid=APIKEY'
#uri = URI(#url)
#response = Net::HTTP.get(#uri)
#output = JSON.parse(#response)
#name = #output['name']
# Check for empty return result
if #output.empty?
#final_output = "Error"
elsif !#output
#final_output = "Error"
else
#final_output = ((#output['main']['temp'] - 273.15) * 9/5 +32).round(2)
end
end
end
in the controller.rb file
and add
post "zipcode" => 'home#zipcode'
get "home/zipcode"
in the routes file
but I'm sure this is not the best practice
I have implemented user provisioning/deprovisioning with SCIM like so :
users_controller.rb
class Scim::UsersController < Scim::ScimController
before_action :set_scim_provider
def index
startIndex = params[:startIndex].to_i
startIndex = 1 if startIndex < 1# if the user send a startIndex < 1, it is bad data, we don't take it.
itemsPerPage = params[:count].to_i
if itemsPerPage < 1 || itemsPerPage > #scim_provider.max_results
itemsPerPage = #scim_provider.default_number_of_results
end
scim_users = #scim_provider.identity_provider.communaute_accesses.from_scim
if params["filter"]
parser = Scim::QueryFilter::Parser.new
rpn_array = parser.parse(params["filter"])
tree = parser.tree
if tree.length == 3 and tree[0]== 'eq' and tree[1] == 'userName'
userName = tree[2]
scim_users = scim_users.where(provider_identifier: userName.delete('"'))
else
fail 'e'
end
end
paginated_users = scim_users.order(:created_at).offset(startIndex - 1).limit(itemsPerPage)
r = {
"schemas": ["urn:ietf:params:scim:api:messages:2.0:ListResponse"],
"totalResults": scim_users.size,
"Resources": paginated_users.map { |ca| #scim_provider.representation_for_user(ca) },
"startIndex": startIndex,
"itemsPerPage": itemsPerPage
}
render_json_result(r, 200)
end
def create
if #scim_provider.identity_provider.communaute_accesses.from_scim.find_by(provider_identifier: #body_params['userName'])
render_409_conflict("uniqueness")
else
ca = #scim_provider.identity_provider.communaute_accesses.find_by(provider_identifier: #body_params['userName'], communaute_id: #scim_provider.identity_provider.communaute.id)
if ca.nil?
ca = #scim_provider.identity_provider.communaute_accesses.create(provider_identifier: #body_params['userName'], communaute_id: #scim_provider.identity_provider.communaute.id)
end
ca.update_last_raw_value("scim", #body_string)
ca.extract_values_from_scim
ca.queue_send
end
render_json_result(#scim_provider.representation_for_user(ca), 201)
end
def show
user = #scim_provider.identity_provider.communaute_accesses.from_scim.find_by(provider_identifier: #body_params['userName'])
if user
render_json_result(#scim_provider.representation_for_user(user), 200)
else
render_404_not_found(params[:id])
end
end
def update
ca = #scim_provider.identity_provider.communaute_accesses.from_scim.find_by(provider_identifier: #body_params['userName'])
uc = UserCommunaute.find_by(provider_identifier: #body_params['userName'])
ca.update_last_raw_value("scim", #body_string)
ca.extract_values_from_scim
unless ca.nil?
if ca.pending?
ca.update_last_raw_value("scim", #body_string)
ca.update(active: false)
if ca.active == false
fail "Unable to delete this user because of activeness" if ca.active == true
ca.destroy!
end
render_json_result(#scim_provider.representation_for_communaute_access_patch(ca), 200)
end
end
unless uc.nil?
uc.update(active: #body_params['active'])
if uc.active == false
uc.user.communaute_accesses.from_scim.destroy_all
uc.user.user_communautes.from_scim.destroy_all
render_json_result(#scim_provider.representation_for_user_communaute_patch(uc), 200)
end
end
end
end
Explanations:
When updating a user, SCIM sends a PATCH request like this:
{"schemas"=>["urn:ietf:params:scim:api:messages:2.0:PatchOp"], "Operations"=>[{"op"=>"Replace", "path"=>"active", "value"=>"False"}]} (#body_params in the code)
Which is what i am expecting. But, for a while, i was receiving the userName also in the body response during the PATCH operation.
This is how I fetch the correct user in my DB.
Actual result:
I don't receive the userName anymore when SCIM hits my update action.
Expected results:
Being able to receive information about the user during the PATCH operation to fetch the userName and find the right user in my database.
I have tried almost everything. When SCIM hits the index action, which it does everytime before going anywhere else, it does return me a userName et everything ends up as a 200 OK.
Then, when passing through update, it sends me nothing.
What I have tried last is to isolate the userName as an instance variable in the index action to fetch it after in the update like so:
# index
...
if params["filter"]
parser = Scim::QueryFilter::Parser.new
rpn_array = parser.parse(params["filter"])
tree = parser.tree
if tree.length == 3 and tree[0]== 'eq' and tree[1] == 'userName'
#user_name = tree[2]
scim_users = scim_users.where(provider_identifier: #user_name.delete('"'))
else
fail 'e'
end
end
...
# update
def update
ca = #scim_provider.identity_provider.communaute_accesses.from_scim.find_by(provider_identifier: #user_name)
uc = UserCommunaute.find_by(provider_identifier: #user_name)
ca.update_last_raw_value("scim", #body_string)
ca.extract_values_from_scim
...
But, #user_name in update seems to disappear as its value is nil.
I am deprovisioning from Azure Active Directory and Okta in a production environment.
Mapping is ok in both platforms.
Provisioning is working like a charm.
Please refer to https://developer.okta.com/docs/reference/scim/scim-20/#update-a-specific-user-patch for PATCH /Users/{userId}. Could you not make use of the userId in the url to identify the user ?
i am using spree gem in my app. I import products from product_controller_decorator.rb using csv file.Three tables like spree_products,spree_variants and
spree_prices are relation. As #prod.save happen then it create one row in spree_products,one in spree_variants and one in spree_prices. then another
spree_variants and spree_prices row form due to #var and #price for corresponding #prod which create duplicacy in both table.This is happens due to
association among three tables.can any one resolve this complicacy.It means it form one row to each table.
def import
require 'csv'
file = params[:file]
CSV.foreach(file.path, headers: true, encoding:'iso-8859-1:utf-8') do |row|
#prod = Spree::Product.new()
#prod.name = row["name"]
#prod.shipping_category_id = row["shipping_category_id"]
#prod.description = row["description"]
#prod.available_on = row["available_on"]
#prod.meta_description = row["meta_description"]
#prod.meta_keywords = row["meta_keywords"]
#prod.tax_category_id = row["tax_category_id"]
#prod.shipping_category_id = row["shipping_category_id"]
#prod.promotionable = row["promotionable"]
#prod.meta_title = row["meta_title"]
#prod.featured = row["featured"]
#prod.supplier_id = row["supplier_id"]
#prod.master.price = row["master_price"]
#prod.save!
#var = Spree::Variant.create(cost_price: row["cost_price"], is_master:1, product_id: #prod.id, sku: row["sku"])
if #var.errors.present?
render json: #var #.errors and return
end
#price = Spree::Price.create(variant_id: #var.id)
if #price.errors.present?
render json: #price #.errors and return
end
end
redirect_to admin_products_path, notice: "products imported."
end
With Spree, the master is created when the product is saved, and the defaults for master is applied to it. Instead of creating a new master variant, #var = Spree::Variant.create, update the master instead. You update the master earlier in the file by assigning #prod.master.price = row["master_price"]. That code isn't failing because master is created with default values before the call to #product.save. The line #price = Spree::Price.create(variant_id: #var.id) Is associating an empty (default) Price with the #prod.master. That is already done, they are supposed to be unique, so there is probably fallout from doing this.
Avoid checking for errors after executing .create. If you want to validate errors, do so before saving. The way to do this is:
#var = Spree::Variant.new(cost_price: row["cost_price"], is_master:1, product_id: #prod.id, sku: row["sku"])
if #var.errors.present?
render json: #var #.errors and return
else #var.save
I would rewrite the block as this:
#prod = Spree::Product.new()
#prod.name = row["name"]
#prod.shipping_category_id = row["shipping_category_id"]
#prod.description = row["description"]
#prod.available_on = row["available_on"]
#prod.meta_description = row["meta_description"]
#prod.meta_keywords = row["meta_keywords"]
#prod.tax_category_id = row["tax_category_id"]
#prod.shipping_category_id = row["shipping_category_id"]
#prod.promotionable = row["promotionable"]
#prod.meta_title = row["meta_title"]
#prod.featured = row["featured"]
#prod.supplier_id = row["supplier_id"]
#prod.master.price = row["master_price"]
#prod.master.sku = row["sku"]
[#prod, #prod.master, #prod.master.price].each do |obj|
render json: obj if obj.errors.present?
end
#prod.save!
Here is the spree-core source: Spree Product Model
I am trying to get a users friends checkins, I am using omniauth and koala gem.
When a user gets saved this method hits:
def add_friends
friends_data = facebook.get_connections("me", "friends", :fields => "id, name, link, picture, gender, checkins")
friends_data.map do |h|
friend = Friend.new
friend.uid = h["id"]
friend.name = h["name"]
friend.image = h["picture"]
friend.gender = h["gender"]
friend.urls = h["link"]
friend.user_id = self.id
friend.save!
if (!h["checkins"].blank?)
checkin = Checkin.new
checkin.checkin_id = h["id"]
checkin.user_id = h["checkins"]["data"]["from"]["id"]
checkin.user_name = h["checkins"]["data"]["from"]["name"]
checkin.tags = h["checkins"]["tags"]["data"]["name"]
checkin.place_id = h["checkins"]["place"]["id"]
checkin.place_name = h["checkins"]["place"]["name"]
checkin.message = h["checkins"]["message"]
checkin.created_time = h["checkins"]["created_time"]
checkin.friend_id = friend.id
checkin.save!
end
end
end
But I get this error:
Koala::Facebook::APIError: HTTP 500: Response body: {"error_code":1,"error_msg":"An unknown error occurred"}
I dont really know what that means, any ideas? And does anybody know how to define a limit on checkins with the koala gem?
I tried something like this:
u.facebook.get_connections("me","friends", :fields => "checkins.limit(2)")
But I got the same error!
In fields, you're requesting information about a friend, but 'checkins' isn't a profile field, it's an entirely different connection altogether.
What you must do is loop through all the friend IDs and get the checkins for each:
friend_checkins = []
friend_ids = u.facebook.get_connections("me","friends", :fields => "id")
friend_ids.each do |friend_id|
friend_checkins << u.facebook.get_connections(friend_id,"checkins")
end
Also, when doing this, it would be a great time to look into batch requests with Koala, as you could potentially be making a lot of calls to the API here..