I'm trying to get all Stripe::BalanceTransaction except those they are already in my JsonStripeEvent
What I did =>
def perform(*args)
last_recorded_txt = REDIS.get('last_recorded_stripe_txn_last')
txns = Stripe::BalanceTransaction.all(limit: 100, expand: ['data.source', 'data.source.application_fee'], ending_before: last_recorded_txt)
REDIS.set('last_recorded_stripe_txn_last', txns.data[0].id) unless txns.data.empty?
txns.auto_paging_each do |txn|
if txn.type.eql?('charge') || txn.type.eql?('payment')
begin
JsonStripeEvent.create(data: txn.to_json)
rescue StandardError => e
Rails.logger.error "Error while saving data from stripe #{e}"
REDIS.set('last_recorded_stripe_txn_last', txn.id)
break
end
end
end
end
But It doesnt get the new one from the API.
Can anyone could help me for this ? :)
Thanks
I think it's because the way auto_paging_each works is almost opposite to what you expect :)
As you can see from its source, auto_paging_each calls Stripe::ListObject#next_page, which is implemented as follows:
def next_page(params={}, opts={})
return self.class.empty_list(opts) if !has_more
last_id = data.last.id
params = filters.merge({
:starting_after => last_id,
}).merge(params)
list(params, opts)
end
It simply takes the last (already fetched) item and adds its id as the starting_after filter.
So what happens:
You fetch 100 "latest" (let's say) records, ordered by descending date (default order for BalanceTransaction API according to Stripe docs)
When you call auto_paging_each on this dataset then, it takes the last record, adds its id as the
starting_after filter and repeats the query.
The repeated query returns nothing because there are noting newer (starting later) than the set you initially fetched.
As far as there are no more newer items available, the iteration stops after the first step
What you could do here:
First of all, ensure that my hypothesis is correct :) - put the breakpoint(s) inside Stripe::ListObject and check. Then 1) rewrite your code to use starting_after traversing logic instead of ending_before - it should work fine with auto_paging_each then - or 2) rewrite your code to control the fetching order manually.
Personally, I'd vote for (2): for me slightly more verbose (probably), but straightforward and "visible" control flow is better than poorly documented magic.
Related
On today's Advent of Code I needed to parse strings into integers. The function I wrote for that was
function fd(s::String, fromto::UnitRange)::Bool
try
parse(UInt, s) in fromto
catch ArgumentError
false
end
end
That function was called several times within an isvalid which was called for all inputs to count the number of valid things.
The result was always 0 and the respective tests kept failing. Then I extracted one of the failing test things and debugged into isvalid, it passed!
I rearraged a few things and tested more, the same thing kept happening:
Running the code regularly, fd never returned true.
When stepping through, I got true where expected.
After replacing the parse to
function fd(s::String, fromto::UnitRange)::Bool
parsed = tryparse(UInt, s)
if isnothing(parsed)
false
else
parsed in fromto
end
end
it immediately worked always and the exercise was solved.
Shouldn't these two versions of the function always return the same result? What happened here?
Update1
This was the example that I used:
println("Sample is ", isvalid(Dict(
"hcl" => "#623a2f", # check with regex
"ecl" => "grn", # checked with set
"pid" => "087499704", # check with regex
"hgt" => "74in", # check with regex
"iyr" => "2012", # this was parsed
"eyr" => "2030", # this was parsed
"byr" => "1980", # this was parsed
)))
Update 2
This post only has a subset of the code. If you want to try yourself, you can get the full file at GitHub. I also recorded a video showing the differing behavior with and without debugging.
The thing is demonstrated quite clearly with #assert fd("2010", 2000:2020) == fdt("2010", 2000:2020) failing in one scenario and not in the other.
I'm trying to get all connections (interactions) on a facebook page since a certain time period. I'm using the koala gem and filtering the request with "since: 1.month.ago.to_i" which seems to work fine. However, this gives me 25 results at a time. If I change the limit to 446 (the maximum it seems) that works better. But...if I use .next_page to give me the next set of results within the given time range, it instead just gives me a next set of results without obeying the time range.
For example, let's say I don't increase the limit and I have 25 results per request. I do something like:
#api.get_connections(#fan_page_id, "feed", {since: 1.month.ago.to_i})
let's assume there are 30 results for this and the first request gets me 25 (the default limit). then, if I do this:
#api.get_connections(#fan_page_id, "feed", {since: 1.month.ago.to_i}).next_page
instead of returning the last 5 results, it returns 25 more, 20 of which are not "since: 1.month.ago.to_i". I have a while loop cycling through the pages but I don't know where to stop since it just keep returning results to me no matter what as long as I keep calling .next_page.
is there a better way of doing this?
if not, what's the best way to check to make sure the post i'm looking at in the loop is still within the time range i want and to break out if not?
here's my code:
def perform(fan_page_id, pagination_options = {})
#since_date = pagination_options[:since_date] if pagination_options[:since_date]
#limit = pagination_options[:limit] if pagination_options[:limit]
#oauth = Koala::Facebook::OAuth.new
#api = Koala::Facebook::API.new #oauth.get_app_access_token
fb_page = #api.get_object(fan_page_id)
#fan_page_id = fb_page["id"]
# Collect all the users who liked, commented, or liked *and* commented on a post
process_posts(#api.get_connections(#fan_page_id, "feed", {since: #since_date})) do |post|
## do stuff based on each post
end
end
private
# Take each post from the specified feed and perform the provided
# code on each post in that feed.
#
# #param [Koala::Facebook::API::GraphCollection] feed An API response containing a page's feed
def process_posts(feed, options = {})
raise ArgumentError unless block_given?
current_feed = feed
begin
current_feed.each { |post| yield(post) }
current_feed = current_feed.next_page
end while current_feed.any?
end
current = #api.get_connections(#fan_page_id, "feed", {since: 1.month.ago.to_i})
next = current.next_page
next = next.next_page
.....
Please try these, I think they work.
I am currently using MongoDB for tracking of various things in a Rails 2 app. I am using the following code to see if MongoDB is up and running and, depending upon the status, displaying a link or an "Offline" message.
This is only for admins, so it's not mission-critical, as the app will continue to run without MongoDB, but I do want to keep disabling the link in the menu when it's not running. However, I don't like the overhead of the below code (doesn't take long to run, but hope that there is a cleaner, faster way):
def verify_mongodb_status
begin
track = Track.first
#mongodb_running = true
rescue
#mongodb_running = false
logger.debug("***MongoDB not running.***")
notify_admin_about_errors("***MongoDB is not running***)
end
end
EDIT: I forgot to mention that I'm already doing a before_filter for this; the method sits in application_controller.rb.
I decided to go with action_caching as there doesn't seem to be a great way to do this. The result was quite a large speed increase from ~120ms to ~16-25ms:
def verify_mongodb_status
begin
track = Track.first
#mongodb_running = true
rescue => e
#mongodb_running = false
logger.debug("***MONGODB OFFLINE***: #{e}")
notify_admin_about_errors("MongoDB", "MongoDB error:\n#{e}", nil)
expire_action :action => :verify_mongodb_status
return
end
end
I'm adding logic now to keep from getting bombarded by emails when MongoDB goes offline (1 is enough).
I am retrieving results from NCBI's online Blast tool with 'net/http' and 'uri'. To do this I have to search through an html page to check if one of the lines is "Status=WAITING" or "Status=READY". When the Blast tool has finished the status will change to ready and results will be posted on the html page.
I have a working version to check the status and then retrieve the information that I need, but it is inefficient and is broken into two methods when I believe that there could be some way to put them into one.
def waitForBlast(rid)
get = Net::HTTP.post_form(URI.parse('http://www.ncbi.nlm.nih.gov/blast/Blast.cgi?'), {:RID => "#{rid}", :CMD => 'Get'})
get.body.each{|line| (waitForBlast(rid) if line.strip == "Status=WAITING") if line[/Status=/]}
end
def returnBlast(rid)
blast_array = Array.new
get = Net::HTTP.post_form(URI.parse('http://www.ncbi.nlm.nih.gov/blast/Blast.cgi?'), {:RID => "#{rid}", :CMD => 'Get'})
get.body.each{|line| blast_array.push(line[/<a href=#\d+>/][/\d+/]) if line[/<a href=#\d+>/]}
return blast_array
end
The first method checks the status and is my main concern because it is recursive. I believe(and correct me if I'm wrong) that designed as is takes too much computing power when all that I need is some way to recheck the results within the same method(adding in a time delay is a bonus). The second method is fine, but I would prefer if it was combined with the first somehow. Any help appreciated.
Take a look at this implementation. This is what he does:
res='http://www.ncbi.nlm.nih.gov/blast/Blast.cgi?CMD=Get&FORMAT_OBJECT=SearchInfo&RID=' + #rid
while status = open(res).read.scan(/Status=(.*?)$/).to_s=='WAITING'
#logger.debug("Status=WAITING")
sleep(3)
end
I think using the string scanner might be a bit more efficient than iterating over every line in the page, but I haven't looked at it's implementation so I may be wrong.
Hi I've this piece of code
class Place < ActiveRecord::Base
def self.find_or_create_by_latlon(lat, lon)
place_id = call_external_webapi
result = Place.where(:place_id => place_id).limit(1)
result = Place.create(:place_id => place_id, ... ) if result.empty? #!
result
end
end
Then I'd like to do in another model or controller
p = Post.new
p.place = Place.find_or_create_by_latlon(XXXXX, YYYYY) # race-condition
p.save
But Place.find_or_create_by_latlon takes too much time to get the data if the action executed is create and sometimes in production p.place is nil.
How can I force to wait for the response before execute p.save ?
thanks for your advices
You're right that this is a race condition and it can often be triggered by people who double click submit buttons on forms. What you might do is loop back if you encounter an error.
result = Place.find_by_place_id(...) ||
Place.create(...) ||
Place.find_by_place_id(...)
There are more elegant ways of doing this, but the basic method is here.
I had to deal with a similar problem. In our backend a user is is created from a token if the user doesn't exist. AFTER a user record is already created, a slow API call gets sent to update the users information.
def self.find_or_create_by_facebook_id(facebook_id)
User.find_by_facebook_id(facebook_id) || User.create(facebook_id: facebook_id)
rescue ActiveRecord::RecordNotUnique => e
User.find_by_facebook_id(facebook_id)
end
def self.find_by_token(token)
facebook_id = get_facebook_id_from_token(token)
user = User.find_or_create_by_facebook_id(facebook_id)
if user.unregistered?
user.update_profile_from_facebook
user.mark_as_registered
user.save
end
return user
end
The step of the strategy is to first remove the slow API call (in my case update_profile_from_facebook) from the create method. Because the operation takes so long, you are significantly increasing the chance of duplicate insert operations when you include the operation as part of the call to create.
The second step is to add a unique constraint to your database column to ensure duplicates aren't created.
The final step is to create a function that will catch the RecordNotUnique exception in the rare case where duplicate insert operations are sent to the database.
This may not be the most elegant solution but it worked for us.
I hit this inside a sidekick job that retries and gets the error repeatedly and eventually clears itself. The best explanation I've found is on a blog post here. The gist is that postgres keeps an internally stored value for incrementing the primary key that gets messed up somehow. This rings true for me because I'm setting the primary key and not just using an incremented value so that's likely how this cropped up. The solution from the comments in the link above appears to be to call ActiveRecord::Base.connection.reset_pk_sequence!(table_name) This cleared up the issue for me.
begin
result = Place.where(:place_id => place_id).limit(1)
result = Place.create(:place_id => place_id, ... ) if result.empty? #!
rescue ActiveRecord::StatementInvalid => error
#save_retry_count = (#save_retry_count || 1)
ActiveRecord::Base.connection.reset_pk_sequence!(:place)
retry if( (#save_retry_count -= 1) >= 0 )
raise error
end