I'm trying out Heroku's Temporize Add-on and am having trouble interacting with their recurring API. Any help would be appreciated!
When I make a POST request to create a new recurring task, I receive the following message: "failed to parse either date or cron expression".
Unfortunately I get this message with every cron statement I try, even the one used in their example. To make matters worse, the API endpoint they list on their website is different than what they list in their API docs "/events/cron/url" vs "/events/when/cron/url".
Some cron statements I've tried:
0 12 * * * *
0 0 12 1/1 * ? *
15 * * * * *
The code and cron syntax I'm using is similar to what they use on their website but I'll post it here just in case. For what it's worth the "single task" endpoint works as expected. Hopefully I'm just doing something silly...
Links:
http://www.temporize.net; http://docs.temporize.apiary.io
require 'rubygems'
require 'httparty'
require 'cgi'
require 'time'
class Temporize
include HTTParty
base_uri "https://api.temporize.net/v1"
attr_accessor :credentials
def initialize
uri = URI(ENV["TEMPORIZE_URL"])
self.credentials = {:username => uri.user, :password => uri.password}
end
# Schedule a test event to run right away
def single(callback_url, date = Time.now.utc.iso8601)
url = CGI::escape(callback_url)
Temporize.post("/events/#{date}/#{url}", :basic_auth => self.credentials)
end
# Schedule a test event to run on a schedule
def recurring(callback_url, cron_schedule)
cron = CGI::escape(cron_schedule) # ex "5 10 * * ?"
url = CGI::escape(callback_url)
Temporize.post("/events/#{cron}/#{url}", :basic_auth => self.credentials)
end
end
t = Temporize.new
t.recurring("http://example.com/callback", "0 12 * * * *")
# => Result: failed to parse either date or cron expression
Per the docs, cron syntax, the syntax is:
<minutes> <hours> <day of month> <month> <day of week>
All of your examples have 6 fields, not 5. Double check to ensure you have 5 space-separated fields for your cron syntax.
Related
I have a rake task which loops over pages of card game database and checks for the cards in each deck. Until recently this was working fine (it's checked 34000 pages of 25 decks each no problem) but recently this has stopped working when I run the rake task and I get the error:
JSON::ParserError: 765: unexpected token at ''
In order to debug this I have tried running each line of the get request and json parse manually in the rails console and it works fine every time. Weirder still I have installed pry and it works every time I go through the json parse manually with pry (takes ages though).
Here is the rake task:
desc "Create Cards"
require 'net/http'
require 'json'
task :create_cards => :environment do
# Get the total number of pages of decks
uri = URI("https://www.keyforgegame.com/api/decks/")
response = Net::HTTP.get(URI(uri))
json = JSON.parse(response)
deck_count = json["count"]
# Set variables
page_number = 1
page_size = 25 # 25 is the max page size
page_limit = deck_count / 25
card_list = Card.where(is_maverick: false)
# Updates Card List (non-mavericks) - there are 740 cards so we stop when we have that many
# example uri: https://www.keyforgegame.com/api/decks/?page=1&page_size=30&search=&links=cards
puts "Updating Card List..."
until page_number > page_limit || Card.where(is_maverick: false).length == 740
uri = URI("https://www.keyforgegame.com/api/decks/?page=#{page_number}&page_size=#{page_size}&search=&links=cards")
response = Net::HTTP.get(URI(uri))
json = JSON.parse(response) # task errors here!
cards = json["_linked"]["cards"]
cards.each do |card|
unless Card.exists?(:card_id => card["id"])
Card.create({
card_id: card["id"],
amber: card["amber"],
card_number: card["card_number"],
card_text: card["card_text"],
card_title: card["card_title"],
card_type: card["card_type"],
expansion: card["expansion"],
flavor_text: card["flavor_text"],
front_image: card["front_image"],
house: card["house"],
is_maverick: card["is_maverick"],
power: card["power"],
rarity: card["rarity"],
traits: card["traits"],
})
end
end
puts "#{page_number}/#{page_limit} - Cards: #{Card.where(is_maverick: false).length}"
page_number = (page_number + 1)
end
end
The first json parse where it gets the total number of pages of decks works okay. It's the json parse in the until block that is failing (I've marked the line with a comment to that effect).
As I say, if I try this in the console it works fine and I can parse the json without error, literally copying and pasting the lines from the file into the rails console.
Since you're looping over an api, it's possible there are rate limits. Public APIs normally have per second rate limits. You could try adding a sleep to slow down your requests, not sure how many your making per second. I tested with a simple loop and looks like response returns an empty string if you hit the api too fast.
url='https://www.keyforgegame.com/api/decks/?page=1&page_size=30&search=&links=cards'
uri = URI(url)
i = 1
1000.times do
puts i.to_s
i += 1
response = Net::HTTP.get(URI(uri))
begin
j = JSON.parse(response)
rescue
puts response
#= ""
end
end
I played with this until the loop stopped returning empty string after the 3rd request and got it to work with sleep 5 inside each loop, so you can probably add as the first line inside your loop. But you should probably add error handling to your rake task in case you encounter any other API errors.
So for now you can probably just do this
until page_number > page_limit || Card.where(is_maverick: false).length == 740
sleep 5
# rest of your loop code, maybe add a rescue like I've shown
end
Is there a way in RSpec to show every single test duration and not just the total suite duration?
Now we have
Finished in 7 minutes 31 seconds (files took 4.71 seconds to load)
but I'd like to have something like
User accesses home and
he can sign up (finished in 1.30 seconds)
he can visit profile (finished in 3 seconds)
.
.
.
Finished in 7 minutes 31 seconds (files took 4.71 seconds to load)
You can use rspec --profile N, which would show you the top N slowest examples.
For a quick solution see #maximf's answer. For an alternative solution, you could write your own rspec formatter, which would give you greater control over what you are measuring.
For example, extnding rspec's base text formatter:
RSpec::Support.require_rpec_core "formatters/base_text_formatter"
module RSpec::Core::Formatters
class TimeFormatter < BaseTextFormatter
Formatters.register self, :example_started, :example_passed
attr_accessor :example_start, :longest_example, :longest_time
def initialize(output)
#longest_time = 0
super(output)
end
def example_started(example)
#example_start = Time.now
super(example)
end
def example_passed(example)
time_taken = Time.now - #example_start
output.puts "Finished #{example.example.full_description} and took #{Helpers.format_duration(time_taken)}"
if #time_taken > #longest_time
#longest_example = example
#longest_time = time_taken
end
super(example)
end
def dump_summary(summary)
super(summary)
output.puts
output.puts "The longest example was #{#longest_example.example.full_Description} at #{Helpers.format_duration(#longest_time)}"
end
end
end
Note that this will only log times on passed examples, but you could add an example_failed failed to do similar, it also only works with RSpec 3. This is based on my work on my own formatter: https://github.com/yule/dots-formatter
Instead of doing rspec --profile Neverytime we run specs (as #maximf said), we can add it to our RSpec configuration:
RSpec.configure do |config|
config.profile_examples = 10
end
I meet an encoding problem... No errors in the console, but the output is not well encoded.
I must use Digest::SHA1.hexdigest on a string and then must pack the result.
The below example should outputs '{´p)ODýGΗ£Iô8ü:iÀ' but it outputs '{?p)OD?GΗ?I?8?:i?' in the console and '{�p)OD�G^BΗ�I�8^D�:i�' in the log file.
So, my variable called pack equals '{?p)OD?GΗ?I?8?:i?' and not '{´p)ODýGΗ£Iô8ü:iÀ'. That's a big problem... I'm doing it in a Rails task.
Any idea guys?
Thanks
# encoding: utf-8
require 'digest/sha1'
namespace :my_app do
namespace :check do
desc "Description"
task :weather => :environment do
hexdigest = Digest::SHA1.hexdigest('29d185d98c984a359e6e6f26a0474269partner=100043982026&code=34154&profile=large&filter=movie&striptags=synopsis%2Csynopsisshort&format=json&sed=20130527')
pack = [hexdigest].pack("H*")
puts pack # => {?p)OD?GΗ?I?8?:i?
puts '{´p)ODýGΗ£Iô8ü:iÀ' # => {´p)ODýGΗ£Iô8ü:iÀ
end
end
end
This is what I did (my conversion from PHP to Ruby)
# encoding: utf-8
require 'open-uri'
require 'base64'
require 'digest/sha1'
class Allocine
$_api_url = 'http://api.allocine.fr/rest/v3'
$_partner_key
$_secret_key
$_user_agent = 'Dalvik/1.6.0 (Linux; U; Android 4.2.2; Nexus 4 Build/JDQ39E)'
def initialize (partner_key, secret_key)
$_partner_key = partner_key
$_secret_key = secret_key
end
def get(id)
# build the params
params = { 'partner' => $_partner_key,
'code' => id,
'profile' => 'large',
'filter' => 'movie',
'striptags' => 'synopsis,synopsisshort',
'format' => 'json' }
# do the request
response = _do_request('movie', params)
return response
end
private
def _do_request(method, params)
# build the URL
query_url = $_api_url + '/' + method
# new algo to build the query
http_build_query = Rack::Utils.build_query(params)
sed = DateTime.now.strftime('%Y%m%d')
sig = URI::encode(Base64.encode64(Digest::SHA1.digest($_secret_key + http_build_query + '&sed=' + sed)))
return sig
end
end
Then call
allocine = Allocine.new(ALLOCINE_PARTNER_KEY, ALLOCINE_SECRET_KEY)
puts allocine.get('any ID')
get method return 'e7RwKU9E%2FUcCzpejSfQ4BPw6acA%3D' in PHP and 'cPf6I4ZP0qHQTSVgdKTbSspivzg=%0A' in Ruby...
thanks again
I think this "encoding" issue has turned up due to debugging other parts of a conversion from PHP to Ruby. The target API that will consume a digest of params looks like it will accept a signature variable constructed in Ruby as follows (edit: well this is guess, there may also be relevant differences between Ruby and PHP in URI encoding and base64 defaults):
require 'digest/sha1'
require 'base64'
require 'uri'
sig_data = 'edhefhekjfhejk8edfefefefwjw69partne...'
sig = URI.encode( Base64.encode64( Digest::SHA1.digest( sig_data ) ) )
=> "+ZabHg22Wyf7keVGNWTc4sK1ez4=%0A"
The exact construction of sig_data from the parameters that are being signed is also important. That is generated by the PHP method http_build_query, and I do not know what order or escaping that will apply to input params. If your Ruby version gets them in a different order, or escapes differently to PHP, the signature will be wrong (edit: Actually it is possible we are looking here for a signature on the exact query string sent the API - I don't know). It is possibly an issue of that sort that has led you down the rabbit hole of how the signature is constructed?
Thank you guys for your help.
Problem is solved. With the following code I obtain exactly the same string as with PHP:
http_build_query = Rack::Utils.build_query(params)
sed = DateTime.now.strftime('%Y%m%d')
sig = CGI::escape(Base64.strict_encode64(Digest::SHA1.digest($_secret_key + http_build_query + '&sed=' + sed)))
Now I've another problem for which I opened a new question here.
thanks you very much.
Good afternoon,
I have two separate, but related apps. They should both have their own background queues (read: separate Sidekiq & Redis processes). However, I'd like to occasionally be able to push jobs onto app2's queue from app1.
From a simple queue/push perspective, it would be easy to do this if app1 did not have an existing Sidekiq/Redis stack:
# In a process, far far away
# Configure client
Sidekiq.configure_client do |config|
config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
end
# Push jobs without class definition
Sidekiq::Client.push('class' => 'Example::Workers::Trace', 'args' => ['hello!'])
# Push jobs overriding default's
Sidekiq::Client.push('queue' => 'example', 'retry' => 3, 'class' => 'Example::Workers::Trace', 'args' => ['hello!'])
However given that I would already have called a Sidekiq.configure_client and Sidekiq.configure_server from app1, there's probably a step in between here where something needs to happen.
Obviously I could just take the serialization and normalization code straight from inside Sidekiq and manually push onto app2's redis queue, but that seems like a brittle solution. I'd like to be able to use the Client.push functionality.
I suppose my ideal solution would be someting like:
SidekiqTWO.configure_client { remote connection..... }
SidekiqTWO::Client.push(job....)
Or even:
$redis_remote = remote_connection.....
Sidekiq::Client.push(job, $redis_remote)
Obviously a bit facetious, but that's my ideal use case.
Thanks!
So one thing is that According to the FAQ, "The Sidekiq message format is quite simple and stable: it's just a Hash in JSON format." Emphasis mine-- I don't think sending JSON to sidekiq is too brittle to do. Especially when you want fine-grained control around which Redis instance you send the jobs to, as in the OP's situation, I'd probably just write a little wrapper that would let me indicate a Redis instance along with the job being enqueued.
For Kevin Bedell's more general situation to round-robin jobs into Redis instances, I'd imagine you don't want to have the control of which Redis instance is used-- you just want to enqueue and have the distribution be managed automatically. It looks like only one person has requested this so far, and they came up with a solution that uses Redis::Distributed:
datastore_config = YAML.load(ERB.new(File.read(File.join(Rails.root, "config", "redis.yml"))).result)
datastore_config = datastore_config["defaults"].merge(datastore_config[::Rails.env])
if datastore_config[:host].is_a?(Array)
if datastore_config[:host].length == 1
datastore_config[:host] = datastore_config[:host].first
else
datastore_config = datastore_config[:host].map do |host|
host_has_port = host =~ /:\d+\z/
if host_has_port
"redis://#{host}/#{datastore_config[:db] || 0}"
else
"redis://#{host}:#{datastore_config[:port] || 6379}/#{datastore_config[:db] || 0}"
end
end
end
end
Sidekiq.configure_server do |config|
config.redis = ::ConnectionPool.new(:size => Sidekiq.options[:concurrency] + 2, :timeout => 2) do
redis = if datastore_config.is_a? Array
Redis::Distributed.new(datastore_config)
else
Redis.new(datastore_config)
end
Redis::Namespace.new('resque', :redis => redis)
end
end
Another thing to consider in your quest to get high-availability and fail-over is to get Sidekiq Pro which includes reliability features: "The Sidekiq Pro client can withstand transient Redis outages. It will enqueue jobs locally upon error and attempt to deliver those jobs once connectivity is restored." Since sidekiq is for background processes anyway, a short delay if a Redis instance goes down should not affect your application. If one of your two Redis instances goes down and you're using round robin, you've still lost some jobs unless you're using this feature.
As carols10cents says its pretty simple but as I always like to encapsulate the capability and be able to reuse it in other projects I updated an idea from a blog from Hotel Tonight. This following solution improves upon Hotel Tonight's that does not survive Rails 4.1 & Spring preloader.
Currently I make do with adding the following files to lib/remote_sidekiq/:
remote_sidekiq.rb
class RemoteSidekiq
class_attribute :redis_pool
end
remote_sidekiq_worker.rb
require 'sidekiq'
require 'sidekiq/client'
module RemoteSidekiqWorker
def client
pool = RemoteSidekiq.redis_pool || Thread.current[:sidekiq_via_pool] || Sidekiq.redis_pool
Sidekiq::Client.new(pool)
end
def push(worker_name, attrs = [], queue_name = "default")
client.push('args' => attrs, 'class' => worker_name, 'queue' => queue_name)
end
end
You need to create a initializer that sets redis_pool
config/initializers/remote_sidekiq.rb
url = ENV.fetch("REDISCLOUD_URL")
namespace = 'primary'
redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))
RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }
EDIT by Aleks:
In never versions of sidekiq, instead of lines:
redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))
RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }
use lines:
redis_remote_options = {
namespace: "yournamespace",
url: ENV.fetch("REDISCLOUD_URL")
}
RemoteSidekiq.redis_pool = Sidekiq::RedisConnection.create(redis_remote_options)
You can then simply the include RemoteSidekiqWorker module wherever you want. Job done!
**** FOR MORE LARGER ENVIRONMENTS ****
Adding in RemoteWorker Models adds extra benefits:
You can reuse the RemoteWorkers everywhere including the system that has access to the target sidekiq workers. This is transparent to the caller. To use the "RemoteWorkers" form within the target sidekiq system simply do not use an initializer as it will default to using the local Sidekiq client.
Using RemoteWorkers ensure correct arguments are always sent in (the code = documentation)
Scaling up by creating more complicated Sidekiq architectures is transparent to the caller.
Here is an example RemoteWorker
class RemoteTraceWorker
include RemoteSidekiqWorker
include ActiveModel::Model
attr_accessor :message
validates :message, presence: true
def perform_async
if valid?
push(worker_name, worker_args)
else
raise ActiveModel::StrictValidationFailed, errors.full_messages
end
end
private
def worker_name
:TraceWorker.to_s
end
def worker_args
[message]
end
end
I came across this and ran into some issues because I'm using ActiveJob, which complicates how messages are read out of the queue.
Building on ARO's answer, you will still need the redis_pool setup:
remote_sidekiq.rb
class RemoteSidekiq
class_attribute :redis_pool
end
config/initializers/remote_sidekiq.rb
url = ENV.fetch("REDISCLOUD_URL")
namespace = 'primary'
redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))
RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }
Now instead of the worker we'll create an ActiveJob Adapter to queue the request:
lib/active_job/queue_adapters/remote_sidekiq_adapter.rb
require 'sidekiq'
module ActiveJob
module QueueAdapters
class RemoteSidekiqAdapter
def enqueue(job)
#Sidekiq::Client does not support symbols as keys
job.provider_job_id = client.push \
"class" => ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper,
"wrapped" => job.class.to_s,
"queue" => job.queue_name,
"args" => [ job.serialize ]
end
def enqueue_at(job, timestamp)
job.provider_job_id = client.push \
"class" => ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper,
"wrapped" => job.class.to_s,
"queue" => job.queue_name,
"args" => [ job.serialize ],
"at" => timestamp
end
def client
#client ||= ::Sidekiq::Client.new(RemoteSidekiq.redis_pool)
end
end
end
end
I can use the adapter to queue the events now:
require 'active_job/queue_adapters/remote_sidekiq_adapter'
class RemoteJob < ActiveJob::Base
self.queue_adapter = :remote_sidekiq
queue_as :default
def perform(_event_name, _data)
fail "
This job should not run here; intended to hook into
ActiveJob and run in another system
"
end
end
I can now queue the job using the normal ActiveJob api. Whatever app reads this out of the queue will need to have a matching RemoteJob available to perform the action.
I'm trying to do the following:
Run a Worker and a method within it every 15 minutes
Have a log of the job last runtime, in the database table
bdrd_job_queue.
What I've done:
I have a schedule every 15 minutes in my backgroundRB.yml file
The method call has a persistent_job.finish! call, but it's not working,
because the persistent_job object is nil.
How can I ensure it's logged in the DB, but still automatically
scheduled from backgroundRB.yml?
I was finally able to do it.
The workaround is to schedule a task that will queue it to the database, scheduled to run right away.
In your worker ...
class NotificationWorker < BackgrounDRb::MetaWorker
set_worker_name :notification_worker
def create(args = nil)
end
def queue_notify_changes(args = nil)
BdrbJobQueue.insert_job(:worker_name => 'notification_worker',
:worker_method => 'notify_new_changes_DAEMON',
:args => 'hello_world',
:scheduled_at => Time.now.utc,
:job_key => 'email_changes_notification_task')
end
def notify_new_changes_DAEMON
#Do Incredibly cool stuff here
end
In the config file backgroundrb.yml
---
:backgroundrb:
:ip: 0.0.0.0
:port: 11006
:environment: production
:log: foreground
:debug_log: true
:persistent_disabled: false
:persistent_delay: 10
:schedules:
:notification_worker:
:queue_notify_changes:
:trigger_args: 0 0 0 * * *