I am running systemu from within a delayed_job process. I had a job fail, and found this error in my delayed_jobs table:
execution expired
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:76:in `close'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:76:in `popen'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:76:in `block (2 levels) in systemu'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:168:in `quietly'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:75:in `block in systemu'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:236:in `call'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:236:in `block in tmpdir'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:221:in `loop'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:221:in `tmpdir'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:68:in `systemu'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:10:in `systemu'
It was working fine before, but this is a larger job I'm running. Is there some timeout that I need to set?
Thanks!
OK the problem was that the delayed_job timed out. I had to up the time limit with:
Delayed::Worker.max_run_time = 48.hours #2 days
in app/config/initializers/delayed_job_config.rb
Related
I'm trying to set Continuous integration with gitlab and a private docker registry.
I've followed the whole tutorials to set it up, I managed to set the registry tab on the left in a project environment but when I click on this tab, gitlab brings me immediately an error 500.
I use gitlab as a docker container, a private registry as another container.
I Have to edit the gitlab.yml file? I think this problem is caused by a certificate.
here are my logs located on : /var/log/gitlab/gitlab/production.log
Errno::ENOENT (No such file or directory # rb_sysopen - /certs/registry.key):
lib/json_web_token/rsa_token.rb:20:in `read'
lib/json_web_token/rsa_token.rb:20:in `key_data'
lib/json_web_token/rsa_token.rb:24:in `key'
lib/json_web_token/rsa_token.rb:28:in `public_key'
lib/json_web_token/rsa_token.rb:33:in `kid'
lib/json_web_token/rsa_token.rb:12:in `encoded'
app/services/auth/container_registry_authentication_service.rb:31:in `full_access_token'
app/models/container_repository.rb:13:in `registry'
app/models/container_repository.rb:7:in `client'
app/models/container_repository.rb:36:in `manifest'
app/models/container_repository.rb:41:in `tags'
app/models/container_repository.rb:53:in `has_tags?'
app/controllers/projects/registry/repositories_controller.rb:48:in `block (2 levels) in ensure_root_container_repository!'
app/controllers/projects/registry/repositories_controller.rb:47:in `tap'
app/controllers/projects/registry/repositories_controller.rb:47:in `block in ensure_root_container_repository!'
app/controllers/projects/registry/repositories_controller.rb:44:in `tap'
app/controllers/projects/registry/repositories_controller.rb:44:in `ensure_root_container_repository!'
lib/gitlab/i18n.rb:47:in `with_locale'
lib/gitlab/i18n.rb:53:in `with_user_locale'
app/controllers/application_controller.rb:337:in `set_locale'
lib/gitlab/middleware/multipart.rb:93:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/middleware/go.rb:17:in `call'
lib/gitlab/etag_caching/middleware.rb:11:in `call'
lib/gitlab/middleware/read_only.rb:30:in `call'
lib/gitlab/request_context.rb:18:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
Thanks for helping me, I've been stuck here for 1 month, (i'm a rookie in docker and CI)
Thanks for your help,
Grtz,
Tom
I am currently running sidekiq 4.1.2. I've never managed to be able to run more than a handfull of jobs concurrently. Recently it's been looking like I've run into an issue described in the Sidekiq's Troubleshooting WIKI called Too many connections to MongoDB. Apparently, mongoid 3 doesn't properly disconnect workers. However, I am using mongoid 5.1.3.
My issue surfaces when a job, while a few other jobs are running, tries to hit the database with a query:
Timeout::Error: Timed out attempting to dequeue connection after 30 sec.
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:190:in `wait_for_next!'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:176:in `block in dequeue_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:190:in `wait_for_next!'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:176:in `block in dequeue_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:172:in `loop'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:172:in `dequeue_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:62:in `block in dequeue'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:61:in `synchronize'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:61:in `dequeue'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool.rb:51:in `checkout'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool.rb:107:in `with_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/context.rb:63:in `with_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/operation/executable.rb:34:in `execute'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:80:in `send_initial_query'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:41:in `block in each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/retryable.rb:51:in `call'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/retryable.rb:51:in `read_with_retry'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:39:in `each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/query_cache.rb:207:in `each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual/mongo.rb:121:in `each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual/mongo.rb:295:in `map'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual/mongo.rb:295:in `map'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual.rb:20:in `map'
/home/me/applications/myapp/releases/20160618143407/app/jobs/myjob.rb:8:in `block in perform'
After one job fails, the other jobs fail soon after. This most often happens after a few handfull of jobs have finished succesfully, which could indicate that those jobs don't disconnect from the database. Looking at top I can't see that mongo load cpu is too much.
At the same time this started to occur, I noticed that my sidetiq 0.7.0 enabled recurring jobs were not scheduled properly. One job has stopped being queued, and others are only queued once after restart.
According to my Sidekiq web interface I have 1 queue, called default, with 25 threads. Max. 12-15 of them get busy at the same time.
Any idea how to troubleshoot this issue?
The default max pool queue size is 5. Bumping max_pool_size up to e.g. 25 will enable more connections to your db.
production:
clients:
default:
options:
max_pool_size: 25
Change adding wait_queue_timeout attribute in mongoid.yml production config:
production:
clients:
default:
uri: mongodb://aaaaa.com:27017/mongo
options:
connect_timeout: 30
wait_queue_timeout: 30
I have installed the que gem and I would like to run the tasks I'm scheduling with active job, so far I have added in config/application.rb
config.active_job.queue_adapter = :que
and it stores the jobs in the database correctly but when I try to run
rake que:work
It starts but jobs are not working and I have reviewed run_at field, also tried running setting the env to RAIls_ENV=development. Tried with
que ./config/application.rb
and generates and error
I, [2016-06-15T17:30:27.592632 #30871] INFO -- : {"lib":"que","hostname":"boris-Satellite-U845","pid":30871,"thread":9851100,"event":"worker_count_change","value":"4"}
I, [2016-06-15T17:30:27.592851 #30871] INFO -- : {"lib":"que","hostname":"boris-Satellite-U845","pid":30871,"thread":9851100,"event":"mode_change","value":"async"}
/home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/activerecord-4.2.6/lib/active_record/connection_handling.rb:109:in `connection_pool': ActiveRecord::ConnectionNotEstablished (ActiveRecord::ConnectionNotEstablished)
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/adapters/active_record.rb:54:in `checkout_activerecord_adapter'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/adapters/active_record.rb:7:in `checkout'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/job.rb:85:in `work'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:80:in `block in work_loop'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:75:in `loop'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:75:in `work_loop'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:19:in `block in initialize'
A bit late, but Que requires you to be running a Postgres Server for it work.
I would imagine this is why you are having issues.
I am trying to connect to sftp server via ruby console.
Code snippet:::
Net::SFTP.start(IP, USERNAME, password: PASS) do |sftp|
.....
end
It throws the following error::::
Net::SSH::Disconnect: disconnected: No Host (7)
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:188:in `block in poll_message'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:178:in `loop'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:178:in `poll_message'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:163:in `next_message'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/authentication/session.rb:95:in `block in next_message'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/authentication/session.rb:94:in `loop'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/authentication/session.rb:94:in `next_message'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/authentication/methods/none.rb:14:in `authenticate'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/authentication/session.rb:79:in `block in authenticate'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/authentication/session.rb:66:in `each'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh/authentication/session.rb:66:in `authenticate'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-ssh-2.9.2/lib/net/ssh.rb:211:in `start'
from /Users/venkatesan/.rvm/gems/ruby-1.9.3-p385#helpkit/gems/net-sftp-2.1.2/lib/net/sftp.rb:31:in `start'
from (irb):143
from /Users/venkatesan/.rvm/rubies/ruby-1.9.3-p385/bin/irb:16:in `<main>'
I tried using Net::SSH as well.. Got same error.
But when i do sftp USERNAME and then it prompts for password. It connects properly when i give the password in terminal. But cant connect via ruby code alone.
Can someone throw some light on it?
Finally found the reason.. I was giving wrong parameter. it was user#host. My parameter was incorrect.
Thanks guys
I have a really simple job:
class MyJob
#queue = :high
def self.perform(user_id)
user = User.find(user_id)
MyMailer.send_email(user).deliver
end
end
If I run it manually MyJob.perform(some_id)it works perfect. However, when Rescue is executing it, it returns this error:
Exception
Errno::ENOENT Error
No such file or directory - getcwd
shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:221:in `expand_path' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:221:in `initialize' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:251:in `new' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:251:in `instances' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/lookup_context.rb:16:in `<class:LookupContext>' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/lookup_context.rb:12:in `<module:ActionView>' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/lookup_context.rb:5:in `<top (required)>' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/abstract_controller/view_paths.rb:45:in `lookup_context' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:456:in `process' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:452:in `initialize' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:439:in `new' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:439:in `method_missing' releases/1111111111111/app/jobs/my_job.rb:6:in `perform'
Have any ideas why might be happening?
Thanks!
Yes Sky. You are right about it needing to be restarted.
Some people received this error after trying to run from an already deleted directory.
I received this error after switching databases and leaving the server running. The old server info was still showing up but I was getting this error. Restarted my rails server and everything works fine with the new db.
Basically it means that there is a significant state change on the server, and your environment needs to be reset/restarted.
I started having this same issue on my production environment. After some investigation I found that this was caused by my resque workers failing to be properly restarted on each capistrano deployment.