Today airbrake reported an exception. Its summary said that the problem occures when sidekiq tries retry the job after a failure. Here's what summary params look like:
{
"retry"=>"true",
"queue"=>"default",
"class"=>"AdwordsReportWorker",
"args"=>"[\"2\", \"2012-11-13\"]",
"jid"=>"51d568e46c412adc327153c8",
"error_message"=>"wrong number of arguments(1 for 0)",
"error_class"=>"ArgumentError",
"failed_at"=>"2012-11-14 13:56:12 UTC",
"retry_count"=>"0",
"controller"=>"",
"action"=>""
}
I seems that the exception is only taking palce when the job fails and retries. I would like to debug this, but I can't get my head around where to start :-(
My questions are:
Under what conditions worker understands it failed finishing the job (from which he descides to retry)? How do I force the worker to fail in attempt to, say, reproduce the problem?
Is there a good tutorial on debugging/examples sidekiq workers?
I am using sidekiq 2.3.3. Will upgrading to newer version solve the problem?
Bonus track.
The actual stacktrace unexpectedly ends up in
[GEM_ROOT]/gems/activerecord-3.2.8/lib/active_record/associations/association.rb:98:in `initialize'
I couldn't find how do debug sidekiq, but apperently the problem I am referring to is a known bug in Rails.
Here are a few lines:
https://github.com/mperham/sidekiq/issues/601
https://github.com/rails/rails/issues/7770
Related
some other team is calling our FileNet custom app for searching documents. I believe some of the users are facing intermittent failure because we are getting tickets(although none of the user has reported this issue) for the error below and I tried validating our service with different scenarios and they all worked but I don't know what's causing this error. any suggestions/help would be greatly appreciated.
<stackTrace>
at com.filenet.apiimpl.transport.ejb.EJBSession.throwException(EJBSession.java:1122)
at com.filenet.apiimpl.transport.ejb.EJBSession.throwException(EJBSession.java:1045)
at com.filenet.apiimpl.transport.ejb.EJBSession$EJBImpl._getObjects(EJBSession.java:650)
at com.filenet.apiimpl.transport.ejb.EJBSession$EJBImpl.getObjects(EJBSession.java:575)
at com.filenet.apiimpl.transport.ejb.EJBSession.getObjects(EJBSession.java:471)
at com.filenet.apiimpl.util.SessionHandle.getObjects(SessionHandle.java:346)
at com.filenet.apiimpl.core.Session.callGetObjects(Session.java:132)
at com.filenet.apiimpl.core.Session.executeGetObject(Session.java:340)
at com.filenet.apiimpl.core.Session.getObject(Session.java:354)
at com.filenet.apiimpl.core.DispatchEntries.FetchObject_28(DispatchEntries.java:907)
at com.filenet.apiimpl.core.ObjectStoreImpl.fetchObject(ObjectStoreImpl.java:1643)
at com.filenet.api.core.Factory$ClassDescription.fetchInstance(Factory.java:21761)
at ecm.service.p8ceservices.implementation.integration.SearchP8ObjectAdapter.retrievePropertyDefinitions(SearchP8ObjectAdapter.java:352)
at ecm.service.p8ceservices.implementation.integration.SearchP8ObjectAdapter.integrate(SearchP8ObjectAdapter.java:158)
at ecm.service.p8ceservices.implementation.integration.ContentEngineAdapter.execute(ContentEngineAdapter.java:37)
</stackTrace>
</exception><exception name="java.lang.IndexOutOfBoundsException" message="Data to be copied (length 2124) exceeds space available (480)" sequence="0" guid="sfr2mx3l:jewe2wkf:00000000:00000121"><source class="com.ibm.rmi.util.buffer.SimpleByteBuffer" archive="" vendor="" version="" /><stackTrace>
at com.ibm.rmi.util.buffer.SimpleByteBuffer.write(SimpleByteBuffer.java:166) at com.ibm.rmi.iiop.ClientRequestImpl.reInvoke(ClientRequestImpl.java:489)
at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:637) at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1377)
at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:695) at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1407)
at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:484) at com.filenet.apiimpl.transport.ejbstubs._Engine_Stub.getObjects(Unknown Source)
at com.filenet.apiimpl.transport.ejb.EJBSession$EJBImpl._getObjects(EJBSession.java:638)</stackTrace>
It could be the following issue:
ClientRequestImpl.reInvoke retry requests which failed due to errors.
The retry itself fails with IndexOutOfBoundsException because ORB is using fixed length SimpleByteBuffer.
See also this IBM Fix
I'm following this tutorial and I've double and triple checked the code compared to the tutorial and their GH repo and I can't figure out why I'm getting this error when running mix test
I'm pretty green when it comes to Elixir so I'm not terribly good at debugging yet so please excuse me if this is super simple
board_server.ex Gist
game_of_life.ex Gist
And my test file
defmodule GameOfLife.BoardServerTest do
use ExUnit.Case
doctest GameOfLife.BoardServer
end
The error I'm getting is:
test/board_server_test.exs:3
** (exit) exited in: GenServer.call({:global, GameOfLife.BoardServer}, {:start_game, 1000}, 5000)
** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
stacktrace:
(elixir) lib/gen_server.ex:729: GenServer.call/3
(for doctest at) lib/game_of_life/board_server.ex:121: (test)
Thank you in advance
This error states that the server is not started. You can start it with the GameOfLife.BoardServer.start_link function.
In short, you should start your server with GenServer#start or GenServer#start_link functions.
GenServer docs
We are using distributed JMS on weblogic and some times ResourceAccessException occurs. We tried deleting filestores, restarting servers and changing JTA timeout to 300 seconds. (default is 30 seconds).
But we are getting the same error. What could be the reason?
Thanks
Caused by: javax.transaction.SystemException: start() failed on resource 'WLStore_COLUMBUS-ADA-PROD-DMN_ColumbusADAFileStore': XAER_RMERR : A resource manager error has occured in the transaction branch
weblogic.transaction.internal.ResourceAccessException: Transaction has timed out when making request to XAResource 'WLStore_COLUMBUS-ADA-PROD-DMN_ColumbusADAFileStore'.
at weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:670)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1230)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1164)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:296)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:522)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:449)
at weblogic.store.gxa.internal.GXAResourceImpl.enlist(GXAResourceImpl.java:442)
at weblogic.messaging.kernel.internal.KernelImpl.getGXATransaction(KernelImpl.java:570)
at weblogic.messaging.kernel.internal.QueueImpl.send(QueueImpl.java:329)
at weblogic.jms.backend.BEDestinationImpl.sendIssueMessage(BEDestinationImpl.java:1873)
at weblogic.jms.backend.BEDestinationImpl.send(BEDestinationImpl.java:2108)
at weblogic.jms.backend.BEDestinationImpl.wrappedSend(BEDestinationImpl.java:2051)
at weblogic.jms.backend.BEDestinationImpl.invoke(BEDestinationImpl.java:1539)
at weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:961)
at weblogic.messaging.dispatcher.DispatcherImpl.dispatchAsyncInternal(DispatcherImpl.java:139)
at weblogic.messaging.dispatcher.DispatcherImpl.dispatchAsync(DispatcherImpl.java:115)
at weblogic.messaging.dispatcher.Request.dispatchAsync(Request.java:1303)
at weblogic.jms.dispatcher.Request.dispatchAsync(Request.java:96)
at weblogic.jms.frontend.FEProducer.doDispatch(FEProducer.java:888)
at weblogic.jms.frontend.FEProducer.sendRetryDestination(FEProducer.java:1021)
at weblogic.jms.frontend.FEProducer.send(FEProducer.java:1405)
at weblogic.jms.frontend.FEProducer.invoke(FEProducer.java:1466)
at weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:961)
at weblogic.messaging.dispatcher.DispatcherImpl.syncRequest(DispatcherImpl.java:184)
at weblogic.messaging.dispatcher.DispatcherImpl.dispatchSyncNoTran(DispatcherImpl.java:287)
at weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:59)
at weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1293)
at weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:796)
at weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:541)
at weblogic.jms.client.JMSProducer.sendWithListener(JMSProducer.java:394)
at weblogic.jms.client.JMSProducer.send(JMSProducer.java:384)
at weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:970)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:592)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:569)
at org.springframework.jms.core.JmsTemplate$4.doInJms(JmsTemplate.java:546)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:543)
at com.turkcellteknoloji.calypso.jms.SendToJms.insertJms(SendToJms.java:35)
at com.turkcellteknoloji.calypso.action.impl.SentTransactionToJms.execute(SentTransactionToJms.java:15)
at com.turkcellteknoloji.calypso.service.impl.Service.execute(Service.java:36)
at com.turkcellteknoloji.calypso.service.impl.JmsService.onMessage(JmsService.java:21)
at com.turkcellteknoloji.calypso.service.impl.JmsService.onMessage(JmsService.java:12)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:534)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:323)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:977)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:969)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:871)
at java.lang.Thread.run(Thread.java:619)
Caused by: weblogic.transaction.internal.TimedOutException: Transaction has timed out when making request to XAResource 'WLStore_COLUMBUS-ADA-PROD-DMN_ColumbusADAFileStore'.
at weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:673)
... 50 more
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1188)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:296)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:522)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:449)
at weblogic.store.gxa.internal.GXAResourceImpl.enlist(GXAResourceImpl.java:442)
... 44 more
Have you tried to change the"Synchronous Write Policy"? It affects the JMS file store's performance, scalability, and reliability.
Adding similar issue found with Weblogic hosted application kept failing with little or no reason with stack trace:
Caused by: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Error enlisting GXA transaction
at weblogic.jms.dispatcher.DispatcherAdapter.convertToJMSExceptionAndThrow(DispatcherAdapter.java:110)
at weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncTran(DispatcherAdapter.java:53)
at weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1300)
at weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:807)
at weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:543)
at weblogic.jms.client.JMSProducer.sendWithListener(JMSProducer.java:394)
at weblogic.jms.client.JMSProducer.send(JMSProducer.java:384)
at weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:970)
at weblogic.deployment.jms.WrappedMessageProducer.send(WrappedMessageProducer.java:188)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:626)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:597)
at org.springframework.jms.core.JmsTemplate$3.doInJms(JmsTemplate.java:562)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:484)
With little information available, followed some other suggestions to re-create the JMS data store which could have been corrupted.
In our case located here:
..fmw_home/user_projects/domains/ipp_domain/servers/myapplicationName/data/store/default - filename WLS******.DAT
A graceful shutdown to allow as many as possible messages to be resolved is suggested, then shut down application and Weblogic Server before removing the file (would suggest a backup).
After a restart, these issues disappeared, and no more errors were logged, but I'm not sure what the cause was. You would probably have to accept that any transactional data in the JMS queue will be lost.
I keep getting this error in my App, no matter what I try to fix.
"can't modify frozen array"
Any idea what could be possibly wrong here? I am running a jRuby (1.6.2) app on Torquebox (1.0.1) on my local Ubuntu (10.04) machine. Here is a complete stack trace
HTTP Status 500 -
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException: org.jruby.exceptions.RaiseException: (TypeError) can't modify frozen array
org.torquebox.rack.core.servlet.RackFilter.doRack(RackFilter.java:118)
org.torquebox.rack.core.servlet.RackFilter.doFilter(RackFilter.java:103)
org.torquebox.rack.core.servlet.RackFilter.doFilter(RackFilter.java:75)
root cause
org.jruby.exceptions.RaiseException: (TypeError) can't modify frozen array
org.jruby.RubyArray.concat(org/jruby/RubyArray.java:1450)
#<Class:0x1009c9a44>.Plugin(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/railties-3.0.7/lib/rails/plugin.rb:71)
org.jruby.RubyKernel.instance_exec(org/jruby/RubyKernel.java:2045)
rubyjit.run_7761DFB9328DE427D0E75A527B886E70C475D69F.run(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/railties-3.0.7/lib/rails/initializable.rb:25)
#<Class:0x1000dfe45>.run_initializers(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/railties-3.0.7/lib/rails/initializable.rb:50)
org.jruby.RubyArray.each(org/jruby/RubyArray.java:1602)
Rails::Initializable.run_initializers(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/railties-3.0.7/lib/rails/initializable.rb:49)
Rails::Application.initialize!(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/railties-3.0.7/lib/rails/application.rb:134)
rubyjit.method_added_79E9152BC344C327B5B6A61AD8A4DB7112D305D2.method_added(classpath:/org/torquebox/rails/core/boot.rb:78)
org.jruby.RubyKernel.send(org/jruby/RubyKernel.java:2059)
#<Class:0x101305815>.method_missing(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/railties-3.0.7/lib/rails/application.rb:77)
#<Class:0x1002cf032>.(root)(vfs:/home/Ruby_Projects/Sample-Torquebox-App/config/environment.rb:5)
org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1038)
rubyjit.require_48BC09453A4C02A82A5E47520814DA9EA3718525.require(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/polyglot-0.3.1/lib/polyglot.rb:64)
rubyjit.require_3E6CA858403268171F46C45451765CD50640768C.require(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/activesupport-3.0.7/lib/active_support/dependencies.rb:239)
rubyjit.load_dependency_04443C3CFAC2D53B173F0FF308908A72BD893CCC.load_dependency(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/activesupport-3.0.7/lib/active_support/dependencies.rb:227)
rubyjit.require_3E6CA858403268171F46C45451765CD50640768C.require(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/activesupport-3.0.7/lib/active_support/dependencies.rb:239)
#<Class:0x101f38e01>.(root)(vfs:/home/Ruby_Projects/Sample-Torquebox-App/config/environment.rb:6)
org.jruby.RubyKernel.instance_eval(org/jruby/RubyKernel.java:2028)
Rack::Builder.initialize(/home/torquebox-1.0.1/jruby/lib/ruby/gems/1.8/gems/rack-1.2.3/lib/rack/builder.rb:46)
#<Class:0x1002cf032>.(root)(vfs:/home/Ruby_Projects/Sample-Torquebox-App/config.ru:3)
note The full stack trace of the root cause is available in the JBoss Web/3.0.0-CR1 logs.
JBoss Web/3.0.0-CR1
If you get the error during startup, chances are big you're actually trying to redefine a frozen array in an initializer (could be as simple as redifining ABBR_DAYNAMES). If the application can run under MRI you could try that, since it gives you verbose warnings about this.
If there's nothing in your initializer, then it might be a gem that you included in your application. Try taking out gems one by one and see what the result is.
For some reason I keep getting a response code of 500 when I run cucumber, even though all the tests pass. The error occurs randomly for different tests, every time I run it. Sometimes all tests pass as well.
I thought it was a memory issue, so I tried restarting my computer, but that didn't do anything.
An example of the error is:
And I follow "link" # features/step_definitions/web_steps.rb:33
Failed. Response code = 500. Response message = Internal Server Error. (ActiveResource::ServerError)
./app/controllers/companies_controller.rb:23:in `show'
./features/step_definitions/web_steps.rb:35
./features/step_definitions/web_steps.rb:14:in `with_scope'
./features/step_definitions/web_steps.rb:34:in `/^(?:|I )follow "([^"]*)"(?: within "([^"]*)")?$/'
features/manage_sites.feature:256:in `And I follow "link"'
It seems related to the tagging feature. I was using the #wip tag, and when I moved it around, it would cause the sporadic 500 errors. Removing all #wip tags makes all the tests pass.
I could be wrong though. I'll need to try and replicate it consistently.
Would anybody be able to help?