Using: ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]
I have a ruby batch job that the last 2 nights in a row has hung up at about the same time.
The weird thing is when I do a kill -QUIT on the process it frees it up and continues processing.
Here is the stack when I send the SIGQUIT:
Wed Mar 23 2011 11:07:55 SignalException: SIGQUIT: SELECT * FROM `influencers` WHERE (`influencers`.`external_id` = 199884972) LIMIT 1
Wed Mar 23 2011 11:07:55 /usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record /connection_adapters/abstract_adapter.rb:219:in `log'/usr/lib/ruby/gems/1.8/gems/activerecord- 2.3.5/lib/active_record/connection_adapters/mysql_adapter.rb:323:in `execute'/usr/lib/ruby/gems/1.8 /gems/activerecord-2.3.5/lib/active_record/connection_adapters/mysql_adapter.rb:608:in `select'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all_without_query_cache'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/abstract/query_cache.rb:62:in `select_all'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:661:in `find_by_sql'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:1548:in `find_every'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:1505:in `find_initial'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:613:in `find'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:1900:in `find_by_twitter_id'/u/apps/myapp/releases/20110323011051/app/models/influencer.rb:148:in `add_follower'/u/apps/myapp/releases/20110323011051/app/models/influencer.rb:93:in `sync_follower_list'/u/apps/myapp/releases/20110323011051/app/models/influencer.rb:91:in `each'/u/apps/myapp/releases/20110323011051/app/models/influencer.rb:91:in `sync_follower_list'/u/apps/myapp/releases/20110323011051/lib/twitter_helper.rb:379:in `retrieve_followers_of_competitors'/u/apps/myapp/releases/20110323011051/lib/twitter_helper.rb:372:in `each'/u/apps/myapp/releases/20110323011051/lib/twitter_helper.rb:372:in `retrieve_followers_of_competitors'/u/apps/myapp/releases/20110323011051/vendor/gems/will_paginate-2.3.11/lib/will_paginate/finder.rb:168:in `method_missing'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_collection.rb:369:in `method_missing_without_paginate'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_proxy.rb:215:in `method_missing'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_proxy.rb:215:in `each'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_proxy.rb:215:in `send'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_proxy.rb:215:in `method_missing'/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_collection.rb:369:in `method_missing_without_paginate'/u/apps/myapp/releases/20110323011051/vendor/gems/will_paginate-2.3.11/lib/will_paginate/finder.rb:168:in `method_missing'/u/apps/myapp/releases/20110323011051/lib/twitter_helper.rb:370:in `retrieve_followers_of_competitors'/u/apps/myapp/releases/20110323011051/lib/twitter_helper.rb:42:in `retrieve_twitter_data'/u/apps/myapp/releases/20110323011051/lib/tasks/fetch_data.rake:19:in `fetch_data'/u/apps/myapp/releases/20110323011051/lib/tasks/fetch_data.rake:11:in `each'/u/apps/myapp/releases/20110323011051/lib/tasks/fetch_data.rake:11:in `fetch_data'/u/apps/myapp/releases/20110323011051/lib/tasks/fetch_data.rake:128/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `each'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `execute'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'/usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `each'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in `run'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run'/usr/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31/usr/bin/rake:19:in `load'/usr/bin/rake:19
I suspect that from looking at the code im getting some kind of deadlock on the rails logger. Any suggestions on how to troubleshoot. Maybe something to do with rails logger folding. Not sure why this would start in last couple days....
I'm not sure if you use logrotate on your logs, but it could be that the logs get rotated and Rails.logger can't write anything any more. Doing a kill -QUIT would re-open the log file and continue processing.
Related
First of all I am sorry for my newbie question, but I am kinda stuck here.
Could anyone please tell me step by step how to use Redis in my Rails 6 application, if I have Windows OS?
I have installed Redis, currently it is in my C:\Program Files,
with these files inside it
I started the redis-server.exe, when it starts running it says:
[8020] 20 Nov 17:26:06 # Warning: no config file specified, using the default config.
In order to specify a config file use 'redis-server /path/to/redis.conf'
[8020] 20 Nov 17:26:06 * Server started, Redis version 2.4.6
[8020] 20 Nov 17:26:06 # Open data file dump.rdb: No such file or directory
[8020] 20 Nov 17:26:06 * The server is now ready to accept connections on port 6
379
[8020] 20 Nov 17:26:07 - 0 clients connected (0 slaves), 672768 bytes in use
[8020] 20 Nov 17:26:12 - 0 clients connected (0 slaves), 672768 bytes in use
[8020] 20 Nov 17:26:17 - 0 clients connected (0 slaves), 672768 bytes in use
.......... (it just keep outputs this same text every 5 seconds)
Also in my Rails application, I configured some things. I changed config/cable.yml file to:
development:
adapter: redis
url: redis://localhost:6379/1
test:
adapter: test
production:
adapter: redis
url: <%= ENV.fetch("REDIS_URL") { "redis://localhost:6379/1" } %>
channel_prefix: actioncable_test_production
My concept was to create a channel, named 'room', and output a message in the console, to see its successfully connected. But my problem is that it doesn't output anything.
I set app/javascript/channels/room_channel.js to:
import consumer from "./consumer"
consumer.subscriptions.create("RoomChannel", {
connected() {
// Called when the subscription is ready for use on the server
console.log("Connected succesfully!")
},
disconnected() {
// Called when the subscription has been terminated by the server
},
received(data) {
// Called when there's incoming data on the websocket for this channel
}
});
and my app/channels/room_channel.rb file to:
class RoomChannel < ApplicationCable::Channel
def subscribed
stream_from "room_channel"
end
def unsubscribed
# Any cleanup needed when channel is unsubscribed
end
end
Also, on my command prompt shows this: (after the usual rendering messages and things like that):
Started GET "/cable" for 127.0.0.1 at 2021-11-20 18:24:05 +0100
Started GET "/cable/" [WebSocket] for 127.0.0.1 at 2021-11-20 18:24:06 +0100
Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: keep-a
live, Upgrade, HTTP_UPGRADE: websocket)
When I go to the browser console, it says Firefox cannot create connection with ws://localhost:3000/cable
Any idea what's the problem here?
Thanks for the answers and best regards, Rererbit!
You are using a very old and deprecated version of Redis, an abandoned project by Microsoft called OpenTech Redis.
I suggest you checkout Memurai.
Memurai is an up-to-date native Windows port of Redis that derives from that project (see this commit).
Memurai is also available with a free Developer Edition.
Disclaimer: I work in Memurai.
We recently upgraded our database from 2.0.1 to 2.1.2 (Enterprise) using the explicit upgrade procedure.
When trying to take a backup post-upgrade, full backups succeed, but incremental backups fail.
When running this command the first time, it succeeds:
~/neo4j-enterprise-2.1.2/bin/neo4j-backup -from single://127.0.0.1 -to /mnt/backups/neo4j-test-backup
Running it a second time gives the following error:
Performing backup from '127.0.0.1'
00:18:44.907 [main] INFO o.n.k.InternalAbstractGraphDatabase - No locking implementation specified, defaulting to 'forseti'
Transactions applied
Exception in thread "main" org.neo4j.consistency.ConsistencyCheckingError: Inconsistencies in transaction:
Start[3,xid=GlobalId[NEOKERNL|2772027681176372421|40044|-1], BranchId[ 52 49 52 49 52 49 ],master=-1,me=-1,time=2014-06-23 23:56:53.637+0000/1403567813637,lastCommittedTxWhenTransactionStarted=752027]
1PC[3, txId=752028, 2014-06-23 23:56:53.647+0000/1403567813647]
ConsistencySummaryStatistics{
Number of errors: 2
Number of warnings: 0
Number of inconsistent RELATIONSHIP records: 2
}
at org.neo4j.consistency.checking.incremental.intercept.CheckingTransactionInterceptor.complete(CheckingTransactionInterceptor.java:181)
at org.neo4j.kernel.impl.transaction.xaframework.LogEntryVisitorAdapter.apply(LogEntryVisitorAdapter.java:62)
at org.neo4j.kernel.impl.transaction.xaframework.LogEntryVisitorAdapter.apply(LogEntryVisitorAdapter.java:28)
at org.neo4j.kernel.impl.nioneo.xa.command.LogFilter.endLog(LogFilter.java:87)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.applyTransaction(XaLogicalLog.java:1120)
at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.applyCommittedTransaction(XaResourceManager.java:856)
at org.neo4j.kernel.impl.transaction.xaframework.XaDataSource.applyCommittedTransaction(XaDataSource.java:246)
at org.neo4j.com.ServerUtil.applyReceivedTransactions(ServerUtil.java:461)
at org.neo4j.backup.BackupService.unpackResponse(BackupService.java:401)
at org.neo4j.backup.BackupService.incrementalWithContext(BackupService.java:315)
at org.neo4j.backup.BackupService.doIncrementalBackup(BackupService.java:257)
at org.neo4j.backup.BackupService.doIncrementalBackup(BackupService.java:210)
at org.neo4j.backup.BackupService.doIncrementalBackupOrFallbackToFull(BackupService.java:231)
at org.neo4j.backup.BackupTool.doBackup(BackupTool.java:240)
at org.neo4j.backup.BackupTool.run(BackupTool.java:168)
at org.neo4j.backup.BackupTool.main(BackupTool.java:71)
Any help/workarounds are appreciated.
Update: The same behavior persists after upgrading to 2.1.3
Could you please check again in the issue is resolved with 2.1.4? I darkly remember a resolved issue regarding incremental backups.
How can I print for each file how long is it taking.
I am thinking about monkey patching load or some load method in Rails, but I don't know a lot about Rails 2 inner things.
I would expect this kind of output:
config/environment.rb ---> 60 s
config/development.rb ---> 30 s
config/initializers/some.rb ---> 20 s
Any starting point is really appreciated.
Update
I am measuring time with:
from = Time.now; p from.to_s + '----- env.rb'
to = Time.now - from; p "env.rb ---" + to.to_s
Changing env.rb to boot.rb in boot.rb file, these are my results:
"Thu Feb 13 18:54:17 +0000 2014----- boot.rb"
"boot.rb ---0.001493"
"boot.rb ---0.941835"
Loading development environment (Rails 2.3.18)
"Thu Feb 13 18:54:19 +0000 2014----- env.rb"
"env.rb ---0.001378"
"Thu Feb 13 18:54:19 +0000 2014----- boot.rb"
"boot.rb ---0.001712"
"boot.rb ---0.02536"
"env.rb ---67.272745"
"env.rb ---70.847089"
I am not sure why I am getting that config/boot is being called twice. From what I see most of the time is spent in environment.rb, but I removed code and time remains the same, so it should be related with other thing.
This other thing is not bundling gems, since I have:
from = Time.now; p from.to_s + '----- env.rb'
# Be sure to restart your web server when you modify this file.
#
#
require 'rubygems'
require "bundler"
Bundler.setup
to = Time.now - from; p "env.rb ---" + to.to_s
And this is printing:
"env.rb ---0.001378"
Turned out that most of the time was spent in loading files, I started a new Rails 2.3 project with part of the models of the bigger app, and time reduced from more than a minute to about 5 seconds.
I have a rake task and when I run it in console, it is killed. This rake task operates with a table of cca 40.000 rows, I guess that may be a problem with Out of memory.
Also, I believe that this query used is optimized for dealing with long tables:
MyModel.where(:processed => false).pluck(:attribute_for_analysis).find_each(:batch_size => 100) do |a|
# deal with 40000 rows and only attribute `attribute_for_analysis`.
end
This task will not be run in the future on regular basis, so I want to avoid some job monitoring solutions like God etc...but considering background jobs e.g.Rescue job.
I work with Ubuntu, ruby 2.0 and rails 3.2.14
> My free memory is as follows:
Mem: total used free shared buffers cached
3891076 1901532 1989544 0 1240 368128
-/+ buffers/cache: 1532164 2358912
Swap: 4035580 507108 3528472
QUESTIONS:
How to investigate why rake task is always killed (answered)
How to make this rake task running ( not answered - still is killed )
What is the difference between total-vm, aton-rs, file-rss (not answered)
UPDATE 1
-Can someone explain the difference between?:
total-vm
anon-rss
file-rss
$ grep "Killed process" /var/log/syslog
Dec 25 13:31:14 Lenovo-G580 kernel: [15692.810010] Killed process 10017 (ruby) total-vm:5605064kB, anon-rss:3126296kB, file-rss:988kB
Dec 25 13:56:44 Lenovo-G580 kernel: [17221.484357] Killed process 10308 (ruby) total-vm:5832176kB, anon-rss:3190528kB, file-rss:1092kB
Dec 25 13:56:44 Lenovo-G580 kernel: [17221.498432] Killed process 10334 (ruby-timer-thr) total-vm:5832176kB, anon-rss:3190536kB, file-rss:1092kB
Dec 25 15:03:50 Lenovo-G580 kernel: [21243.138675] Killed process 11586 (ruby) total-vm:5547856kB, anon-rss:3085052kB, file-rss:1008kB
UPDATE 2
modified query like this and rake task is still killed.
MyModel.where(:processed => false).find_in_batches do |group|
p system("free -k")
group.each do |row| # process
end
end
I compiled CouchDB and installed. It seems to work great except when I use views on the database, then it just spins the wheel and nothing happens and the cpu load spikes to 100% and slowly it eats away all memory and starts to swap a lot which in return increases the cpu load.
I have tried both with the js-1.70-12 that comes with centos 6.4, as well as build and install my own js-1.85-1. All erlang packages are installed from epel :
erlang-crypto-R14B-04.2.el6.x86_64
erlang-syntax_tools-R14B-04.2.el6.x86_64
erlang-mnesia-R14B-04.2.el6.x86_64
erlang-ssl-R14B-04.2.el6.x86_64
erlang-cosProperty-R14B-04.2.el6.x86_64
erlang-asn1-R14B-04.2.el6.x86_64
erlang-cosEventDomain-R14B-04.2.el6.x86_64
erlang-eunit-R14B-04.2.el6.x86_64
erlang-erl_docgen-R14B-04.2.el6.x86_64
erlang-toolbar-R14B-04.2.el6.x86_64
erlang-debugger-R14B-04.2.el6.x86_64
erlang-tools-R14B-04.2.el6.x86_64
erlang-typer-R14B-04.2.el6.x86_64
erlang-megaco-R14B-04.2.el6.x86_64
erlang-oauth-1.1.1-1.el6.x86_64
erlang-stdlib-R14B-04.2.el6.x86_64
erlang-hipe-R14B-04.2.el6.x86_64
erlang-kernel-R14B-04.2.el6.x86_64
erlang-runtime_tools-R14B-04.2.el6.x86_64
erlang-snmp-R14B-04.2.el6.x86_64
erlang-public_key-R14B-04.2.el6.x86_64
erlang-inets-R14B-04.2.el6.x86_64
erlang-ibrowse-2.2.0-4.el6.x86_64
erlang-cosEvent-R14B-04.2.el6.x86_64
erlang-cosNotification-R14B-04.2.el6.x86_64
erlang-edoc-R14B-04.2.el6.x86_64
erlang-otp_mibs-R14B-04.2.el6.x86_64
erlang-cosFileTransfer-R14B-04.2.el6.x86_64
erlang-cosTransactions-R14B-04.2.el6.x86_64
erlang-inviso-R14B-04.2.el6.x86_64
erlang-jinterface-R14B-04.2.el6.x86_64
erlang-erl_interface-R14B-04.2.el6.x86_64
erlang-diameter-R14B-04.2.el6.x86_64
erlang-gs-R14B-04.2.el6.x86_64
erlang-tv-R14B-04.2.el6.x86_64
erlang-appmon-R14B-04.2.el6.x86_64
erlang-odbc-R14B-04.2.el6.x86_64
erlang-wx-R14B-04.2.el6.x86_64
erlang-et-R14B-04.2.el6.x86_64
erlang-observer-R14B-04.2.el6.x86_64
erlang-sasl-R14B-04.2.el6.x86_64
erlang-dialyzer-R14B-04.2.el6.x86_64
erlang-common_test-R14B-04.2.el6.x86_64
erlang-os_mon-R14B-04.2.el6.x86_64
erlang-examples-R14B-04.2.el6.x86_64
erlang-compiler-R14B-04.2.el6.x86_64
erlang-erts-R14B-04.2.el6.x86_64
erlang-xmerl-R14B-04.2.el6.x86_64
erlang-orber-R14B-04.2.el6.x86_64
erlang-cosTime-R14B-04.2.el6.x86_64
erlang-ssh-R14B-04.2.el6.x86_64
erlang-docbuilder-R14B-04.2.el6.x86_64
erlang-percept-R14B-04.2.el6.x86_64
erlang-parsetools-R14B-04.2.el6.x86_64
erlang-ic-R14B-04.2.el6.x86_64
erlang-pman-R14B-04.2.el6.x86_64
erlang-webtool-R14B-04.2.el6.x86_64
erlang-test_server-R14B-04.2.el6.x86_64
erlang-reltool-R14B-04.2.el6.x86_64
erlang-R14B-04.2.el6.x86_64
erlang-mochiweb-1.4.1-5.el6.x86_64
Every thing configures and makes and installs as expected. You can dump data into the database, you can create documents and all that. But I can not run any view, temporary or not.
The only error I see in the logs is like this one, and it is a lot of these errors :
[Sun, 18 Aug 2013 23:10:38 GMT] [error] [<0.124.0>] {error_report,<0.30.0>,
{<0.124.0>,crash_report,
[[{initial_call,
{mochiweb_socket_server,init,['Argument__1']}},
{pid,<0.124.0>},
{registered_name,[]},
{error_info,
{exit,eaddrinuse,
[{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_secondary_services,couch_server_sup,
<0.31.0>]},
{messages,[]},
{links,[<0.93.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,987},
{stack_size,24},
{reductions,459}],
[]]}}
But I have no idea what they mean.
Do I need to compile and install erlang as well ? All the above packages or just erlang ?
Your compilation and installation looks fine. At least your error (note: eaddrinuse in traceback) is about that there is some process that listens same address and port as your CouchDB try to. Check other listening processes with netstat -anp command or change CouchDB's listen port to different.