Why my service freezes when over 100 users access to it? - ruby-on-rails

Question

I'm running the service that I developed by myself.
Ruby on Rails.3.2.11, Passenger, and Apache2 are being used.
It seemed working fine until there are over 100 registered users accessing to the service at the same time.
When it happens, my service completely freezes and there won't be any response(Just keep loading forever)
So, all I can do is restarting Apache. It solves the problem for a moment but it occurs again and again!
I thought that handling about 100 users won't be that big problem in Ruby on Rails App.
But I'm guessing that my unique feature is preventing that.
There are 2 things that I care about.
All the registered user's last_active_at(datetime) will be updated when every load
(Every page, and Every time)
All the registered user's point will be increased by 100 when it's his first access in a day(If user access to the service, he can earn 100 points. But only once in a day)
This will be checked in every page, too. Just like last_active_at
The codes for that is just like this
application_controller.rb
class ApplicationController < ActionController::Base
before_filter :record_user_activity
def record_user_activity
if current_user
#Retrieving current_user
#myself_user = User.includes(:profile).find(current_user)
#Checking if current_user hasn't received bonus for today yet
if #myself_user.point_added_at.nil? || !#myself_user.point_added_at.today?
#Checking if current_user shows his online status to public(If so he can earn 100 points)
if #myself_user.profile.activity_invisible.blank?
plus_point(#myself_user, 100)
flash[:alert] = '100 points for today's bonus is added!'
#myself_user.touch :point_added_at
#myself_user.save
end
end
#Updating last_active_at(datetime)
if #myself_user.profile.activity_invisible.blank?
#myself_user.touch :last_active_at
#myself_user.save
else
#myself_user.touch :updated_at
#myself_user.save
end
end
end
end
And this is the result of performance monitoring.
Please, tell me what would be the bottle neck problem, and how to solve it!
Thanks!
UPDATE:
my.cnf
# The following options will be passed to all MySQL clients
[client]
#password = your_password
port = 3306
socket = /var/lib/mysql/mysql.sock
# Here follows entries for some specific programs
# The MySQL server
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 1M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
character_set-server=utf8
innodb_buffer_pool_size=384M
innodb_log_file_size=128M
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
#skip-networking
# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin
# binary logging format - mixed recommended
binlog_format=mixed
# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1
# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
# MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
# where you replace <host>, <user>, <password> by quoted strings and
# <port> by the master's port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
# MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables' values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host = <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user = <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password = <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port = <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin
# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /var/lib/mysql
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /var/lib/mysql
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 768M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 100M
#innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 2
#innodb_lock_wait_timeout = 50
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
default_character_set=utf8
[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
UPDATE2:
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-external-locking
key_buffer_size = 256M
join_buffer_size = 1M
thread_cache = 8
thread_concurrency = 8
thread_cache_size = 60
query_cache_size = 32M
max_connections = 200
max_allowed_packet = 1M
table_open_cache = 256
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 1M
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
character_set-server=utf8
innodb_buffer_pool_size=384M
innodb_log_file_size=128M

Answer 1

Passenger defaults to a max of 6 concurrent processess. 6 does not sound like a lot, but in general, even with 100 users at the same time, you will not need 100 processes at the same time.
http://www.modrails.com/documentation/Users%20guide%20Apache.html#PassengerMaxPoolSize
You can increase this to 12 in passenger.
Note that each process in passenger will take up ram. Significantly more ram.
Here are 2 alternatives:
1) Move to a threaded web server, Puma. The default concurrency with Puma is 25.
2) Move the processing offline
* use Sidekiq or Resque to store the record-activity offline
Or, do all of them.
I would create a test environment, and use blitz.io to test your setup and find when your system will show slowdowns, and then stoppages.

Answer 2

Posting this as an Answer since unable to comment on original question.
The behavior you describe is consistent with thread or database connection management issues. Could you tell us this size of your database connection pool (e.g., 100?)? Is it possible your application is not releasing their database connections? If all the db connections in the pool are used up and not released, it would result in similar behavior you are describing.