I have a rails app deployed on heroku, and there is an annoying issue: my router heroku gets an url that doesn't exist anymore (I have deleted everything on my WebApp regarding the notifications).
Do you have any idea where it could come from?
Here are what my logs :
2015-06-25T12:22:30.357205+00:00 heroku[router]: at=info method=GET path="/notifications" host=www.krawd.com request_id=6b03fbec-88ee-48bd-8afd-aa6d59a9bf53 fwd="82.237.217.103" dyno=web.1 connect=1ms service=12ms status=500 bytes=377
2015-06-25T12:22:30.334224+00:00 app[web.3]: source=rack-timeout id=8b3b9046-8613-44fb-8274-c0dc976d3472 wait=16ms timeout=25000ms state=ready
2015-06-25T12:22:30.362821+00:00 app[web.3]: source=rack-timeout id=8b3b9046-8613-44fb-8274-c0dc976d3472 wait=16ms timeout=25000ms service=29ms state=active
2015-06-25T12:22:30.406680+00:00 app[web.3]: Completed 500 Internal Server Error in 25ms
2015-06-25T12:22:30.341720+00:00 app[web.1]: source=rack-timeout id=6b03fbec-88ee-48bd-8afd-aa6d59a9bf53 wait=0ms timeout=25000ms service=0ms state=active
2015-06-25T12:22:30.344956+00:00 app[web.1]: ActionController::RoutingError (No route matches [GET] "/notifications"):
Make sure you search your web app folder for "/notifications" and see if you forgot some reference somewhere. Does the request happen when you run on localhost? You could trace it more easily that way.
Related
I'm getting mysterious H18 errors on Heroku. Theses requests seems to be made by bots or crawlers sending POST requests on the root URL of my application. What concerns me is that these requests all take more than 30 seconds to process before they get terminated, most likely by a timeout. Here is a sample of my log file and Metric graph:
Dec 14 03:52:26 poll-en heroku/router sock=backend at=error code=H18 desc="Server Request Interrupted" method=POST path="/" host=app.do request_id=72252a6a-d4b5-4ecc-ae3c-bc69c273eb16 fwd="138.201.76.138" dyno=web.1 connect=1ms service=30034ms status=503 bytes=71 protocol=http
Dec 14 04:05:11 poll-en heroku/router sock=backend at=error code=H18 desc="Server Request Interrupted" method=POST path="/" host=app.do request_id=78ea5ae4-9e8a-4c90-b26e-c2eae40116b4 fwd="148.251.231.105" dyno=web.1 connect=0ms service=31392ms status=503 bytes=71 protocol=http
Dec 14 04:19:07 poll-en heroku/router sock=backend at=error code=H18 desc="Server Request Interrupted" method=POST path="/" host=app.do request_id=bcc76545-24e0-4fc7-8e63-a08bb463bb31 fwd="148.251.231.105" dyno=web.1 connect=0ms service=30195ms status=503 bytes=71 protocol=http
Dec 14 04:45:35 poll-en heroku/router sock=backend at=error code=H18 desc="Server Request Interrupted" method=POST path="/" host=app.do request_id=87efe086-9879-4506-a7ac-52504219126d fwd="144.76.141.230" dyno=web.1 connect=0ms service=31733ms status=503 bytes=71 protocol=http
My application doesn't handle POST requests on the root URL and I can't replicate the error using cURL. Just to be sure, I also added a Rack Middleware that runs first and terminates all POST requests on the root URL. So the request shouldn't even reach the Rails router but I still get these H18 errors in my log file.
How can this happen? What kind of request could hang for 30 seconds when it is not even reaching my app?
What version of Puma are you using and what does your Puma config look like?
Heroku documents that the request do reach your app when H18 happens:
An H18 signifies that the socket connected, some data was sent as part of a response by the app, but then the socket was destroyed without completing the response.
Perhaps you are experiencing this bug in Puma, where is sounds like the request can be stuck in Puma (never reaching your app): https://github.com/puma/puma/issues/2282 (solved in Puma 5.0.3 and above)
You could try setting first_data_timeout in your Puma config to something that's lower than 30 seconds. Then Puma should respond with 408 Request Timeout before Heroku gives you the H18 error.
I want to generate goaccess dashboard result from heroku logs of rails application, As I checked there is no production.rb file in heroku
Format of logs in heroku is
2020-07-16T17:18:54.972063+00:00 app[web.1]: I, [2020-07-16T17:18:54.971939 #10] INFO -- : [2712edd9-bca1-4d23-9cfd-399e6dedfbdc] Started GET "/api/v1/articles" for 18.225.35.172 at 2020-07-16 17:18:54 +0000
2020-07-16T17:18:54.973433+00:00 app[web.1]: I, [2020-07-16T17:18:54.973345 #10] INFO -- : [2712edd9-bca1-4d23-9cfd-399e6dedfbdc] Processing by Api::V1::ArticlesController#index as JSON
2020-07-16T17:18:54.973525+00:00 app[web.1]: I, [2020-07-16T17:18:54.973450 #10] INFO -- : [2712edd9-bca1-4d23-9cfd-399e6dedfbdc] Parameters: {"article"=>{}}
2020-07-16T17:18:55.082811+00:00 heroku[router]: at=info method=GET path="/api/v1/article" host=abc-1234.herokuapp.com request_id=2779d556-67fc-4602-a58b-60dbcb209833 fwd="18.225.35.172" dyno=web.1 connect=1ms service=41ms status=200 bytes=539 protocol=https
I tried installing goaccess on my heroku server , but I am not able to do that, and I am not able to get log file also.
I want response time for each request in goaccess dashboard.
Anyone has idea about this ???How I can use goaccess + rails + heroku together
Try this:
cat ./production.log | grep 'Started GET' | goaccess --log-format='I, "%^" "%r" %^ %h %^ %d %t %^' --date-format=%F --time-format=%T
Note: it will work only for line like:
2020-07-16T17:18:54.972063+00:00 app[web.1]: I, [2020-07-16T17:18:54.971939 #10] INFO -- : [2712edd9-bca1-4d23-9cfd-399e6dedfbdc] Started GET "/api/v1/articles" for 18.225.35.172 at 2020-07-16 17:18:54 +0000
other lines were ignored by grep, which were:
2020-07-16T17:18:54.973433+00:00 app[web.1]: I, [2020-07-16T17:18:54.973345 #10] INFO -- : [2712edd9-bca1-4d23-9cfd-399e6dedfbdc] Processing by Api::V1::ArticlesController#index as JSON
2020-07-16T17:18:54.973525+00:00 app[web.1]: I, [2020-07-16T17:18:54.973450 #10] INFO -- : [2712edd9-bca1-4d23-9cfd-399e6dedfbdc] Parameters: {"article"=>{}}
2020-07-16T17:18:55.082811+00:00 heroku[router]: at=info method=GET path="/api/v1/article" host=abc-1234.herokuapp.com request_id=2779d556-67fc-4602-a58b-60dbcb209833 fwd="18.225.35.172" dyno=web.1 connect=1ms service=41ms status=200 bytes=539 protocol=https
My original answer was at goaccess issue, Good luck!
I'm going through my 404 logs and I noticed this:
302 <158>1 2018-10-20T19:12:36.942085+00:00 heroku router - - at=info method=GET path="/wp-login.php" host=www.makerspro.io request_id=f5929aab-8671-433e-92d0-1e779f997d0a fwd="194.99.106.147" dyno=web.1 connect=0ms service=1923ms status=404 bytes=6042 protocol=http
307 <158>1 2018-10-20T19:12:39.076998+00:00 heroku router - - at=info method=GET path="/blog/wp-login.php" host=www.makerspro.io request_id=ce175bfd-6411-48c2-9328-1b6fccb6ae30 fwd="194.99.106.147" dyno=web.1 connect=0ms service=1629ms status=404 bytes=6042 protocol=http
305 <158>1 2018-10-20T19:12:40.699840+00:00 heroku router - - at=info method=GET path="/wp/wp-login.php" host=www.makerspro.io request_id=17bd87b1-3d8f-48bb-8748-ff9d47250dba fwd="194.99.106.147" dyno=web.1 connect=0ms service=1420ms status=404 bytes=6042 protocol=http
312 <158>1 2018-10-20T19:12:42.599811+00:00 heroku router - - at=info method=GET path="/wordpress/wp-login.php"
My website is built on rails, I'm the only developer and it seems like someone is trying to hack my website.
Of course, I don't have wordpress installed or anything like that.
What can I do about this and how can I prevent future attacks?
These are generic attacks that happen constantly, and is done by scripts.
The best course is to make sure you are not vulnerable, then ignore them. If you do not have Wordpress, then you have nothing to worry about regarding requests to /wp-login.php.
If you want to harden your application, make sure you have security measures like described in the RoR documentation (https://guides.rubyonrails.org/security.html).
My Heroku Rails is always loading extremely slowly and not only at first start.
It doesnt load this slowly in production mode.
I have New-Relic Installed, I have two Dynos running and I have PostgreSQL Crane Plan. But when I start my app it will normally take about 30 seconds to load and 60% of the time it'll go straight to Application Error, if I check my logs I get Memory Quota Exceeded. Even if I go to a static page with no Ruby in its source, it'll still load slowly. Sometimes I manage to connect to my app and it will normally take around 10 seconds to load when navigating from page to page.
I've been looking online for ages and all the most common answer I find is to just add an extra dyno, but that doesnt help.
Im also running on Unicorn
These are the logs from a movies#show page which loaded at first and then crashed
Started GET "/movies/61708" for 81.34.154.155 at 2013-08-29 23:08:23 +0000
2013-08-29T23:08:30.093034+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path=/?page=7 host=www.websitename.com fwd="81.34.154.155" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0
2013-08-29T23:08:31Z app[postgres.8916]: [BLUE] duration: 6945.946 ms statement: SELECT "movies".* FROM "movies"
2013-08-29T23:08:41+00:00 app[heroku-postgres]: source=HEROKU_POSTGRESQL_BLUE measure.current_transaction=2844 measure.db_size=60732536bytes measure.tables=23 measure.active-connections=10 measure.waiting-connections=0 measure.index-cache-hit-rate=1 measure.table-cache-hit-rate=1
2013-08-29T23:08:45.401673+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path=/movies/63724 host=www.websitename.com fwd="81.34.154.155" dyno=web.1 connect=2ms service=30001ms status=503 bytes=0
2013-08-29T23:08:45.445116+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path=/movies/63704 host=www.websitename.com fwd="81.34.154.155" dyno=web.2 connect=3ms service=30000ms status=503 bytes=0
2013-08-29T23:08:52.684401+00:00 heroku[web.2]: Process running mem=512M(100.0%)
2013-08-29T23:08:52.684613+00:00 heroku[web.2]: Error R14 (Memory quota exceeded)
2013-08-29T23:08:53.647059+00:00 app[web.1]: E, [2013-08-29T23:08:53.384868 #2] ERROR -- : worker=0 PID:5 timeout (61s > 60s), killing
2013-08-29T23:08:54.375154+00:00 app[web.1]: E, [2013-08-29T23:08:54.374971 #2] ERROR -- : reaped #<Process::Status: pid 5 SIGKILL (signal 9)> worker=0
Does anyone know a solution for this?
You mentioned in the comments you have 60,000 rows in your movies table. From your logs it appears you're doing something like the following:
Movie.all
This will generate an SQL query like what we see in your logs.
SELECT "movies".* FROM "movies"
The memory/storage capacity of Postgres doesn't matter with respect to your issue. Your database could be on a dedicated server with 96G RAM. You'll still end up with the same problem on Heroku's end.
When you do something like Movie.all, you're creating in-memory ruby class instances of Movie for all 60,000 records. This takes time and space.
You're probably also rendering too many (dare I say all?!) of these records to a single webpage at once. This is simply way too much data to render during a single request, whether you're on Heroku or not.
Heroku has a hard 30s time limit for it's request's running time. You're hitting this limit. Try a much smaller subset of data, even just as a test, and see if it solves your issues. I'm betting it will.
Movie.limit(10)
I've got a simple business Rails 3.2.13 app with a base of only one user, with very small Postgres DB tables, and very simple Active Record DB queries. I'm running it on two dynos, zero workers, on the Unicorn web server. The root page of the app typically loads in about 150 ms in development, and in under 1 s in production at Heroku.
EXCEPT, when it randomly times out. I'm seeing timeouts in the log, but NOT the typical R12 Request Timeout. The timeouts look like this:
2013-05-28T16:04:28.004290+00:00 app[web.2]: Started GET "/" for 72.28.209.197 at 2013-05-28 16:04:28 +0000
2013-05-28T16:04:44.225643+00:00 app[web.2]: E, [2013-05-28T16:04:44.225444 #2] ERROR -- : worker=2 PID:87 timeout (16s > 15s), killing
2013-05-28T16:04:44.238422+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=GET path=/ host=[REDACTED].herokuapp.com fwd="72.28.209.197" dyno=web.2 connect=7ms service=16240ms status=503 bytes=0
2013-05-28T16:04:44.255813+00:00 app[web.2]: E, [2013-05-28T16:04:44.255554 #2] ERROR -- : reaped #<Process::Status: pid 87 SIGKILL (signal 9)> worker=2
2013-05-28T16:04:44.298023+00:00 app[web.2]: I, [2013-05-28T16:04:44.297643 #103] INFO -- : worker=2 ready
When that request works, it looks like this:
2013-05-28T16:03:36.965507+00:00 app[web.1]: sequenceId="102642"] Started GET "/" for 72.28.209.197 at 2013-05-28 16:03:36 +0000
2013-05-28T16:03:37.388261+00:00 app[web.1]: sequenceId="102690"] Processing by OverviewController#index as HTML
2013-05-28T16:03:38.238877+00:00 app[web.1]: sequenceId="102707"] Rendered overview/index.html.erb within layouts/application (315.8ms)
2013-05-28T16:03:38.311974+00:00 app[web.1]: sequenceId="102710"] Completed 200 OK in 924ms (Views: 432.4ms | ActiveRecord: 92.1ms)
2013-05-28T16:03:38.322786+00:00 heroku[router]: at=info method=GET path=/ host=[REDACTED].herokuapp.com fwd="72.28.209.197" dyno=web.1 connect=2ms service=1375ms status=200 bytes=136795
2013-05-28T16:03:42.625787+00:00 app[web.1]: sequenceId="102940"] Started GET "/assets/favicon.ico" for 72.28.209.197 at 2013-05-28 16:03:42 +0000
2013-05-28T16:03:58.753408+00:00 app[web.1]: sequenceId="104359"] E, [2013-05-28T16:03:58.753173 #2] ERROR -- : worker=2 PID:66 timeout (16s > 15s), killing
2013-05-28T16:03:58.801152+00:00 app[web.1]: sequenceId="104362"] E, [2013-05-28T16:03:58.800993 #2] ERROR -- : reaped #<Process::Status: pid 66 SIGKILL (signal 9)> worker=2
2013-05-28T16:03:58.798722+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=GET path=/assets/favicon.ico host=[REDACTED].herokuapp.com fwd="72.28.209.197" dyno=web.1 connect=3ms service=16175ms status=503 bytes=0
2013-05-28T16:03:59.127150+00:00 app[web.1]: sequenceId="104397"] I, [2013-05-28T16:03:59.126795 #84] INFO -- : worker=2 ready
Things to note: The request response time was 924 ms. But then some kind of strange error happened after that.
This is what that request looks like in terms of DB queries. There is nothing exciting happening there, and those DB tables only contain a few hundred rows of data. There is no reason why the response should take 1 second sometimes, and 15+ seconds on other times. Here's the code for the extremely simple controller action:
class OverviewController < ApplicationController
def index
#current_event = Event.recent
#areas = Area.order(:position)
#reservations = Reservation.find(:all,
:conditions => { :event_id => #current_event.id },
:order => 'created_at DESC',
:limit => 5)
end
end
Here's the Event#recent method, just so that you can see that it's not doing anything that could potentially take 15+ seconds:
def self.recent
Event.last || Event.create
end
I'm at a loss for how to get this app stabilized, other than moving it off of Heroku. I don't know of any way to troubleshoot the problem, and I guess Heroku won't help. Am I simply stuck? Do I have any options for troubleshooting the problem other than moving the app to Engine Yard?
Nothing here helped, and this site is Heroku's technical support. Paying $35/mo doesn't buy me any support, so I'm stuck and there is no way forward with Heroku.
So my answer to this problem is to move the app to Engine Yard. I don't see that I have any other options. Heroku is awesome when it works and it's a complete mystery with no technical support when it doesn't work.