I never wrote any complex regular expression before, and what I need seems to be (at least) a bit complicated.
I need a Regex to find matches for the following:
Here below show the logs for this i need regexp plesase help Thanking you in advance
Started GET \"/\" for 1x2.x6.1xx.2x at 2016-10-20 11:04:00 +0200
Processing by WelcomeController#index as HTML
Current user: anonymous
Redirected to http://example.pro.local/login?back_url=http%xx%xx%2Fexample.pro.local%2F
Filter chain halted as :check_if_login_required rendered or redirected
Completed 302 Found in 3.4ms (ActiveRecord: 1.9ms)"
Extracting information from unstructured logs with regex is tedious and brittle.
Instead it is preferable to make the application output logs in a structured format (as suggested by #ndn).
Consider using lograge and/or logstasher in your Rails application to output structured logs.
Related
I'm a newbie in Rails, and I had a problem with my Rails project. I used Log4r for my logger. I also used config for Log4r at How to configure Log4r with Rails 3.0.x? . But when I send some request at the same time, the output of Log4r has error. It is unordered. :(
Example about error output log file:
Started GET "/task_results.json" for 172.29.73.182 at 2013-06-17 17:36:38 +0700
Processing by TaskResultsController#index as JSON
Started GET "/favicon.ico" for 172.29.73.182 at 2013-06-17 17:36:38 +0700
Processing by ApplicationController#missing_page as
Parameters: {"path"=>"favicon"}
Completed 404 Not Found in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms)
[1m [36m (994.0ms) [0m [1mSELECT task_result_id,task_id,worker_id,product_id,passed,date_time,details FROM task_results ORDER BY task_result_id DESC; [0m
Completed 200 OK in 8656ms (Views: 0.2ms | ActiveRecord: 994.0ms)
And I wanna ask that how can I config Log4r to synchronize output? Or how can I fix my problem?
ruby is single-threaded, so I assume you are running your rails app on a server like unicorn?
The only solution I can think of is timestamps (with micro seconds) and not using the multi-line Rails log output which is not suitable for anything but single requests in dev environments. But even this does not guarantee "ordered" logs because they can be written when the request was served or at a random time afterwards when the buffer was flushed again.
This is an in-depth tutorial on how to use Log4r that also shows how to get custom timestamps into your logs and have all the important information on a single line. This prevents logs from other requests writing inside your main request and you can have a timestamp resolution of microseconds and order afterwards manually. However the order should not matter if you are running concurrent app-servers because the order is not guaranteed there anyway.
I have a Ruby on Rails app that works fine locally with a sqlite3 database and can save and retrieve records without issue.
When deployed to Heroku at http://moviedata.herokuapp.com/ using a postgresql database, records are not saving even though it looks like the logs say they are. Records read from the db fine and data is displayed as expected.
The tailed logs for adding a record are:
2012-08-21T19:51:31+00:00 app[web.1]:
2012-08-21T19:51:31+00:00 app[web.1]:
2012-08-21T19:51:31+00:00 app[web.1]: Started POST "/" for 50.53.6.156 at 2012-08-21 19:51:31 +0000
2012-08-21T19:51:31+00:00 app[web.1]: Parameters: {"utf8"=>"✓", "authenticity_token"=>"+BYQLzhrfDkUVW8UaHikHpmtGHxpeQ/yF4VByHh9m1I=", "movie"=>{"title"=>"The Running Man", "description"=>"A documentary about a public execution game show.", "year"=>"1987", "genre"=>"science fiction"}, "commit"=>"Create Movie"}
2012-08-21T19:51:31+00:00 app[web.1]: Processing by MoviesController#index as HTML
2012-08-21T19:51:31+00:00 app[web.1]: Rendered movies/index.html.erb within layouts/application (5.1ms)
2012-08-21T19:51:31+00:00 app[web.1]: Completed 200 OK in 9ms (Views: 6.7ms | ActiveRecord: 0.9ms)
2012-08-21T19:51:31+00:00 heroku[router]: POST moviedata.herokuapp.com/ dyno=web.1 queue=0 wait=0ms service=17ms status=200 bytes=3479
The 'heroku pg' command shows the same number of rows (11) on the postgres database after a record is added.
This is a simple app I built to learn Rails and the Heroku platform. To reproduce this, just visit http://moviedata.herokuapp.com/ and click "New Movie", enter some junk data in the form, and hit "create movie". The record should be saved and show up in the list on the front page, but it doesn't.
Is there perhaps something I have to turn on, configure, or activate in order to be able to write to the postgres database? Seems very strange to me that it could be read from but not written to. Any better way to troubleshoot than the logs?
Locally I'm using Ruby 1.9.3, Rails, 3.2.8, PostgreSQL 9.1.5, SQLite 3.7.9, Heroku Toolbelt 2.30.3.
Edit/Update: I switched the local version to use psql. It also experiences the same problem where records are not saved. With the user set to log_statement='all' The log in at /var/log/postgresql/posgresql-9.1.main.log shows lots of selects, but when the record add is attempted, the log shows the database never being hit.
Foreman shows the data being posted, like so:
22:38:03 web.1 | Started POST "/" for 127.0.0.1 at 2012-08-21 22:38:02 -0700
22:38:03 web.1 | Processing by MoviesController#index as HTML
22:38:03 web.1 | Parameters: {"utf8"=>"✓", "authenticity_token"=>"0AyxRbwl/Kgi05uI1KX8uxVUJjx9ylAA1ltdWgmunm4=", "movie"=>{"title"=>"Army of Darkness", "description"=>"A man fights the living dead using a boomstick.", "year"=>"1997", "genre"=>"horror"}, "commit"=>"Create Movie"}
22:38:03 web.1 | Movie Load (0.8ms) SELECT "movies".* FROM "movies" ORDER BY title
22:38:03 web.1 | Rendered movies/index.html.erb within layouts/application (14.9ms)
A failed commit does sound like a great explanation. I'm not yet sure how to check whether the driver is set to commit or to see how/when a commit might have failed.
This is a very simple application, with no load balancing or complex configuration and most of the code was generated by the 'generate scaffold' command, but it's entirely possible that there's some constraint that's being violated somewhere before the db is ever hit. Perhaps there's a way to crank the Foreman (or Rails) log level up to 11? I also tried using thin instead and scoured the log files in the log/ folder and didn't find anything other than what's logged above.
This sounds a lot like a transaction issue, where you aren't COMMITting your transactions after you do work, so the changes are lost. If your SQLite driver defaults to COMMITting transactions that're closed without an explicit COMMIT or rollback, and your Pg driver defaults to ROLLBACK, you'd get the behaviour described. The same will happen if the SQLite defaults to autocomitting each statement by default, and the Pg driver driver defaults to opening a transaction.
This is one of the many good reasons to use the same local database for testing as you're going to deploy to when you want to go live.
If you were on a normal Pg instance I'd tell you to enable log_statement = 'all' in postgresql.conf, reload Pg, and watch the logs. You can't do that on Heroku, but you do have access to the Pg logs with heroku logs --ps postgres. Try running ALTER USER my_heroku_user SET log_statement = 'all';, re-testing, and examining the logs.
Alternately, install Pg locally.
Other less likely possibilities that come to mind:
You're using long-running SERIALIZABLE transactions for reads, so their snapshot never gets updated. Pretty unlikely.
Permissions on database objects are causing INSERTs, UPDATEs, etc to fail, and your app is ignoring the resulting errors. Again, unlikely.
You have DO INSTEAD rules that don't do what you expect, or BEFORE triggers that return NULL, thus silently turning operations into no-ops. Seems unlikely if you're testing with SQLite.
You're writing to a different DB than you're reading from. Not impossible in setups that're attempting to read from a cluster of hot standbys, etc.
I have two rails applications (both now on Rails 3.1.1), and they work nicely. However, I have a dependence between the two. Application A uses data of application B by linking to it. These links are created automatically, but they have to be computed by doing a lookup to the data of application B. I'm working on Windows 7 with Ruby 1.9.2 and Thin as web server, and this will not be changed :-(
I have tried the following:
Use just a RESTful resource, so defined a controller, called its action (get_xml_obj with some params in it), read the needed values from the XML. Worked, but needs around 0.5s to 1s per call.
Replaced it by ActiveResource#find which worked as well, but with the same performance as the solution before.
I have installed nginx and configured it so, that the connection are keepalive, so that the connection handling should be much faster. But noticed no difference at all when calling B from A.
When I compare the time spent, these are typical examples (here with 4 references in one web page):
Application A:
Started GET "/tasks/search_task/1803" for 127.0.0.1 at 2011-11-02 14:11:04 +0100
Processing by TasksController#search_task as HTML
Parameters: {"id"=>"1803"}
Rendered tasks/_tooltip.html.haml (4529.5ms)
Completed 200 OK in 4532ms (Views: 4527.5ms | ActiveRecord: 2.0ms)
cache: [GET /tasks/search_task/1865] miss
Application B:
cache: [GET /service/get_xml_obj?key=notice&value=rails] miss
Started GET "/service/get_xml_obj?key=notice&value=rails" for 127.0.0.1 at 2011-
11-02 14:11:05 +0100
Processing by ServiceController#get_xml_obj as */*
Parameters: {"key"=>"notice", "value"=>"rails"}
Completed 200 OK in 6ms (Views: 3.0ms | ActiveRecord: 1.0ms)
and 3 other calls with a similar length (< 10ms).
So is there something I can do to tune the retrieval (without accessing the database directly)? Do you know of any good documentation how to measure and tune the web server and middleware? These are only personal applications, so there is no way of deploying them on a decent server. I use a cache for the retrieved information, so it gets better over time, but 1 second is too much to wait for. And there may be more than 1 or 2 links in a page I want to render.
Ok, I finally gave up and implemented the following:
Added file b.rb to my models directory in application A.
Included there all raw models, where the base models (used sti) are defined like that:
class Notice < ActiveRecord::Base
self.establish_connection(
:adapter => "sqlite3",
:database => "../b/db/dev.db"
)
end
...
I am now able to ask: Notice.where(:key => 'rails') which results in a real Rails model object.
The whole thing was implemented in around 20 minutes, and now there is no difference in including no link from application A to B to include 5 links.
At some point in time, I would like to know what is the slow part in using RESTful resources here ...
I have a problem with integration between CouchRest and Sunspot. When I search the book detail, the result from Sunspot is empty. I try to google it for a long time but no help.
Started GET "/books/search?utf8=%E2%9C%93&query=Book of Life&commit=Search%21" for 127.0.0.1 at 2011-09-08 11:27:41 +0700
Processing by BooksController#search as HTML
Parameters: {"utf8"=>"?", "query"=>"Book of Life", "commit"=>"Search!"}
Rendered books/index.html.erb within layouts/application (10.7ms)
Completed 200 OK in 145ms (Views: 20.6ms | ActiveRecord: 0.0ms)
[] <-- I got empty result
My System
Ruby 1.9.2p290
Rails 3.0.10
CouchDB 1.1.0
File structure ( https://gist.github.com/1164637/ )
Model (/app/models/book_detail.rb)
Controller (/app/controllers/books_controller.rb)
Sunspot Adapter for CouchRest (/config/initializers/couchdb.rb)
Sunspot Adapter Module (/config/initializers/sunspot_couch.rb)
NOTE: Sorry about code link. I always got "Please indent all code by 4 spaces using the code toolbar button". I try to remove all tab and follow SO code formatting guideline but it not work anymore.
Forgive me if I'm missing something, but I can't see how Sunspot is mapping "keywords" to the searchable fields on your CouchRest objects.
To debug first I'd visit Couch in the browser admin UI to make sure my that end is working. Then I'd double check that sunspot is getting anything. If sunspot contains your records then the bug is on the search side, if it is empty that maybe something is up with the object lifecycle management code it injects into your model class.
It's been ages since I did any serious Ruby, wish I could be more helpful. One option is to take advantage of some of the direct CouchDB full text offerings like CouchDB Lucene: https://github.com/rnewson/couchdb-lucene
there is a scaffold created Story... and in the create action, there is
#story = Story.new(params[:story])
i was curious as to what is in params... so i want to dump out params... but there is no view associated with the create action... is there a way to dump out its content? is there a way to dump out the POST variables in of my code too? (to see what's going on in the lower level)
The easiest thing to do is just dump params out to the log:
Rails.logger.info("PARAMS: #{params.inspect}")
If you're in development mode, just look in your development.log and that line will be there.
The params scope is a combination of URL/FORM (GET/POST) fields, and it will be printed out in the log as part of the normal output processing, so you might not need your own dumping of it - any development or production log contains the params dump at the top of the log line, e.g.
Processing Clients::ClientsController#show (for x.x.x. at 2009-05-24 00:34:26) [GET]
Parameters: {"id"=>"303", "user_id"=>"2"}
Now I know Rails more, you can also simply use a
p params
in your code and look at the console's output (the log shown on the console)
If you're on a Mac, Spike is a great little app the analyses log files and will let you inspect params for requests, amongst other things.
Using Fiddler on Windows, it is shown
the HTTP line #1 is:
POST /stories HTTP/1.1
this is the POST content:
authenticity_token=62iw%2BrsxlCFsbnxsS7FXKRn6CcvJfjottrsBPlM5lZo%3D&story%5Bname%5D=Google+Main+Site&story%5Blink%5D=www.google.com&commit=Create
listed in a table:
authenticity_token 62iw+rsxlCFsbnxsS7FXKRn6CcvJfjottrsBPlM5lZo=
story[name] Google Main Site
story[link] www.google.com
commit Create
and the server log is:
Parameters: {"commit"=>"Create", "story"=>{"name"=>"Google Main Site", "link"=>"www.google.com"}, "authenticity_token"=>"62iw+rsxlCFsbnxsS7FXKRn6CcvJfjottrsBPlM5lZo="}
You don't need to anything except look in your logs (they live in /log). Unless you're fiddling with something, the logging of parameters is turned on by default in all logs.
Processing PostsController#create (for 127.0.0.1 at 2009-05-24 13:03:24) [POST]
Parameters: {"commit"=>"Create", "authenticity_token"=>"2G6BKOs8xNAaXiToVf4r1ko8QZzP9QAomi2PHVQC5Oc=", "story"=>{"something"=>"asdfafd"}}
Parameters lists all parameters, and the hash following "story" is the equivalent of params[:story] (everything comes to the server as strings, and Rails turns it into a HashWithIndifferentAccess so that you can access it with a symbol).
If you're on a *NIX system (including OS X) open a new terminal window/tab and type the following command:
tail -f log/development.log
You'll get a constant stream of requests coming in -- including params -- and the resulting DB actions. Invaluable for development/debugging, IMO.