I'm using Rails with Sunspot gem and Websolr for Solr hosting. Let's say I have a Solr indexed ActiveRecord model like this:
class User < ActiveRecord::Base
searchable do
text :name
end
end
Now I want to remove indexing from this model, so I just remove this indexing code (searchable block). Will the indexed data be removed automatically from Solr? I assume not and that I need to remove/reindex in manually. I can reindex everything like this:
Sunspot.index
Sunspot.commit
But reindexing my whole database will be slow. Is there another - faster way to stop indexing 1 model? I don't see any interface in Websolr to look through and delete records from the index and I can't seem to find any information on how to remove models from indexing with Sunspot.
I can reindex everything like this:
No, you can't, if you need to reindex again then open Rails console by typing rails c and then
=> Model.index
# User Load (15.6ms) SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1 [["LIMIT", 50]]
# [2018-02-27 18:45:04 +0600] Start Indexing
# D, [2018-02-27T18:45:05.439714 #15296] DEBUG -- : SOLR Request (762.4ms) [ path=update parameters={} ]
And for the removing indexed data then try to the following
curl http://localhost:8982/solr/default/update?commit=true -d '<delete><query>*:*</query></delete>'
Let's describe on the url part
localhost: is the hostname.
8982: port number where your Solr server running if you confusing what is your port number then open rails console and type this puts Sunspot.config.inspect it will show your port number.
/solr/default/update: this is the path where your command execute.
?commit=true -d: That is the instruction.
'<delete><query>*:*</query></delete>': that is the method.
I have found this here but it's not working for me because there are port number and path different, I have edited my own way and it's working perfectly.
I think it will help.
Related
When creating the actual database table in rails console, User.table_exists? does not give me the correct result.
But when I look into ActiveRecord::Base.connection.tables it shows that the table exists. I have to exit rails console and go back in before User.table_exists? give me the correct value. Is there some caching going on? A bug in Rails 5.2.1? Or am I missing something?
Code to reproduce:
in terminal/bash
rails generate model User name:string
rails console
in rails console
User.table_exists? => false
ActiveRecord::Base.connection.tables => []
ActiveRecord::Migration[5.2].create_table :users =>
-- create_table(:users)
(0.1ms) SELECT sqlite_version(*)
(1.5ms) CREATE TABLE "users" ("id" integer PRIMARY KEY AUTOINCREMENT NOT NULL)
ActiveRecord::Base.connection.tables => ["users"]
User.table_exists? => false
exit
in terminal/bash
rails console
in rails console
User.table_exists? => true
using:
Ruby 2.5.0
Rails 5.2.1
ActiveRecord::ModelSchema::ClassMethods#table_exists? is a wrapper around ActiveRecord::ConnectionAdapters::SchemaCache#data_source_exists?.
As the name implies, this is a cache, probably populated when the Rails app starts. If you modify the database schema the old schema will still be cached (arguably a bug or missing feature). You can reload User's schema cache with User.reload_schema_from_cache or possibly connection.schema_cache.clear_data_source_cache(Users.table_name.
I guess it's a cache thing. I couldn't tell if a bug or not, I don't have much knowledge of rails internals but I gues it's that way by design. It's not even a common use case to create tables from within the console.
You could skip the exit > enter steps calling reload! on the console to reload everything though.
I'm running a Rails app (v 3.1.10) on a Heroku Cedar stack with Papertrail add-on going crazy because of the size of the logs.
My app is really verbose and the logs are getting huge (really huge):
Sometimes because I serialize a lots of data in one field and that makes a huge SQL request. In my model I have many:
serialize :a_game_data, Hash
serialize :another_game_data, Hash
serialize :a_big_set_of_game_data, Hash
[...]
Thanks to my AS3 Flash app working with bigs sets of json...
Sometimes because there's a lots of partials to render:
Rendered shared/_flash_message.html.erb (0.1ms)
Rendered shared/_header_cart_info.html.erb (2.7ms)
Rendered layouts/_header.html.erb (19.4ms)
[...]
It's not the big issue here, but I've added this case too because Jamiew handle it, see below...
Sometimes because there's lots of sql queries on the same page:
User Load (2.2ms) SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1
Course Load (5.3ms) SELECT "courses".* FROM "courses" WHERE (id = '1' OR pass_token = NULL)
Session Load (1.3ms) SELECT "sessions".* FROM "sessions" WHERE "sessions"."id" = 1 LIMIT 1
Training Load (1.3ms) SELECT "trainings".* FROM "trainings" WHERE "trainings"."id" = 1 LIMIT 1
[...]
It's a big (too) complex App we've got here... yeah...
Sometimes because there's a lots of params:
Parameters: {"_myapp_session"=>"BkkiJTBhYWI1MUVlaVdtbE9Eb1Y2I5BjsAVEkiEF9jc3JmX3Rva2VlYVWZyM2I0dEZaR1YwNXFjZhZTQ1uBjsARkkiUkiD3Nlc3Npb25faWQGOgZFRhcmRlbi51c2yN1poVm8vdWo3YTlrdUZzVTA9BjsARkkiH3dAh7CMTQ0Yzc4ZDJmYzg5ZjZjOGQ5NVyLmFkbWluX3VzZXIua2V5BjsAVFsISSIOQWRtaW5Vc2VyBjsARlsGaQZJIiIkMmEkMTAkcmgvQ2Rwc0lrYzFEbGJFRG9jMnZvdQY7AFRJIhl3YXJkZW4udXNlci51c2VyLmtleQY7AFRbCEkiCVVzZXIGOwBGWwZpBkkiIiQyYSQxMCRBUFBST2w0aWYxQmhHUVd0b0V5TjFPBjsAVA==--e4b53a73f6b622cfe7550b2ee12678712e2973c7", "authenticity_token"=>"EeiWmlODoYXUfr3b4tFZGV05qr7ZhVo/uj7a9kuFsU0=", "utf8"=>"✓", "locale"=>"fr", "id"=>"1", "a"=>1, "a"=>1, "a"=>1, "a"=>1, "a"=>1, "a"=>1, [...] Hey! You've reach the end of the line but it's not the end of the parameters...}
The AS3 Flash app send big json data to the controller...
I didn't mention the (in)famous "Assets pipeline logging problem" because now I'm using the quiet_assets gem to handle this:
https://github.com/evrone/quiet_assets
So... what did I try?
1: Dennis Reimann's middleware solution:
http://dennisreimann.de/blog/silencing-the-rails-log-on-a-per-action-basis/
2: Spagalocco's gem (inspired by solution #1):
https://github.com/spagalloco/silencer
3: jamiew's monkeypatches (inspired by solution #1 + a bonus):
https://gist.github.com/1558325
Nothing is really working as expected but it's getting close.
I would rather use a method in my ApplicationController like this:
def custom_logging(opts={}, show_logs=true)
disable_logging unless show_logs
remove_sql_requests_from_logs if opts[:remove_sql_requests]
remove_rendered_from_logs if opts[:remove_rendered]
remove_params_from_logs if opts[:remove_params]
[...]
end
...and call it in any controller method: custom_logging({:remove_sql_requests=>1, :remove_rendered=>1})
You got the idea.
So, is there any good resource online to handle this?
Many thanks for your advices...
I"m the author of the silencer gem mentioned above. Are you looking to filter logging in general or for a particular action? The silencer gem handles the latter problem. While you can certainly use it in different ways, it's mostly intended for particular actions.
It sounds like what you are looking for less verbose logging. I would recommend you take a look at lograge. I use that in production in most of my Rails apps and have found it to be quite useful.
If you need something more specialized, you may want to look at implementing your own LogSubscriber which is essentially the lograge solution.
Set your log level in the Heroku enviroment
View your current log level:
heroku config
You most likely have "Info", which is just a lot of noise
Change it to warn or error
heroku config:add LOG_LEVEL=WARN
Also, when viewing the logs, only specify the "app" server
heroku logs --source app
I personally, append --tail to see the logs live.
heroku logs --source app --tail
I am writing a rails applications that utilize solr for full text search
In development mode, I used the sunpost solr gem which is really handy. I used the sqlite3 database in development and everything went smooth.
Now is the time to move to production server and I installed the solr-tomcat package and moved to my production database which is Mysql. I moved the conf files of solr from my application folder to /usr/share/solr/conf
Suddenly, I cannot reindex, and solr returned this
rake RAILS_ENV=production sunspot:solr:reindex
[# ] [ 50/7312] [ 0.68%] [00:00] [00:41] [ 175.82/s]rake aborted!
Mysql::Error: Unknown column 'barangs.' in 'where clause': SELECT `barangs`.* FROM `barangs` WHERE (`barangs`.`` >= 0) ORDER BY `barangs`.`` ASC LIMIT 50
Intrigued, I tried to reindex with the development database, all is well and can be reindexed. This behavior left me baffled
Any help will be appreciated
I got the problem. It was due to me dropping and restoring tables recklessly and it loses some relationships definitions.
Hence the "missing" column on table 'barangs'.'MISSING' solr can't find the specified column.
For others who are facing the problem, I recommend inspecting the relationship between models first
How does one create a Rails migration properly so that a table gets changed to MyISAM in MySQL? It is currently InnoDB. Running a raw execute statement will change the table, but it won't update db/schema.rb, so when the table is recreated in a testing environment, it goes back to InnoDB and my fulltext searches fail.
How do I go about changing/adding a migration so that the existing table gets modified to MyISAM and schema.rb gets updated so my db and respective test db get updated accordingly?
I didn't find a great way to do this. You could change your schema.rb like someone suggested and then run: rake db:schema:load, however, this will overwrite your data.
The way I did it was (assuming you are trying to convert a table called books):
Save the existing data from the CLI: CREATE TABLE tmp SELECT * FROM books;
In your new migration file, drop the books table and recreate it with :options => "ENGINE=MyISAM" like someone said in the comment
Copy the contents back: INSERT INTO books SELECT * FROM tmp
i think that if you change your schema format (config.active_record.schema_format) from :ruby to :sql, all sql will be saved there.
i'd do some tests on a fresh app first if i were you, see how it works.
You can run any sql in migrations. This worked for me:
class ChangeMapOnlyUsersEngine < ActiveRecord::Migration[5.1]
def change
MyModel.connection.execute("ALTER TABLE my_models ENGINE = 'MyISAM';")
end
end
When I did this in the other direction (InnoDB -> MyISAM) it worked fine, without loss of data so I don't think it's neccesary to create temporary tables or similar. Note that MyISAM doesn't support transactions, so any tests against the database for a corresponding ActiveRecord model will be persisted, with a risk of test pollution.
I know that in rails 2.3.2 ActiveRecord queries are cached, i.e. you may see something in the development/production log:
CACHE (0.0ms) SELECT * FROM `users` WHERE `users`.`id` = 1
I was wondering if the same principles apply to rake tasks.
I have a rake task that will query a lot of different models, and I want to know if I should implement my own caching, or if this behavior is included by default.
Also, is there a way to see the sql queries that are performed during the rake task? Similar to that of the development/production log
The SQL cache isn't enabled per default for rake tasks. You can wrap your code in a cache block, like this:
task :foobar => :environment do
ActiveRecord::Base.connection.cache do
User.find 1 # Will hit the db
User.find 1 # Will hit the cache
end
end
This is essentially what Rails does for controller actions. Note that the cache uses memory and rake tasks has a tendency to work with large sets of data, which might give you issues. You can selectively opt-out of caching for parts of your code, using uncached
You are talking about ActiveRecord Query Caching. That should work in Rake-Tasks too, provided you're running them in an environment with caching enabled, e.g. production.
See Rails Guide on Caching for examples.
It may or may not be the right sort of caching for your case:
u1=User.find 1 # loads user1 first time from DB
u2=User.find 2 # loads user2 first time from DB
u1again = User.find 1 # loads user1 from cache
all = User.all # loads user1 and user2 from DB again
A rake task will run in the the environment you specify, in which case it will adopt the rules of that environment.
You can set the rails env from the command line:
RAILS_ENV=test
Logging can be set as part of rake and you should see this in your normal Rails log.