I want to fetch customers from remote db in background job. But during this time I want to keep using postgresql for other requests. I want to know if I establish_connection in worker, during this process does it cause any problem? For example during this process my app responses successfully from postgresql?
I've tried establish_connection in background job. But I need to be sure if it cause any problem.
ActiveRecord uses database connection pool so your background job processes and app server processes get separate DB connections. So yes your app and background job can keep running in parallel as long as you have enough connections in the pool.
Not sure why you need establish_connection. See if the newly added load_async is of any help.
Related
I have an application that handles some data in memory.
I'd like to close the operations and persist the data into DB so that a reboot wouldn't destroy it.
My app opens some resources in various third parties and it I'd like to close them. After that the app can happily go down and wait until it reboots.
What I found is that Heroku has various webhooks for application deployment state changes and so on. But I couldn't find a way to trigger a webhook before the DB becomes read only.
I would like to have a webhook that tells me that "in 5 minutes PostgreSQL will become read only". And then later the app can reboot and for now it doesn't matter.
Also I couldn't find any info if this is even possible. I couldn't find an email for support as well.
Is there a way to do it? Is it even possible?
(I have an Event-Sourced app that saves event data into DB but persists the data in-memory as it runs. So I don't want to continuously bash all of my state into the DB).
It sounds like there is some amount of confusion with regards to your understanding about the various parts of dyno and database uptime on Heroku.
Firstly, a database going into read-only mode is a very rare event usually associated with a critical failure. Based on what behavior you're seeking and some of your comments, it seems like you may be confusing database state changes with dyno state changes. Dynos (representing the servers for your application runtime), are restarted once per 24 hours roughly and these servers are ephemeral. Thus the memory is blown away. The 'roughly' part accounts for fuzzing so that all of your dynos aren't restarting at the same time which would cause availability issues.
I don't think you actually need a webhook here. Conveniently, shortly before a dyno is due to be cycled (and blow away your memory) it will receive a SIGTERM and be given 30 seconds to clean up after itself. That SIGTERM can be trapped and you can then save your data to the database.
I'm interested in creating a system that can queue user ids into categories and then poll values at regular intervals in order to run some code with them.
I'm unsure of how to do this in rails however, but my first thoughts would be to have some sort of temporary db table that stores the ids alongside categories, and resets if the server restarts. I have no idea how I would implement the background process to repeatedly process entries. Perhaps I can possibly achieve all of this with some sort of background worker?
For executing background jobs in rails there are so many ways so try anyone
sidekiq
delayed_job
microservices
our rails web app has to download/unpack archives with html pages from ftp on request for user's viewing through the browser.
the archive can be quite big, so user has to wait until it downloads/unpacks on the server.
i implemented progress bar the way that i call fork/Process.detach in user's request, so that his request is done but downloading/unpacking process continues running in the background. and javascript rendered in his browser pings our server for status until all is ready and then it redirects him to unpacked html pages.
as long as user requests one archive, everything goes smoothly, but if he tries to run 2 or more requests at the same time(so that more forks are started), it seems that only one of them completes, and the rest expires/times outs/gets killed by passenger(?). i suppose its the issue with Passenger/forking.
i am not sure if its possible to fix it somehow so i guess i need to switch to another solution. the solution needs to permit immediate and parallel processing of downloads. so that if user requests multiple archives, he has to see download/decompression progress in all of them at the same time.
i was thinking about running background rake job immediately but it seems very slow to startup(also there's a lot of cron rake tasks happening every minute on our server). reason i liked fork was that it was very fast to start. i know there is delayed job, we also use it heavily for other tasks. but can it start multiple processes at the same time immediately without queues?
solved by keeping the fork and using single dj worker. this way i can have as many processes starting at the same time as needed without trouble with passenger/modifying our product's gemset (which we are trying to avoid since it resulted in bugs in the past)
not sure if forking inside dj worker can cause any troubles, so asked at
running fork in delayed job
if id be free to modify gemset, id probably use resque as wrdevos suggested, or sidekiq, or girl_friday(but thats less probable because it depends on the server running).
Use Resque: https://github.com/defunkt/resque
More on bg jobs and Resque here.
https://github.com/blog/542-introducing-resque
I have a Rails app hosted on Heroku. I have to do long backend calculations and queries against a mySQL database.
My understanding is that using DelayedJob or Whenever gems to invoke backend processes will still have impact on Rails (front-end) server performance. Therefore, I would like to set up two different Rails servers.
The first server is for front-end (responding to users' requests) as in a regular Rails app.
The second server (also a Rails server) is for back-end queries and calculation only. It will only read from mySQL, do calculation then write results into anothers Redis server.
My sense is that not lot of Rails developers do this. They prefer running background jobs on a Rails server and adding more workers as needed. Is my sever structure a good design, or is it an overkill? Is there any pitfall I should be aware of?
Thank you.
I don't see any reason why a background job like DelayedJob would cause any more overhead on your main application than another server would. The DelayedJob runs in it's own process so the dyno's for your main app aren't affected. The only impact could be on the database queries but that will be the same whether from a background job or another app altogether that is accessing the same database.
I would recommend using DelayedJob and workers on your primary app. It keeps things simple and shouldn't be any worse performance wise.
One other thing to consider if you are really worried about performance is to have a database "follower", this is effectively a second database that keeps itself up to date with your primary database but can only be used for reads (not writes). There may be better documentation about it, but you can get the idea here https://devcenter.heroku.com/articles/fast-database-changeovers#create_a_follower. You could then have these lengthy background jobs read data from here leaving your main database completely unaffected.
This page http://devcenter.heroku.com/articles/maintenance-mode doesn't give any indication.
Server queries to Heroku could run up to 30 seconds before they got terminated forcefully. So I am wondering what would happen if I push new code to the busy server, or set it to be in Maintenance mode? Would the existing queries just stopped? What if it is writing to a database, etc? Would it leave my data in a corrupted state?
Is there a correct way to let Rails app to shut down gracefully (finishing existing queries but not accepting any new one), so that I can upgrade the server code?
Thanks.
When you put your app in maintenance mode you are not changing your codebase at all. It's a front-end configuration.
It means, if a query was sent to the database, the database won't be stopped and the query will be executed. The connections are not dropped when you switch to maintenance mode.