difficult architectural problem involving recurring payments and future events - ruby-on-rails
I'm looking for guidance on how to architect an elegant solution to what has become a bit of a thorny problem. Although I am using Ruby (and Rails) I think my problem is largely an architectural one, though my choice of language obviously has an impact in terms of suggestions involving libraries, etc., so the language remains relevant.
Anyway, in a nutshell: my application contains objects representing memberships, belonging to people who are members of fitness facilities. Memberships contain a series of recurring payments. Some memberships automatically renew at the end of their term, while others do not.
So for example, you may have a membership that is for an initial period of one year, and then renews month-to-month after that. In the application, creating a membership of this kind causes 12 recurring payments to be created. When the last month expires, so does the membership. A daily cron task is responsible for causing memberships to expire based on completed payments. If the membership is set to automatically renew, the same cron task will renew the membership.
You may also have memberships that have no initial term and simply run month-to-month or week-to-week. These work in a similar manner, minus the initial payment scheduling.
So far so good. What makes things complicated are the additional requirements:
administrators can "freeze" memberships (put them on hold), for specific durations, after which they automatically reactivate (e.g. to represent people who go away on vacation for a set period of time). I can choose to freeze a membership right now and have it reactivate later, or I can choose to schedule a freeze by setting the freeze date at some point in the future, as well as the reactivation date (note: there is always a reactivation date, which makes things a bit easier).
administrators can cancel memberships, either right now, or by setting a cancellation to occur in the future. (Future cancellations are not yet built.)
administrators can refund memberships, which is like a cancellation except any past payments are refunded.
What makes these difficult to deal with is the effect on recurring payments. When you freeze a membership, the recurring payments must "stretch out" around the freeze period, so that the period of time that represents the freeze is not paid for. This is both conceptually and programmatically difficult to handle. Payments, for example, may extend for different periods (i.e. each payment for someone who pays every other week pays for two weeks of a membership), and the date of cancellation may be anywhere within the period the payment covers.
For the freezes, I have taken the approach where the membership object contains some dates, namely "freeze_on" and "thaw_on" to handle the freeze period. However, the client now wants future cancellations as well, and I have noticed some bugs with the freezing functionality, which leads me to believe I need to reconsider my approach.
I am considering changing things so that future events can be scheduled but have no effect on the recurring payments portion of the application. The idea would be to queue up particular events. For example, a freeze in the future would be accomplished by queuing up a freeze event on a particular date, and a thaw event on a subsequent date (these two events would be connected into a single "scheduled freeze" from the user's perspective). A future cancellation would be handled similarly.
This approach has some benefits, for example, if you wanted to cancel a future cancellation (that's the kind of annoying, tricky stuff I'm talking about), you could simply remove the scheduled cancellation from the events queue.
However, I have the nagging feeling that I may simply be jumping from the frying pan into the fire. I'm wondering if anyone could provide me with some guidance on this issue. Are there design patterns or existing architectural principles for this sort of problem that I can examine?
An additional thing to note is that recurring payments for memberships with scheduled terms (i.e. not month-to-month automatically renewing) must exist as database records that can be edited (moved in time, price adjusted), so using temporal expressions (as Martin Fowler suggests) is not appropriate for this problem, so far as I know. I realize that my proposed solution of an events queue would not display to the user the changes that would happen to any existing recurring payments, but I think I can live with that.
not a scanlife bar code, it's a qr code
toronto, give us your creative people
Edit: To respond to the two great suggestions below (the comment boxes don't allow nearly this level of detail):
Kris Robison:
Yes, the freeze period can be an arbitrary length, although in practice I imagine it would be rare for it to be less than two weeks. But any solution should work regardless of the length of the period.
Yes, the renewal date changes - it is pushed forward by the length of the freeze. So if the freeze is two weeks long, it pushes the payment forward by two weeks. To make things especially tricky, in some businesses, the payments can only be withdrawn on specific dates - for example, some clubs only process payments on the 1st and 15th of each month. So when dates are pushed around, for these clubs, they have to "snap" to a particular date.
Can you explain in more detail why these rules affect event queuing but not management of subscription payments?
I'm interested in your amortization table concept. That's basically exactly what I have built already - a year-long membership with monthly payments creates 12, with weekly it created 52 - and each of these have an amount, tax, etc., associated with them, along with a state machine that governs "pending", "paid", "failed", and "refunded" states.
The part I am struggling with is how this table responds to events. Right now, if you set a freeze, it affects the table immediately by changing the dates of the payments. Set a freeze in the middle of the table, and it pushes payments forward. That sounds effective, but it's actually quite complex and hard to manage. How would your amortization table idea improve this situation?
Arsen7:
This sounds like the event queue I proposed originally. It seems obvious to me that you've worked with stuff like this before (I was impressed by your error check on the processing date, this is a great idea and one I intend to implement ASAP) so I'm hoping that you can explain your suggestion in a little more detail.
Specifically, I'm wondering how your concept would deal with the recurring payment situation I've described in my original question, and in the comment that I just left on Kris Robison's answer. If I have set up a schedule of recurring payments for a given purchase, and a freeze event is scheduled for right in the middle of the payments, would the schedule of payments remain unchanged until the date of the freeze became the current date, at which time the freeze would be instituted and the payments would move forward?
This strikes me as perhaps a great way to simplify my application, but I am wondering how users would perceive it. How would I indicate to them that the schedule of payments they were looking at when a freeze has been scheduled is no longer an accurate schedule, but will change once the freeze takes place?
Is it acceptable to apply a scheme used by banking, where you process all account operations once a day?
Every object may have a set of (future) operations, like freeze periods, and every day the object has to make a simple decision, like: "should I expire today or not?"
The good part is, that such daily processing is very simple to program. Also a strange renewal rules (in case you would want them) are simple to design: "is it Friday? is it the last one in this month? If yes, mark me as renewed, add some amount to required payment, or do anything".
That would be very costly (in terms of computing power) to calculate the status dynamically, every time the object is asked. If you store the current "account", you only need more complex calculations when you want to predict future state.
Consider it a pseudo-code:
def process(day)
raise "Already processed or missed a day" unless day == last_processed_day + 1
check_expiration day
check_frozen day
check_anything day
#...
self.last_processed_day = day
self.save!
end
RESPONSE:
Specifically, I'm wondering how your concept would deal with the recurring payment situation I've described in my original question, and in the comment that I just left on Kris Robison's answer. If I have set up a schedule of recurring payments for a given purchase, and a freeze event is scheduled for right in the middle of the payments, would the schedule of payments remain unchanged until the date of the freeze became the current date, at which time the freeze would be instituted and the payments would move forward?
This strikes me as perhaps a great way to simplify my application, but I am wondering how users would perceive it. How would I indicate to them that the schedule of payments they were looking at when a freeze has been scheduled is no longer an accurate schedule, but will change once the freeze takes place?
The "daily processing" scheme helps you with providing quick responses for questions which require complex calculations.
You have three "groups": current state (asked often), history (almost never changing, asked relatively rarely), and future.
Your daily processing procedure is not constrained to only update "current" state. If some event is scheduled for the processed day, the procedure probably needs to add "history" records.
If your users will often ask questions about future (and as you said, they will), the "processing" may also create a kind of cache for these questions. For example: find (calculate) the next payment date, and write it in a helper table (a schedule).
The main thing you need to do is to decide which questions will be asked by users, and whether you are able to calculate the responses on-the-fly or you need to have the answer prepared.
In a bank (it varies, of course) if you ask about your current balance, they may give you the answer which was true at the beginning of the day. "Better" banks will tell you that you had X$ at the morning, but now there are also Y$ waiting for accounting.
So, if you put a freeze record into the event queue, you may call a method, which will update the schedule at once. The same procedure (or a very similar) will or may be called in the daily processing routine.
Couple of questions come to mind:
Can the freeze be for time periods less than a month?
If so, does the renewal date change? Or how does the monthly payment get applied for the partial month?
Those affect how your event queuing system may work, but don't ultimately change the management of subscription payments.
One way to address subscription issues is to create an amortization table of subscription payments. If they are paying monthly for a year, twelve payments are queued up in the table. A weekly customer may have 52 payments in the table for the same year. Each time the renewal date comes up, after checking whether a freeze is in place, apply the next payment.
The amortization table keeps track of which payments have been made. If an account cancels, unpaid rows are refunded. If an account freezes according to your event queue, no payment is applied and the table remains static until the account is thawed.
Reply
It sounds like you have the concept of renewal date built into the amortization table. I use the amortization table more as a queue, and keep just the next renewal date with the subscription.
If the renewal date is inherent with the amortization table, then yes, it would be complicated to make changes as things go along. However, the renewal date should only affect the day on which you check to see if another payment gets applied.
If you are preserving partial payments while a subscription is on hold, and if a hold can be for an unspecified period of time, having the duration value on the amortization table lets you push a partial payment back in the queue in a "credit" state with a duration equal to the remaining time left in the payment. That way when the account is thawed, that partial credit is applied first and you calculate the next renewal date from the remaining duration.
Use some form of ordered list to preserve payment order at this point. It also comes in handy should someone ever want to insert a renewal period's worth of credit for customer service reasons.
Related
Making sure that item can only be bought by 1 person when 4000 people are trying to buy within a seconds
I run a marketplace iOS app and from time to time we have "competitions", where we have an especially sought after item for sale for a good price, that drops as a specific time. Sometimes thousands of people will try to buy this item within 1-2 seconds and I therefore need to make sure that only 1 person will get the item. The solution I have for it now feels kind of clumsy, so I was wondering how a good solution would look like when I use Firebase as my database. The process is as such: User finds the item on his iOS app and clicks "Purchase". A request is sent to our API (build on RoR) that processes the purchase (usually takes 10-20 seconds for the purchase to go through). Right now, I set the buyers ID temporarily as an attribute on the item, I wait a second and check whether the buyer ID is still the same on the item. It works, but it doesn't feel optimal. Any suggestions on how I can make sure 2 people can't purchase the same item?
To avoid something like this in your rails app, the keywords mutex and race condition should probably help you to find a bunch of appropriate gems. I personally like to use redis for this kind of task, because in redis, transactions are atomic by default (https://en.wikipedia.org/wiki/Atomicity_(database_systems)). So maybe this gem could suit your needs (untested): https://github.com/kenn/redis-mutex. For the theory, refer to this articles: https://en.wikipedia.org/wiki/Mutual_exclusion https://en.wikipedia.org/wiki/Race_condition
Store in /items/foo
a record with the structure:
{id:<blah>, available: <timestamp>, (purchaser: null)}
let buyers write their user name to to buy:
/item/foo/purchaser
You want 3 things to happen.
Block someone writing before the servers timestamp of available
only allow 1 person to do the operation. Once the /item/foo/purchaser is set, you don't want it modifiable (i.e. write once)
only allow the authenticated user id to be used in the purchaser field
To enforce this logic you use security rules, on the subpath of "/items/$itemid/purchaser"
".write": "now > data.parent().child('available').val()" +// 1.
"&& data.val() == null" + // 2.
"&& newData.val() == auth.id" // 3.
My guess is tht you should use locks. On a request coming in, check if you can acquire a lock. If yes, the the user is the first one. Then, the next requests won't be able to acquire the lock. This means the product as already been purchased. Take a look at this redis doc part : http://redis.io/topics/distlock
At the application(RoR) level, you can set a flag(eg: lock_foo=true) that is shared across the cluster(can be in your cache store). If this value is true, don't allow any other users to access the product/make the purchase.
You can definitely implement this with Firebase. As dvxam and Anshul Mengi mentioned, a lock system is the good way to go:
You could have on the document a property called lock:
{
"lock": {
"userId": "myUserId",
"expiresAt": "myTimestamp"
}
}
When a user clicks on the purchase button, you can use a Firebase transaction to make sure only one user can get the lock and that the first one gets it.
When another user clicks the purchase button, if a non-expired lock is present with a different userId, you can deny the purchase.
When the user completes the purchase you can then use another transaction to check if it is the same userId and if the lock is not expired.
Transactions are absolutely necessary here, and they are not available on the Firebase REST api (hence no more in the ruby wrapper), so you would need to run this code client-side using the iOS SDK, or to spin a nodeJS server for this task.
Hope it helps.
How about this for different: When user clicks purchase, immediately create a purchase request record that contains product, user and timestamp, and then poll every few seconds to see if the purchase was successful Run a background job that searches for un-purchased products that have at least one purchase request against them, and marks the product as purchased (selecting one purchase request / user as the "winner")
I'm not sure if there's a specific pattern I can apply in this case or how this is "normally" solved?
I can't speak to Firebase, but I can definitely speak to how this is "normally" solved in Rails and relational databases.
Before jumping in to code, note that it seems like you need linearizability, one of the hardest things to ask of a database, and some databases can't guarantee it even when they say they do. You might be able to hack around needing linearizability if all you need to know is whether it's been purchased or not, but I wouldn't take that hack lightly. Consistency in distributed systems is a really complex and edge-case-ridden topic, especially while under load (which it sounds like you'll be).
In Rails+RDB (postgres, mysql, sqlite) an atomic, linearized quantity update looks roughly like this (with some rails validation niceities thrown in):
class Product
validates :quantity, numericality: {greater_than: 0}, on: :purchase
def purchase
with_lock do # simultaneously aquires a lock and reloads the model
return false if !valid?(:purchase) # immediately release the lock if not valid
update_attribute(:quantity, quantity - 1) # saves without validation; YYMV
end
end
end
This general pattern of "lock+reload -> check -> update" is the gold standard for reliability, but it's "heavy." The first object to acquire the lock will win, but while it's doing its thing, all the other processes asking for a lock will be in queue. Somewhere there's a timeout and max connection pool defined, so if say 4000 locks are asked for within 1 second but it takes 10 seconds to determine success, you'll need 4000 connections and, even worse, the last lock asked for will be waiting for over 11 hours! That will make managing the connection pools and setting reasonable timeouts challenging.
The benefits, though, are that it will "just work" - if the first purchase fails, the next purchase will acquire a lock, and so on, until someone wins. Then, it will return helpful ActiveModel errors to everyone else in the queue. Additionally, it's simple enough code-wise that you know as long as your database provides linearizability, you're in the clear.
To mitigate the 11-hours issue hopefully you can very quickly deny everyone with outstanding locks to flush the queue.
I don't know exactly what you're doing while you try to make a purchase, but if it was just a credit card validation and a data update, I'd highly recommend the approach I've outlined with a database known to be linearizably consistent. Otherwise, you're going to need to consult a true distributed systems expert or run your users under the bus figuring this out.
Set time to send out HTTP request
Is there a way in Rails to send out a request at a certain time? I'm using an external credit card charging API, and I want to adjust each monthly subscription based on how many referrals they have (10% each, 10 referrals max). The API has a beta referral system built in, but it doesn't seem to work the way I need it to. Plus, there are just too many unknowns that I'd rather not get into at the moment. I just want to get it up and working, and since my system is fairly simple, I'd rather just do it manually. There's a billing date for each subscription, and what I want to do is just manually adjust the price of the subscription based on how many active users there are containing the referral code of the user being charged. I'd like to just send out this request to the API just before they're billed. Like sometime around subscription.next_billing_at - 1.minute. Then just set the subscription.price to price - (price * (User.where(referral_code: current_user_code)).count / 10). I'm aware this is far from an optimal approach, considering the amount of extra requests being made each month, but since we're small right now, it shouldn't be a problem. Again, it's just a temporary solution so we can get things running now.
There are two options which directly answer your question. Write a rake task and run it daily with cron via the Whenever gem. If you take this approach, you will have to have the task just load all subscriptions which are due to be billed in the next cycle and update them as required. Alternatively, use something like Resque-scheduler, which would allow you to run some task at next_billing_at - 1.minute or something. But if you are small, why not just update the price every time a new referral is created using a callback? Unless there are specific rate or query limits on this API, I doubt a card processor is going to be affected by the traffic you generate. Of course if there are other requirements, like, a referral only applies after a month or something like that, you are going to be stuck with one of the first 2 options, and the Cron + Rake task is probably the best solution in that case.
What should I use as a starting point each month for counting how much of my resources the user consumes?
Say, I use stripe subscriptions for my users with one plan which is $10 per month for 100 API requests and that's the standard price. However, if the user has used 150 API requests, I charge them $3 more on top on $10. For 200 and more requests it's total of $17. Say, I've subscribed the user on October 9th. On November 9th they will be charged again by Stripe. When should I "freeze" the amount of the APIs the user has used during the month to calculate the overall price and start counting from zero again for the new period of November 9th - December 9th? As far as I'm concerned, it's either on payment.success or invoice.success event in my stripe web hook controller. However, I think it's not reliable because: Which one is it exactly - payment.success or invoice.success - the one I need? They occur both, as far as I know sometimes payment.success occurs first, sometimes - invoice.success and it's unknown which one occurs first in each particular case. Theres's the time period between these 2 events, I don't know how long exactly, probably a few minutes. But during those few minutes the user might consume at least one Invoice I'm not sure if it's guaranteed that for each user the events payment.success and invoice.success occur only once per month. Do they really? If not it's even less reliable to use them to achieve my goal. invoice.created event occurs 1 hour before the Invoice is closed.
Um. Why wouldn't it be the moment your user clicked 'BUY'? The fact it took an hour/day for payment to be received isn't relevant is it? Think about iOS apps. I can buy the app right now, but most of the time I don't get an invoice from Apple for several days (even a week sometimes). I think you're going to be frustrated trying to time it to events from your payment provider. IMHO that should be done within your app.
perform some logic before object is loaded from database
The situation: The magazine accepts submissions. Once you submit, an editor will schedule your submission for review. Once it has been reviewed, you are no longer allowed to edit it. So, I have submission in various states. "Draft", "queued", "reviewed", etc. Most of the switches into these various states are triggered by some action, e.g., a submission becomes queued when an editor schedules it. Easy peasey. However, the switch into the "reviewed" state is not triggered by any action, it just happens after a certain datetime has passed. I have two thoughts on how to accomplish this: Run a daily/hourly cron job to check up on all the queued submissions and switch them to reviewed if necessary. I dislike this because I would prefer it to be hourly, so that I can edit my submission up to three hours before a meeting starts Hourly cron jobs cost money on Heroku, and this application will either never make money or won't make money for months and months to come Somehow construct a before_load ActiveRecord callback, that will perform some logic on submissions each time they are loaded. "Queued? No? Nevermind. Otherwise, switch it to 'Reviewed' if its meeting is less than three hours away." I wanted to get people's input on the second idea. Is that an atrociously smelly way to accomplish this? If so, can you suggest an awesomer third way? If 'no' to both of the above, can you give tips on how to perform such logic each time a record is loaded from the database? I would need to always perform some logic before doing a select from the submissions table (which is gearing up to be the most-queried table in the app...) If there's no good way to accomplish Option Two (or, I hope!, Option Three), I will resort to Option One with a daily cron job. Being able to edit up to a day before a meeting will just have to suffice.
Maybe using after_find, although your performance will sort of suck, same goes if you do something as crazy as before_load, performance would suck, that said money might be more important than performance, if that is so, I would go with the after_find.
Designing a system for handling in-application economy using currency or credits
I'm currently developing an application where I need to implement some sort of way of handling credits which are purchased by users of my application. These credits will then be used to preform certain tasks inside my application such as purchasing hits in mechanical turk. The reason I need to do this is because in the case of mechanical turk there is a possibility that our orders won't be filled and instead of just keeping the extra money for hits they didn't get I want to credit them for future purchases. The important parts I need help fleshing out is how do you accurately manage an ongoing total of credits. I can't obviously calculate it every time. Additionally I need to manage the adding and subtracting of the credits. Also I probably need to track the origin of these credits, ie money or free because it is possible we might give out free credits as a reward but we need to be careful how to handle turning free credits into cash because it opens an incentive for scammers to use the credits to purchase turk hits then do the turk hit themselves and keep the money.
I currently work on a system with something very similar. Users have an Account which tracks all transactions - origin, type and amount. We have a finite number of transactions types (including obvious the ones like credit/debit). The Account table is therefore actually an archive of all of the transactions for a particular user account, rather than being a simple running total. Account totals are therefore calculated every time we need them, although it would be pretty simple to add some aggregates to the database that increment/decrement a total value as transactions are processed. As with all performance-related issues - don't do it until you need to, summing values across some smartly indexed columns is fine for our current system.