Wearing Stripes (II)
This is the second part of my article about how we migrated our entire billing system to Stripe, here at WeTransfer. Head back to part one to read why we chose Stripe and how we designed our new system around it.
We were happy with how our brand new billing system took shape and quite eager to see it out in the wild. It was time to welcome all our users into the new system. Because the two billing systems were isolated from each other, we had a fairly simple job rolling out the new one. There were basically two ways a user could land on the new system: either through signup, or as a result of them being migrated. Let’s tackle the former first.
The Rollout #
We created a feature flag with a few targeting rules, and evaluated it each time a user had signed up. You can think of it as a high level if statement:
post '/signup' do if use_new_system? New::Services::SignupUser.new(user, params).perform else Services::SignupUser.new(user, params).perform end end
We persisted the evaluation on the user record, and used it in a few places to decide which implementation to use:
class User < ActiveRecord::Base has_one :old_subscription has_one :new_subscription def subscription uses_new_system? ? new_subscription : old_subscription end end
In order to minimise any potential issue that could arise, we performed a gradual rollout. We enabled the feature flag for a small percentage of users in certain countries, and then went on to increase percentages and select more countries.
We preferred to roll out small increments rather than the whole new system at once. Our MVP focused on signup and only supported credit cards as payment methods. It was only later that we added the possibility to cancel and reactivate a subscription, to update payment details, or support for other payment methods (such as iDEAL or SOFORT). This allowed us to focus on one feature at a time, which we could quickly validate from real production usage. Remember, we always had the emergency button available, that would switch to the old billing system, if needed.
Migration 101 #
After we made sure we were not adding any more users to the old system, it was time to move the ones that were already there. This was considerably trickier because we had to deal with some very sensitive payment information, that was out of our hands (and for a good reason). We identified two problems here: migrating user information and migrating payment information.
The first problem was not really a problem; we basically mapped data between the two systems, and created some Stripe objects along the way. Quite boring, and I won’t insist more.
The real challenge came with the second problem. As one might expect, it was a bit more involved than simply querying API A for payment information, then
POSTing it to API B. Payment information is considered sensitive and is subject to a number of regulations one must comply with. In other words, there is no API A or API B. The way this works, in theory, is that you contact your current payment provider, you ask them to dump all payment information they have and securely transfer it to your new payment provider. They, in turn, will do a best effort importing that payment information, and provide you with a mapping file between old and new data. In reality, it was obviously much more than that. Let’s see how it went.
We are talking about a threefold coordination effort (us — our old payment provider — Stripe) that we were responsible for. We had to break the news to the old provider that we want to move away from them. We had to create a plan for the migration, way ahead of time, and check that both providers agree with it. We had to make sure the two providers had their secure communication channel set up. And lastly, we had to follow up and mediate any issue that arose during the migration process.
We had to make sure timing was right on our side as well: that no more payment information would creep into the old provider during the migration. It may sound unrelated, given all signups had been already going to the new system, but what about old users that updated their card details? We had to turn some toggles off in order to keep data consistent. On the bright side, our old system gave us an unexpected, but important advantage in this matter: it used to charge users one week before their current period ended. Because Stripe charges users exactly when their current period ends, this resulted in a week when no recurring payments would be made. That made us more at ease with the migration, providing us with a time buffer in the eventuality that something went wrong.
It is important to mention that such migrations are performed in one go. None of the two providers were thrilled with the idea of batch migrations, which would’ve given us the room to fix any error that would’ve arisen. So, we had to be extra sure everything fitted perfectly on our side.
Migration Time #
That being said, on a warm Thursday morning, the big moment arrived: we started the migration. It took less than a day for our old payment provider to dump and transfer all payment information to Stripe, which was considerably faster than we expected. We were set for an early celebration, but reality quickly set us down.
First, there were issues with the dump format. Then, the dump was not complete. We settled these issues with the two providers and, soon after, we noticed various errors happening: invalid payment details, unsupported cards, expired mandates and what not. Luckily, we had anticipated such issues and prepared for the worst case scenarios. Due to our rollback strategies, extensive data validations and idempotent actions we managed to keep our data consistent and our users unaware of the burdens faced during the migration process.
Rinse and repeat, we overcame all these issues with the two providers and in around two weeks we managed to migrate almost all payment information to Stripe. I said almost, because some errors were not fixable and we had to give up on several card details, which meant some users needed to submit them again. Not bad, considering that was the biggest problem we had after such a migration.
Closing Notes #
After little more than ten months, we popped open the secretly stashed champagne bottle and savoured our victory. It was a fun ride and we learned a lot from this joint effort. There had certainly been ups and downs, but they are unavoidable at such a scale. On the bright side, all the challenges that got in our way made the celebration even sweeter in the end.
To conclude this article, I made a list of the most important things that I learnt while working on this project. I will definitely keep them in mind the next time I migrate billing systems:
- design the new system isolated and loosely coupled to the old one; implement the switch between the two systems early in the application flow;
- ensure idempotency for key processes;
- document. every. major. step;
- always keep stakeholders updated.