Maintenance mode on. No customer could place an order or start a payment on a system that might be compromised.
Why first: stop new money and data moving before touching anything else.Engineering note · 05 Incident response
When your cloud host gets breached:
rotate everything, then wait.
In October 2026 Railway, the platform that runs the AP Enterprises backend and database, was attacked, and every secret stored there had to be treated as stolen. This is what I did in the first day, what I checked afterwards, and what went wrong along the way.
Topic
Incident response · Secret rotation, blast radius, post-incident review
Grounded in
The AP Enterprises app and API, 20–25 October 2026
What happened 01
The breach wasn't ours. The secrets were.
On 20 October 2026 Railway became aware of what it called a sustained, targeted campaign against its infrastructure. The next day it disclosed anomalous activity in the systems that store customers' environment variables: an unauthorized party may have accessed them encrypted, and in some cases obtained the decrypted values. Railway paused all deployments while it investigated, and asked every customer to rotate every sensitive variable.
Railway described the attack as part of a broader series of attacks on platform companies, mainly after AI-provider API keys. That's why hosting platforms are attractive targets: one breach gives an attacker the secrets of thousands of customers at once. AP Enterprises was one of those customers. Nothing in the code or infrastructure I'd built was attacked, but the values the server uses to reach everything else were sitting in the place that was.
Railway's email reached me on 22 October at 06:28 IST. It was the general disclosure and didn't say whether the AP workspace was in the affected subset, so I treated everything as exposed.
Sources: Railway's disclosure, preliminary report and status page.
Blast radius 02
What was on Railway
Every backend secret, for both staging and production: the database password, the Razorpay key secret and webhook secret, the Cloudflare R2 storage keys (invoices and return photos), the email provider key, the push-notification credentials, the log-shipping token, a Railway account token, and the JWT signing secret.
That last one is the most dangerous single value on the host. Anyone holding it can mint a valid access token for any user, administrators included, without ever logging in.
What wasn't
The CI secrets in GitHub Actions, the private key for the encrypted database backups, and the Android signing keystore were all stored elsewhere and were never exposed. The website itself is served from Cloudflare and was unaffected.
Splitting secrets across providers wasn't designed for this exact scenario, but it meant the backups, the release pipeline and the ability to ship the app all survived a full compromise of the host.
RailwayPostgreSQLRazorpayCloudflare R2Firebase Cloud MessagingJWTGitHub Actions
The response 03
Every secret revoked at its provider (database, Razorpay, R2, email, notifications, logs, Railway) and a replacement issued. From this point the stolen values didn't work anywhere.
Revoked, not just replaced: a new key does nothing if the old one still works.The new values were deliberately not installed on Railway while its platform was still under investigation. The app stayed down on purpose.
Trade-off: two to three days of downtime instead of a second exposure.Railway confirmed its builds were safe. New values installed, backend redeployed, staff signed in normally.
The JWT secret change signed everyone out once; apps refreshed silently.Full review of payments, refunds, orders, accounts and admin activity. Rotation runbook written; app and server updates released.
The review is what turns "probably fine" into "checked".Key decisions 04
Downtime was the cheap option.
The obvious move after rotating is to put the new keys straight back on the host and bring the app up. I didn't, because the host was the thing that had been breached. Installing fresh secrets on a platform still under investigation risked handing the attacker a second set.
Leaving the app down cost two to three days of availability. For a young app with a small number of orders, that was a cheap price for certainty. A busier business would weigh it differently, and the runbook now records that as a decision to make, not a default.
One detail made this safer than it looks. The old JWT secret stayed loaded in the running backend until it was redeployed, so in theory a forged token could still be accepted. But revoking the database password meant the backend couldn't reach the database at all: within the connection pool's recycle window every request failed. A forged admin token had nothing to act on. Pulling one dependency took the risky secret out of play with it.
What I checked 05
No sign anything was used.
- Payments: none made during the exposure window; every gateway payment on record completed normally before it
- Refunds and disputes: none issued, none open, no failed payments
- Orders: none could be placed in maintenance mode, and no paid order is missing
- Staff and admin accounts: none created, promoted or changed since 20 October
- Admin and staff activity: every audit-log entry from 20 to 25 October reviewed and recognised
- Backups: the encrypted nightly backup ran green as soon as it had the new password
What went wrong 06
The maintenance page needed the thing I'd switched off
The maintenance flag lives in the database. Once the database password was revoked, the server couldn't read the flag, and the app fails open, so visitors probably saw the normal screens with every action failing instead of a clear notice. A maintenance page served by the web host, with no backend behind it, would always show.
Two days of logs, and nothing noticed
Railway's disruption, and then its invalidation of every account token, silently stopped log shipping. About 48 hours of server logs never arrived, and no alert fired. I covered the gap with the database audit log and the payment gateway's own records, but an alert on missing logs would have caught it the same day.
Rotating by hand invites mix-ups
In the same pass that replaced eight secrets, the new storage token was pasted into the GitHub secret that deploys the website. Nothing leaked, but the next production web deploy failed with an authentication error.
One failed check blocked a different deploy
That failed web deploy was a check on the merge commit, and Railway won't deploy a commit with a failing check, so the backend release was quietly skipped too. It was spotted and redeployed by hand. Release checks now confirm the live backend version after every merge.
Lessons 07
Your host's breach is your incident. Know which of your secrets live there before you need to.
Revoke at the provider first; replacing a value isn't the same as killing the old one.
Don't put fresh secrets back on a platform that's still under investigation.
A database reachable from the internet with only a password makes that password the whole game.
A signing secret with no key versioning can only be rotated all at once. Plan for it.
Label each new token with its purpose and environment, and check each consumer right after updating it.
Alert when logs stop arriving, not just when they say something bad.
Write the rotation runbook before the incident. Mine was spread across three documents until this one.
Running on a shared platform? 08
Know your blast radius
before the email arrives.
If you're not sure which secrets your host holds or how fast you could rotate them, I'd be glad to help you work it out.
Start a conversation ↗