Production breaks at 2am. A founder or whoever is awake restarts something, it comes back, and everyone goes to sleep. Two weeks later it happens again. Nobody owns it, because there is no one whose job it is.
The instinct is to solve this with a person, either by hiring one or by formalising who wakes up. That is usually the second thing to do, not the first. The first is to notice that most out-of-hours breakage in a small system comes from a very short list of causes, and that fixing a cause is permanently cheaper than staffing a response to it.
Start by counting, not staffing
Before choosing an option, spend one hour writing down the last ten incidents: what broke, what the trigger was, what the fix was, and how long it took. Most teams have never done this, and it is common for the list to collapse into two or three repeating causes.
That list decides everything below. Ten incidents with ten different causes is a systemic problem and needs ownership. Ten incidents with two causes is a bug list, and no amount of on-call rota fixes a bug list.
The causes that produce most night-time failures
In small production systems, out-of-hours failures cluster heavily. Disks and volumes filling, usually logs. Certificates expiring, which fails at a fixed time rather than under load. Memory leaks that take days to surface, so they hit at whatever hour the threshold is crossed. Scheduled jobs colliding with backups or with each other. A dependency that rate-limits or fails, with retry behaviour that turns a small outage into a large one. And deploys that went out late in the day and degraded slowly.
Every one of those has a permanent fix that costs less than a year of somebody's sleep.
The five options, honestly
Do nothing deliberately. A real option, and sometimes correct. If you are pre-revenue with no paying users, downtime at 3am costs approximately nothing, and engineering time is better spent on the product. The failure mode is doing this by accident and discovering the cost only when a customer notices first.
Rotate the founders or the senior engineers. Cheap and fast, and the standard answer at very small scale. It works while the incident count is low and the causes are known. It stops working quietly, because the cost lands as attrition and slower daytime output rather than as a line item, so nobody notices the bill.
Remove the failure class. Usually the highest return per rupee, and the most skipped. Log rotation and disk alerts. Automated certificate renewal. Health checks that restart before a human is paged. Backups verified rather than assumed. Retries with backoff and a circuit breaker so a dependency blip does not cascade. Most of this is a few days of work and it deletes categories of incident permanently.
Buy your way out with managed services. Moving a self-managed database, queue or search cluster to a managed equivalent transfers a whole class of 3am work to a vendor. It costs more per month and less per incident. For a team with no operations person, that trade is usually correct, and the comparison should be against the fully loaded cost of the person you would otherwise need, not against the raw infrastructure line.
Get ownership, either by hiring or fractionally. If the incident list is long and varied, the problem is that nobody is responsible for reliability as a discipline. A full-time hire is right when the load is continuous. A fractional engagement is right when the work is real but not yet full time, which at five to fifty engineers it usually is not.
What a reasonable order looks like
| Step | Roughly what it costs | What it buys |
|---|
| Write down the last ten incidents | One hour | Tells you whether this is a bug list or a systemic gap. Skipping this is how teams hire for the wrong thing. |
| Fix the repeating causes | Days, once | Permanently deletes the most common categories. |
| Monitoring and alerting that page a human only when a human is needed | Days, once | Turns silent failures into known ones, and stops paging for things that self-heal. |
| Managed services for the noisiest components | Higher monthly bill | Transfers a category of work to a vendor with a real operations team. |
| Named ownership, in-house or fractional | Monthly | Someone whose job includes reliability, so the list does not regrow. |
The signal that you actually need a full-time hire
Not the incident count. The signal is that reliability work has become continuous rather than episodic: deploys many times a day with engineers regularly blocked, a genuine need for a staffed rotation with contractual response times, or a team past roughly fifteen to twenty engineers where coordination alone is a job. Below that, the work is real but intermittent, which is the shape a retainer fits.
We wrote the longer version of that decision in hire a DevOps engineer or use a fractional team, including what the fully loaded cost of hiring in India actually looks like.
If you want a second opinion on your list
Bring the ten incidents to a 20-minute review and we will tell you which bucket you are in. If the answer is "fix these three things and you will stop being woken up", that is the answer you will get, and it does not require hiring us.
