The more that companies invest in internal tools, the more they’re able to enhance the speed and safety of how they operate. But because building good internal tooling has a nonzero upfront cost, companies tend to delay this as long as possible. In my experience, this is a mistake. Companies should invest in tooling sooner, because the upside is higher than people realize, and the upfront costs can be mitigated.
Let’s walk through how a typical early-stage SaaS startup might invest into internal tooling and security practices as they grow, which I would argue is suboptimal:
Stage 1 - everyone has access to prod: They launch their product and start getting customers. One of their users asks for more time on their free trial. The startup hasn’t built any “extend trial” functionality, so one of the engineers goes directly into the database and runs an `update` query directly against the production database to extend the user’s free trial. This is unsafe, but optimized for speed.
Stage 2 - locking down access: The company is growing. They’ve had a couple incidents with junior engineers running bad queries against prod, so they lock down access so only the CTO and a few senior infrastructure engineers have access to the production database and AWS console. This is safer, but slower, and this small group of people becomes a bottleneck for the company.
Stage 3 - building internal tools: Finally, the company actually creates API endpoints for “extend trial” and other common internal tooling needs, and builds a UI so that people in the company can execute these actions. Now, the CTO is no longer a bottleneck, and even non-engineers can do these things through the internal UI. This is both safer and faster than anything they’d done before.
Stage 4 - granular security needs: as the company reaches hundreds of people, the internal tool doesn’t cut it. Everyone in the organization having access to be able to click a button to “extend trial” or “issue refund” becomes untenable. At this point, the company introduces role-based access controls, audit logs, and other controls to make sure only the right people have access to the right internal tools.
This is obviously a safety-enhancing feature, but less obviously, this also improves the speed of how the organization operates. Granular permissions mean that the company feels comfortable adding more sensitive and potentially dangerous functionality to the internal tool, because there are checks in place to make sure these functions are operated properly.
Eventually, companies might even add in more sophisticated organizational security measures, like multi-user approvals (e.g. in order to permanently delete a user, a second person has to approve a deletion request) or time-based access (e.g. allow someone to have read access to a PII-containing database table for only the next 24 hours). Again, this looks like an obvious security measure, but also has the effect of increasing the organization’s speed of execution. Counterintuitively, measures like these let the company trust the widest possible group of employees with sensitive self-serve operations by enforcing least privilege.
As we see above, building internal tools lead to better safety practices while also enhancing the speed of the organization. And investing more deeply in the safety features of those tools continues to pay dividends along both axes.
But startups shouldn’t have to progress through each of the above stages for years at a time. In an ideal world, even a small 50-person startup would benefit from great internal tooling with permissions and approval flows. However the reason this doesn’t happen is that the upfront time spent building these kinds of tools usually trades off against a user-facing product roadmap, and so internal tools remain underinvested in. These situations can get really bad–there are multi-thousand-person companies where some random early engineer is still the only person who knows how to execute that one script.
The solution to this tradeoff is to make it significantly easier to build internal tooling in the first place. Today, there are a few not-mutually-exclusive ways to do this:
- Framework-level: Django is one of the most popular backend frameworks for new startups, and a big part of its value is that Django Admin is bundled in. There are a lot of limitations and it doesn’t scale well, but in practice, the small startups I’ve observed using Django Admin tend to have better internal tooling relative to their stage than others.
- Point solutions: Traditionally, security-enhancing SaaS has been expensive, heavyweight, and sold to large enterprises. But recently, tools like Opal or Doppler have productized specific common patterns in a lightweight way.
- Internal tooling platforms: Our company, Airplane, is a developer platform that lets you take any sensitive operation (e.g. a script that deletes user data) and overlay UI, permissions, audit logs, and approvals with ~0 marginal effort. There are other platforms out there that take other approaches to the same problem, e.g. low-code/no-code tools.
Overall, it’s becoming increasingly untenable for companies to just give everyone on the eng team access to prod and insist that there’s no better solution to let them move fast.
