On-prem AI is an architecture decision

JPMorganChase has picked SambaNova to run production AI inference on-premises — installing SN40 and SN50 systems to keep the workload behind the bank's own walls, with full control over the data and an auditable trail — announced the same week SambaNova closed the first $1 billion of a Series F at an $11 billion valuation. For a bank, this is a defensible, probably correct call. I can't verify the workload details, latency, or cost from the announcement, and it doesn't matter for the point, because the point is what a thousand enterprise architecture reviews will take from this headline: "on-prem equals secure." That inference is wrong, and it is expensive.

Physical location is not a security control. It feels like one — the servers are in your building, behind your badge readers, so surely the data is safer. But every actual security property is orthogonal to where the box sits. Workload isolation, identity and access management, key management, observability, patch cadence, model governance — an on-prem deployment that gets these wrong is less secure than a well-run cloud deployment that gets them right, and it has the added charm of being your sole responsibility. Moving a poorly-isolated agent from someone else's data center to yours changes the postal address of the incident, nothing more.

What on-prem genuinely buys is different, narrower, and worth naming precisely, because only named benefits justify the cost. It buys control over the data path — you can prove, to a regulator, that customer data never crossed a boundary, which for a bank is a real and sometimes mandatory property. It buys latency and capacity control — no shared-tenant contention, no surprise throttling during someone else's launch week. And it buys a certain kind of auditability — an evidence trail you own end to end. Each of those is a specific requirement met by a specific property. None of them is "security" as a vibe, and none of them arrives automatically from the location; you still have to build the isolation, the IAM, the key management, on your own now-heavier shoulders.

The trap is the default. At work, the bank and insurer architecture reviews I sit in increasingly reach for on-prem reflexively — it feels conservative, it feels safe, and nobody gets fired for keeping data in the building. But when I ask which specific threat, latency target, audit requirement, or failure mode the on-prem choice addresses, the honest answer is often "it just seems safer." That is not an architecture decision. It is an anxiety decision, and it transfers a mountain of reliability and lifecycle responsibility — uptime, patching, hardware refresh, capacity planning, the whole operational burden the cloud was quietly carrying — onto a team that has not budgeted for it, in exchange for a security benefit that was never location-dependent.

My rule for the on-prem conversation: the location is a conclusion, not a premise. Start from the constraints — this data legally cannot cross this boundary; this workflow needs sub-X latency guaranteed; this regulator requires an owned audit trail — and let the deployment model fall out. If the constraints point on-prem, as they plausibly do for JPMorgan's most sensitive workloads, deploy on-prem and build every control that location doesn't provide. If the constraints don't point there, on-prem is just cloud with worse economics and your pager number attached.

Steal this decision record: for any on-prem AI proposal, one page, three columns — the constraint (named threat, latency number, or audit rule), the architecture property that satisfies it, and the owner accountable for operating that property. If a row's constraint column says "feels safer," delete the row. If the page is empty, you don't have an on-prem requirement; you have an on-prem habit.

Putting the model in your building doesn't secure it — it just makes every control you forgot to build your problem instead of someone else's. Choose location for constraints, then earn the security separately.