I want to try to clearly differentiate between these two. I do not intend to attack IaaS as a concept itself, but rather the ignorant usage of the implementation of IaaS at the major hyperscale providers.
Contents
Hyperscale IaaS
Principles forced upon customers of hyperscale IaaS include several key things that are purposely built to lower availability/quality standards for cost purposes.
- Data centers
- Servers
- Storage systems
Customers are expected to work around these limitations by distributing workloads in different data centers(“availability zones”) and regions(“groups of data centers in different regions”). Hyperscale SLAs are often specific about not accepting blame for infrastructure failures unless certain thresholds are met(such as an entire region failing for example).
Conceptually I have no problem with this at all if you are operating at hyperscale as I mentioned fifteen years ago. It makes a ton of sense. You have tons of resources that are distributed already, you build high availability into the system at a higher point in the stack. I have no problem with the SLAs, and sub par low level infrastructure design, something I termed as “built to fail” in my original post.
On Prem IaaS
First off, I do think you should be at a certain (subjective) scale that is high enough node count to justify the overhead of IaaS on prem. Maybe that is 1,000 physical systems, or 2,000, or more…or less. Obviously it’s up to you. For me personally I would not consider it likely before at least 1,000 physical systems(a scale I never expect to honestly get to in my career, highest system count I had was two decades ago with around 450 pieces of physical equipment, in 2025 that is well under 100).
If you are on prem, you have the freedom to design your infrastructure availability at every level.
Physical is critical
The physical part is critical. You can run 1,000 virtual machines on just a few physical servers without an issue(depending on workload). So we’re really talking about the likelihood of at least FIVE to TEN THOUSAND virtual machines assuming you are doing virtual machines that is, maybe the bulk of your workload is container based, in which case the number of containers would be a a radically different threshold than VMs, maybe 10-50X more containers required before you consider IaaS on prem. You can run 2,000 containers on just one or two physical servers without blinking.
On Prem IaaS solutions
I can only speak of these solutions from a conceptual standpoint as I have never even considered deploying them(nor do I think I would ever, for the scale reason above). Nor have I had any in depth discussions with anyone who does have experience deploying either of these solutions.
OpenStack
OpenStack is a sophisticated integrated platform to provide many IaaS functions for a customer to self host. Initially popularized by Rackspace who was one of the creators of the platform, along with NASA. Openstack saw a large initial rush of enthusiasm at the time as many players went to adopt it for either their own internal use cases or offering public cloud services themselves. HP for example briefly launched an OpenStack-based cloud in 2014, only to shutter it by 2016(I ripped on them in my blog post, thought the whole idea was stupid from the beginning I guess they agreed in the end). I believe they release one or two major versions per year and at one point I think they introduced a “long term support” version of Openstack.
I recall the telecom sector, and AT&T in particular being staunch advocates and early adopters of the OpenStack platform. After a few years things seemed to quiet down, and years later still was quite surprised that AT&T had dropped off the list of OpenStack sponsors entirely and seemed to have shifted a lot of their workloads to VMware, which later still made news when AT&T sued Broadcom over licensing/support issues in September 2024. In this lawsuit AT&T said they had roughly 8,600 physical serers running VMware, and about 75,000 VMs. That is a pretty huge footprint.
Openstack enthusiasm waned significantly in the years following it’s release, even Rackspace pulled back from it for a time(I think anyway), before later re-committing again to it. However since Broadcom acquired VMware and started swinging it’s wrecking ball across the VMware customer base, enthusiasm has trended upwards quite a bit again towards Openstack as an alternative to VMware Cloud Foundation(VCF).
Background
OpenStack first’s release apparently was October 21, 2010, ironically barely two weeks after my own blog post attacking IaaS which was October 6, 2010. There was a lot of hype around it at the time, hell even I got a bit excited, especially following the launch of VMware vSphere 5.0 in August 2011 where they tried to introduce a “vRAM tax“, which they eventually backed down from. That was about the same time I had moved out of public cloud into a VMware-based utility computing stack. I do specifically recall deploying my initial stack on ESX(not ESXi) 4.1, along with vCenter 5.0 in early 2012. This meant that I was never under the influence of the “vRAM tax” licensing changes. I never used vSphere 5.0, or 5.1. I went straight to 5.5 later.
During the “vRAM tax” fiasco is when I became more optimistic at the possibility of using OpenStack in the future for my own stuff. All that needed to happen was some organization to simplify things for new customers, which I assumed may not take much time, maybe one or two years. Reality soon set in though, and the “vRAM tax” fiasco went away entirely. That reality was that Openstack wasn’t getting any easier. The OpenStack solution I had hoped/expected was on the horizon fourteen years ago never materialized.
VMware Cloud Foundation (VCF)
VMware Cloud Foundation is an enterprise grade private cloud platform. I don’t believe it offers all of the same functions as OpenStack (I don’t think VCF has an object storage solution for example), but it does offer a lot more ease of use benefits backed by long term enterprise grade support(well, at least until Broadcom took over I think). VCF is built on top of the near bulletproof ESXi hypervisor and vCenter management platform, along with VSAN hyperconverged storage and NSX distributed networking, as well as some monitoring tools. They include a lot of other things on top of that and sell it all as a bundle of products which is licensed per physical CPU core per year as a software subscription.
Background
I had been a very loyal fan of VMware for about twenty five years, really was holding out hope really until mid 2025 that Broadcom would do the right thing (after backtracking on certain licensing changes in 2024). I was looking at what I thought was the real possibility I was going to be able to continue using VMware despite the massive price hikes, in large part by slashing the core counts on newer hardware. Even with VMware vSphere Foundation(VVF) being a lot more expensive than vSphere Enterprise Plus(which they temporarily re-introduced the SKU for in 2024), I found a way to make the costs work well.(assuming I could buy a 3 to 5 year license). But then I learned that VVF was likely going away entirely within months, VMware rep was unable to quote anything more than one year, and said further price hikes were coming November 2025 and to ask again then(new fiscal year for them). It was at that point that I felt I had to accept that my two decade plus stint with VMware was coming to an end at least from a professional standpoint. I still will continue to run ESXi 6.7 on some of my personal servers for some time to come, and VMware Workstation at home.
But I have never had the slightest bit of interest in Cloud Foundation, nor really any of their other products outside of the hypervisor, vCenter, and VMware workstation. I never cared about NSX, VSAN, and the 50-70+ other software products VMware was trying to up sell to customers for so many years. You would have had to pry the hypervisor from my cold dead hands for a long time. That product was one of the most solid pieces of software I have ever used in my life. With the caveat that I ran on super conservative configurations far behind bleeding edge. I ran ESX 4.1 till it was well past EOL, same for 5.5, and even 6.5, and even 6.7. The last version of vSphere I was super excited about was version 4.1 released in 2010. Just look at the laundry list of core features in the release notes, many of these features were introduced in 4.0, though 4.1 obviously included a bunch of fixes. IMO every version since has more or less been a snooze fest for me, only real reason to upgrade is a few fixes, newer hardware support, and security updates. Sad that in 2025, there are several competing hypervisors that still cannot properly compete with vSphere 4.1’s full feature set, ease of use, and performance metrics fifteen years later. vSphere 4 even supported up to 64 CPU cores and 1TB of memory, configurations I don’t even have today. I think I’m especially going to miss VMFS which has worked so well for me over the years.
Hyperscale Public Cloud solutions on prem
I’m unsure how common or how scalable these systems are(as in how large do some customers deploy these). But there are solutions like Microsoft Azure Local, Amazon AWS Outposts, and Oracle Compute Cloud@Customer.