"If OpenStack isn't an alternative to VMware, then what the hell is it? "

OpenStack is basically an operating system for large clusters. It exposes it's system api's as REST interfaces you can call over http.
It's components are:
Nova: handles compute resources, such as VM's (work is underway to handle bare-metal provisioning, too), These can be provided by many hypervisors, such as XenServer, KVM, HyperV and VMWare, or containers like LXC. Nova handles resource allocation across the cluster of hosts. When you ask for a VM/container of a certain compute capacity, it finds a host with available resources and sets up a VM. Think of it like Linux's process scheduler and process management functions.

Glance: handles metadata about VM images, and acts as a 'pump' to schlep images to/from storage.

Swift: object store. Someone likened it to a key/value store. Similar, but it is designed to handle large data values (whole files, including multi GB server images) in a fault tolerant fashion. (it replicates your data 3 times on separate hardware in the cluster.) ala S3.

Keystone: identity management. Handles user authentication, multi-user accounts, and information on what users can do. Think PAM/kerberos.

Cinder: Block storage. Handles allocating block (ie iSCSI) devices you can mount filesystems on.

Quantum: Handles virtualized networks between VM's. Basically sets up private tunnels between VM's

Plus web admin gui's for above (Horizon), and all of the admin tools for the operators of the cluster(s) to check who's using what, etc.

Basically, if you you need, say, 3 webheads running CentOS with 2gb of ram, a DB server with 16gb and an attached 500gb storage array, a Windows server, and a private network between those, Openstack is what lets you make a REST call (or click a few buttons in a web gui that then makes the call) and, if you have authorization to request that, and resources are available, it will give you that.