Tech companies are driving unmeetable (for now) demand for new office space. As a result, lease rates are about 56% higher (last time I checked) per sq ft for offices than for Peninsula-area rental housing. You can see why financiers and developers prefer to build offices rather than housing.
It's fashionable in some circles to blame the jobs/housing imbalance on zoning restrictions, but that doesn't seem to be consistent with the ground truth. There are many millions of square feet of new development going on right now, and in many cases these are mixed-use projects with the freedom to build lots more housing, but the mix is overwhelmingly dominated by offices because of the difference in rates of return.
Construction costs are also a factor. Land is expensive and in short supply, of course, but high-rise construction is also expensive. High-rise flats are about 2.8 times as expensive as row houses for equivalent units, and therefore likely to be expensive to lease and not as likely to be profitable for the developers. They're surprisingly candid about this problem; for example, see https://ancillary-proxy.atarimworker.io?url=https%3A%2F%2Fwww.mercurynews.com%2F2018%2F05%2F01%2Fconstruction-costs-could-limit-where-san-jose-homes-are-built-google-adobe-diridon%2F "Construction expenses have pressured developers severely enough that new market-rate apartments are profitable in no more than two districts in San Jose... Even worse, downtown San Jose — seen as a cornerstone of the city’s economy — is one of the sections where development of new housing is unlikely to produce profits for developers..."
Transportation is arguably more important than housing, but it's received little attention so far. The road network is saturated now at enough times and places that additional housing wouldn't always be viable in those places. The population distribution makes rail systems unusable in much of the Peninsula.
If the occasional Marxist analysis doesn't bother you, or if you can put it aside temporarily, chapters 5 through 7 of Richard Walker's "Pictures of a Gone City" offer a tremendous amount of useful data on the situation.
Silicon Valley arose in part because of conscious decisions to distribute strategic industries geographically. (See Margaret O'Mara's "Cities of Knowledge" for a good synopsis.) Silicon Valley is hyper-expensive, earthquake-prone, water-poor, transportation-poor, and at risk from sea-level rise. Learning from past experience and distributing some of the growth elsewhere might be a smart move.