There is almost no infrastructure required for this; It's a single building about the size of a small service center or car wash. In there is all the charging infrastructure you need, too.
The problems with battery swap are nothing related to what you mentioned. The real hurdles to mass adoption are questions about battery ownership, standardization, and to a lesser extend vehicle integration.
Since an EV is useless without a battery, laws in some countries (like the US) make selling an EV without a battery basically impossible. Even if you lease the battery separately. It would be akin to selling a new car without an engine; since it's not in a functional state, it can't be sold as a road-worthy vehicle. Countries like China don't have this hurdle so manufacturers like Nio and BYD have been rolling out that kind of business model - Nio alone has thousands of swap stations.
Standardization also generally means you are vendor locked. The battery pack needs to be physically and electrically compatible with your vehicle. It's not impossible but it's very unlikely that the industry will develop such standards especially when it puts hard constraints on the shape and size of the whole vehicle that needs to be build around the pack. Again, not a problem for single manufacturers but if you buy a Nio you will only ever be able to use battery swapping at Nio locations.
Finally integration; To reduce costs some manufacturers (read: Tesla) tightly integrate the battery pack into the structure of the vehicle making it impossible swap without a multi-day ordeal. Imagine trying to battery swap the Model Y where the top of the battery casing has the front seats and center console bolted to it, because it's also the floor of the cabin.
> With that factored in, what are you paying per swap? 3-4x what normal recharging would cost?
From what I understand, Nio charges a flat fee of 180 renminbi, or about US$25, per swap. I'm aware of some schemes that also charge based on battery SoC but I don't think Nio specifically uses that business model. Remember; under this system you do not own the battery, you're basically renting it. This also makes the car cheaper up front.
> Besides, battery tech is improving daily. Increasingly seeing mentions of 5-6min recharge times.
I have not seen any credible demonstrations of "5-6 min recharge times" though I've seen plenty of sensationalist tech news headlines about the possibility of such... relatedly, a battery swap does take about 5 minutes.
And the fact that battery tech is constantly improving is an argument in favor of the swap model, since you can potentially upgrade to a better battery automatically at no cost or inconvenience.
=Smidge=