I love L4D, I agree that it's light on content and there are quite a few problems, but I've gotten about 300 hours out of a game that cost me $40ish. That's a bargain in my book, regardless of the issues.
A bunch of my friends and I set up a Steam group and got the dedicated server package installed on one of our Linux servers over Christmas.
If you play L4D a lot, this really is worth doing IMO (you can rent an 8 slot L4D game server for $10-15 a month, between a few people this is nothing) - we have a server geographically close to the majority of us on a decent tube, and we generally get around 30-40ms pings with 60ms spikes. It's very smooth, and is not difficult to setup (took me about 10-15 minutes one Sunday).
The best way we have found to connect to this server via the lobby system consistently is to set steam_group_exclusive in the cfg to our group only. This stops people not in your group from starting a lobby on it. Then when we want to play, we bring up the developer console after starting a lobby and use "sv_cheats 1; ms_force_dedicated_server IP:port". This then connects the lobby to our server every time when we click the Search menu item.
I agree Valve need to do something with this mess and it's a bit annoying to have to work around it like this, but this works nicely. You can of course alias the sv_cheats / ms_force_dedicated_server command to make this even easier.
You can apparently achieve similar results with setting an sv_search_key in the cfg and your group's clients, but this has given us problems so we use the above workaround instead. There is also the openserverbrowser command that gives you a browser window of available / recent / favourite L4D servers to connect to, similar to TF2, CS etc.
Given that a) you post on slashdot so I assume you're technically competent enough to install hlds and edit a config file, b) you obviously enjoy the game and c) your enjoyment is only being ruined by bad servers, I highly recommend you look into picking up your own server, sharing the cost with a few people if necessary. I can gaurentee you it will make the game a lot more enjoyable.