A free market, which is your point, demands one specific factor - choosing not to participate with no ill effects.
There are 2 obvious exceptions to this - housing and medical care.
You can't choose to not fix a broken leg.
You can't choose to be homeless.
I'll add a third, access to water.
All three things are direct threat to your life.
An intelligent society would guarantee everyone those basic things as the minimal cost associated in their provision would be far eclipsed by the resulting growth in society. Sadly, we don't live in an intelligent society.
I've lived in slum lord housing...it's not fun and it absolutely is a racket jacking prices on the most vulnerable.
And hey, shouldn't the landlord have done a better job at vetting potential tenants? Shouldn't they suffer the consequences of their bad decision? Or should we find a middle ground to help everybody?