ISPs just burn the candle at both ends, charging those who produce content AND those who consume content, for internet access. Sometimes they find ways to burn the candle in the middle, too.
Instead, content producers should GET paid per web request. And the payor should be the person making the web request. ISPs would just skim off the top.
Automated crawling would suddenly become a lot more expensive (good), but website owners wouldn't mind since they get paid for that instead of paying for that (good), users vote with their wallet simply browsing to whatever interests them (good), site owners have strong incentives to produce stuff that people actually want (good), online advertisement becomes less necessary to fund popular content (good), and ISPs still make money.
Of course it will never happen. ISPs would all need to get on board, and without the ability to burn the candle in three places they simply won't.