Over farming means there aren't enough nutrients in the soil
Fertilizer means that there are. You're confusing reductions in soil carbon with reductions in nutrient availability.
Reduction in soil carbon isn't good, but it's not a nutrient. It does a number of things - reduces water retention (so you have to irrigate during drought), can potentially reduce aeration (depends), leads to less soil biodiversity (can be good or bad, but in general usually more bad than good), and less nutrient *retention* - but that doesn't mean "less nutrients", it means you have to apply more often and there's more runoff. But the plants still grow great.
Or to put it another way, you know hydroponics? Famously a high-yield process? Most hydroponic media are inert. No soil carbon. Plants do not need soil carbon. They need sufficient water at a proper steadiness, sufficient oxygen, and sufficient nutrients.
Indeed, excess nutrients can sometimes damage quality. Nitrogen for example encourages rapid growth, but this can sometimes come at the cost of product quality. Excess water also can tend to "water down" fruits. Again, just the opposite problem to what you think.
And oranges are just a waste of money now. They're just balls of cellulose.
Sadly, the decline in citrus quality in the US isn't about soil, it's about HLB
But for most fruits...
so much of it tastes like garbage
Your main enemy is the overreliance on productivity as the metric rather than quality. For example, tomatoes. It's commonly said that the reason store tomatoes are bad are because they're picked green. Well... kind of. If they're picked light green, with no reddening beginning, yes, that will yield a very bad tomato. But blinded consumer acceptance studies consistently show that if the tomatoes are picked after they've started blushing, then ripened, consumers can't tell the difference in taste vs. vine ripened. I've done the test myself back in hort school, and verified it. The problem isn't when they're picked, it's the cultivar. They're optimized for even ripening, longevity (thick skins), and truly extreme yields. You can grow the same cultivar in your garden at home and give it all of the nurture and love you want... you'll still get a bland tomato out of it. There are things you can do to improve taste, like increasing eC shortly before harvest (basically "salting the tomato while it's still on the vine), but you're fighting an uphill battle.
(Also, if they refrigerate the tomatoes, that's a real quality killer :P
Oh, and just in case this was about bananas (you might have just meant it generally):
they grow the fruit to be too large so the tree cannot get the flavor components to the fruit.
Bananas don't grow on trees. They're more like a giant grass (or more specifically, like a ginger - they're in the same order of monocots as ginger, same clade of monocots as grasses). Not woody. Swing a machete at one and it'll go straight through. The stems are very juicy (the juice tastes like banana skin ;) ).