You're definitely not atypical.
tbh. I don't understand why there is an option to load remote content at all. If you want to make a nice html mail there is no problem attaching images.
In my opinion, the only reason the remote content support even exist is to track the receipient, even if it's only a simple did-read notification, ok, granted, some Mail-apps seem to use a regular webviewer for the email, which automatically allow all remote content and other attack vectors normally not supported in a mail with no option to turn them off.
How I miss the days when people actually sent proper text emails and follow RFC 1855,
and while I'm in complaining mode, what's up with the mail-clients that try to turn emails into messaging? ok, I guess that's one way to solve "Hey, let's quote all previous replies from the beginning of time in the mail" but still.