It actually is that hard, sometimes.
Often, jobs have a culture which have become structured so that you must be responsive, if not 24/7, then at the least during your work hours, to IMs. Step away from your desk for 30m to eat lunch or whatever? People are going to start calling you in many of these (IMO toxic) environments.
And frankly, it's required for some jobs (like in support roles). You've got to be available and IM is used for coordinating on the ground.
I've told people I am simply not available on IM platforms on my phone, I won't even install them if I can avoid it. This has caused some backlash, admittedly, but it's sanity worth preserving. If it's important, think it out a bit more and send me an email.
There's no good solution for this, unfortunately, particularly when everyone's set on using Slack for everything.