If they're sitting there looking at their phone, it's one of two things: Either they're goofing off, or they're in a neighborhood with next to no reception and hence when they hit "I've arrived" on their app it pauses and churns endlessly while trying to communicate this fact, blocking them from moving on to the next screen
where they can see and scan what packages they need.
When they're rummaging about that can be one of two things: Either they're grossly overloaded with packages so everything is out of order (which happens a lot during peak season, i.e. Oct/Nov/Dec), or they left everything in the totes rather than spreading them out on their shelves for easy retrieval. Note that these two possibilities are not mutually exclusive. Actually, now that I think about it, it's also possible these are the same drivers fiddling with their phones, and hence they can't see which packages they need by the sort number, so they're rummaging around looking for anything for that address while waiting for the app to load. By contrast, drivers who use the electric vans generally don't have this problem because they've got three levels of shelving so there's plenty of room to get organized. Amazon Flex drivers (the guys who use their own vehicles) may or may not have this problem depending on how big the flex route is.
These are problems that have relatively straightforward solutions, but that would involve hiring more drivers, and software engineers who didn't get their degree out of a box of crackerjacks. While that may result in a better customer experience, it would impose greater shipping costs on amazon, so naturally they won't do that.