Well, psychiatrists don't always diagnoses people correctly either, and diagnoses don't really capture what's going on. Neuropsychiatric diagnoses are pragmatic, but often vague and always context-dependent. They do not require people sharing a diagnosis to have disorders with a common etiology.
Take Alice and Bob, both of whom have the requisite six symptoms to get a diagnosis of "ADHD, Inattentive presentation". The thing is, it turns Bob's symptoms are caused by sleep apnea. Since the Diagnostic and Statistical Manual does not require the clinician to check for sleep apnea, or even preclude a diagnosis of ADHD for someone known to have untreated sleep apnea, Bob's diagnosis is technically valid. But where Alice is *neurodivergent*, Bob is not.
Now consider Alice's twin sister Charlene. She has exactly the same symptoms as Alice for exactly the same reasons, but she has a creative job that's highly suitable for people with ADHD and a better family situation, so none of those symptoms cause her any distress or impairment. Because there is no impairment, Charlene *can't* be given a diagnosis of ADHD, even though she is neurodivergent in exactly the same way as Alice.
Now there's Erin. She only has *five* of the required six symptoms for "ADHD, Inattentive presentation", so she can't receive an ADHD diagnosis even though she is probably just as neurodivergent as Alice. Finally there's Frank. Frank only has just *one* symptom. You probably wouldn't consider him neurodivergent -- at least not very. But maybe he is; it could be if we could track down the reason for his symptom it would be in a part of his brain that's very weird.
When we call someone or ourselves "neurodivergent", that's inherently speculative because we don't understand the neurological underpinnings of the things we're talking about yet. "Neurodivergent" doesn't really designate any identifiable set of biological distinctions, it's just a genuinely socially useful label we apply to certain people. Where prior generations would try to beat autistic or ADHD behaviors out of people, now we understand those are things a person can't change, but which don't preclude that person from being valuable and productive. I suspect our readiness to apply the neurodivergent label to ourselves and reluctance to let others do the same is just another example of how we're automatically more generous cutting ourselves slack.