I actually agree, you don't have to have struggled to be able to relate.
I don't get how it's damaging, though. Surely it's a good thing to be able to relate no matter what background you're from? I think what I'm actually looking for is empathy from politicians. You can be the richest man on earth and brought up in wealthy household, and still have empathy. Sunak clearly lacks it because he is so robotic