Context is everything
You make a point though. If you can handle yourself, you're not going to feel unsafe anywhere.
It is a little bit an age thing. When you're young, you're invincible, at least, it's how you feel, but as you age and maybe get some bad experiences, not so strong or fast, then your perspective is going to change. I doubt few people feel that where they lived in their youth is safer in their middle to later years.