Yes, but ask yourself why would it happen? In what scenario?
Without agency of goal or need, it is not an emergent behaviour. so it will not evolve this, it would need to be initially instructed to do so.
The question of if people are stupid enough to remove such guardrails is a humanistic question I would argue, and naught to do with AI ultimately.