Vicars always wearing dog collars, even on game shows. You're not giving any sermons, you're not baptising someone, but no, you just have to let people know that you're better than they are (cos that's what it's for).
It's not like surgeons come on in a gown or firemen have their uniform on.