I think it is because some people (whether you like it or not) are aware that a decent chunk of people turn up, fold their arms, behave anti-socially and contribute nothing to supporting the team. Their priority is arriving late, spending money on crap food and beer and leaving early.
And although I agree it is their right, it is also fair game for derision in my eyes. That's how rights work.