You're conflating 'nobody' being offended with 'you' not being offended. It's not up to you or I to determine what is or isn't offensive to a particular person unless we're the subject of the prejudice.
I think there is such a thing as over sensitivity. Of course there is. But Vardy's use of 'Jap' was undeniably racist and it's right that he's apologised. It's not the most racist thing ever, but insidious casual racism should be challenge just as much as any other form.