I’d steer well clear of the city centre.
Our ground is about a mile and a half from the city centre and to be honest, we’re stuck on a retail park with no pubs nearby, apart from a pub/restaurant next to KFC but I think they only serve alcohol with a meal on match day.
Best place to have a beer is Canton, Cowbridge Road East to be precise. Plenty of pubs there, but they will be busy as well.
All depends on how you’re travelling to be honest, coaches usually renedvous on the M4 and get escorted straight to the ground, which is about 10-15 minute walk from the pubs, best ask locals or police for directions. Easiest for you to have a couple of pints though is probably inside the ground.