I'm a drummer (well formerly, would love to have somewhere to practice). I've lived in semi detached houses for years and never once thought it'd be acceptable to set up the kit in the house. If you have a room that purposely soundproofed for it that's alright but a normal party wall is not going to cut the mustard
If you get on with the neighbours, maybe agree to some acceptable practice times? If not then get the council involved, they will act on it - I used to practice in a shed when I lived with my parents, one of the neighbours got sick of it and we received a threatening letter from the council. Made sure to keep practice quieter from then on