Yeah course they will.
I'm happy that Labour has adopted this as I think the definition is fair:
“Antisemitism is a certain perception of Jews, which may be expressed as hatred toward Jews. Rhetorical and physical manifestations of antisemitism are directed toward Jewish or non-Jewish individuals and/or their property, toward Jewish community institutions and religious facilities.”
Boris is lucky that the Conservative Party don't adopt the same standard for all racial and religious groups. For example:
“Islamophobia is a certain perception of Muslims, which may be expressed as hatred toward Muslims. Rhetorical and physical manifestations of Islamophobia are directed toward Muslim or non-Muslim individuals and/or their property, toward Muslim community institutions and religious facilities.”
"I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes."
Would you say that counts as "rhetorical manifestation directed at Muslim individuals...".
Applying your own standards evenly, would you then chuck Boris out of the Party - plus everyone who supported him over that comment? Interesting.