Speaking up when someone's mistreating you is not "being mean" or "being a jerk." Declaring your boundaries and how someone overstepped them is 100% fine. And if they call you mean in response, that's victim-blaming and a childish, abusive mentality on their part.