Yea I've also had the same experience from signing (most of) my reviews too. There was a borderline case recently were I considered not signing. But I'm glad I did since I think was ultimately fair to the authors for contextualising where my criticism was coming from.