The Power of No

I had to say "no" to someone today.  Well, not totally accurate, I choose to say "no" to someone today.  Then I called my husband who said he was glad I did so, because that was what I felt was right and that is "the power of no."  So now I feel guilty; but not so guilty I want to say yes.  I cannot recall the last time I said no when I was asked to participate in a ministry.  As a parent I said no and had no problem.  At work, I can say no.
A few weeks ago I came across a printable page of No's on Pinterest and printed it out for our parish secretary to use on her preteen and three year old sons.  (Of course I can't find it now to show you) 
Why is it hard to say no to others even when you have good reasons and it is the best thing for you?  I think I am old enough to learn that no is acceptable, especially when it brings me peace. 
Still, it is a bit more difficult than I thought.
I think I need some clarification on the power of no.