What is Wrong with me

so i let this guy a few computers over and behind me borrow a pen since he came over and asked for one. Now i am worrying about wanting to leave, and whether i should wait until he returns my pen or not. It's so stupid why can't i just do what i want, why shoudl i care about him at all. I have been programmed to do what i think is the "nice" or right thing to do. Sometimes thoguh, like this case I think it is rooted in a strong inclination towards cowardness and not wanting to confront people (have to ask for my pen back) - I only justify it by saying i want to be nice to him, or i really don't mind not having my pen. I guess this isnt terrible, but the problem is I SO EASILY JUSTIFY AWAY ALL MY HAPPINESS, I don't value my own happiness enough, I am willing to try or self-deny myself way to easily. I have to know what i want, do what i want, not do harm unto others, and take control of my life.