Bummer. I was going to say can you get your fee lowered then? I guess they know when they got you between a rock and a hard place. In your situation I See why you want to hang on. Just don't pay that fee forever.
A mortgage or a car payment will do wonders for your score if you make your payments timely. Good Luck!
I am with you poundwise - my dh is really into only carrying 1 card, 1 card ONLY - yet he has been stranded with a card not working due to overzealous fraud protection. I always carry 2 - 1 for backup. Doesn't hurt the FICO I am sure.