I do it too... I guess I just assumed that everyone did it
[although with me it might be out of sheer laziness
]
I figure I pretty much only wear them after work, and as long as they don't get muddy (wet backyard+playing with the dog) or wet (walks in the rain or snow) or I get sweaty (nope, not in freezing winter) there's no harm in wearing them two or three times