Do you have an office sweater? A sweater you leave at the job to wear whenever the A/C is too high in the summer or when it's cold during the winter. Is this sweater ugly? Do you not care?
I've worked really hard over the last few years to overhaul my wardrobe and to put thought into my outfits everyday (well almost everyday). I'd rather be cold than wear a sweater that doesn't go with my outfit.