Some people erroneously believe that old debt on their credit report is bad, says Ulzheimer. The minute they get their home or car paid off, they're on the phone trying to get it removed from their credit report, he says.

Negative items are bad for your score, and most of them will disappear from your report after seven years. However, "arguing to get old accounts off your credit report just because they're paid is a bad idea," he says.

Good debt -- debt that you've handled well and paid as agreed -- is good for your credit. The longer your history of good debt is, the better it is for your score.

One of the ways to improve your credit score: Leave old debt and good accounts on as long as possible, says Ulzheimer. This is also a good reason not to close old accounts where you've had a solid repayment record.

Trying to get rid of old good debt is "like making straight A's in high school and trying to expunge the record 20 years later," Ulzheimer says. "You never want that stuff to come off your history."