I usually stick with robes. Too many useful force powers are negated by armor.
But, if those powers are not important to your style of play and/or you just like wearing armor, then wear armor by all means (and don't forget the upgrades).
In my experience, the difference between a 29 and 35 Def is almost nil. The tough enemies will hit you regardless of your defense, and the weaker enemies won't last long enough to hurt you very much.
