It depends on the character for me too actually. I tend to feel especially guilty about dumping Minsc. Jaheira and Imoen, since I almost always had them in my BG1 parties, so it does sort of feel like abandoning family.
(Not that I want to overlap with another thread
)
I also feel guilt over Mazzy, and Aerie. Mazzy because she lost Patrick and her friends, so often I'll make room for her in my party to at least go through the Shadow Dungeon with me so that she can find closure at the end. I feel bad about Aerie, because, to me, it seems obvious that Quayle (however obnoxious he was in BG1

) is hoping to entrust you with taking her out into the world. Plus, I like her spell combos.
I even feel some guilt about booting modded NPCs out
Though I successfully managed to kick out both Yasreana and Kivan this time round so I could end up with my planned for party of:
PC (neutral half-elf bard)
Edwin (I'd never played him and I DL'ed the romance mod

)
Fade ( a wonderful alternative for a thief, very refreshing and chatty)
Saerileth (very nicely done paladin! and she's not bigoted like Keldorn)
Anomen (for his conversations with Edwin, and to tank... ROFL!

)
Haer'Dalis (I'd never before kept him in a party)
I very deliberately stuck to a party combo I had never tried before, and I'm having a blast

I'm very pleased I actually managed to overcome the guilt factor. The banter with this party is great, and I like the challenge of being forced to try out different strategies.
It is good to hear I'm not the only person to have trouble with this kind of thing.
@Von Dondu, I used your strategy of creating a story behind every NPC, which made it really interesting. Plus, it made it much easier to kick NPCs out.... except I still had trouble with Minsc... *sniff*