That's strange. You're not suffering from a low reputation or anything, are you? I really have no idea what's going on. If I understand correctly, if you initiate conversation (so I'm assuming you have the NPC Project installed) he leaves your party. You might go over to the
NPC Project Forums and post a query about it. They may have more answers than we non-designers.