Let me first take Kipis job
(lets hope he doesn't mind.)
Welcome to the Fallout forums, and welcome to Gamebanshee.com Jaibo. Stay long... And prosper.
Now about your question:
You are probaly too evil for him. He is a good NPC, and he won't follow you if your karma gets below... Something evil...
Thats what happens to me usually. Are your other party members leaving you as well?