I know what you mean. I finally killed this guy last night after about twelve reloads. I am playing a level 4 thief type character with limited combat skills. It helps if you have a weapon or item that deals magical damage. I have a ring that shoots fireballs. My final strategy was to cast the fireball at him - run up the stairs and quickly slurp down a potion of invisibility - go back down and pound on him with my fire-sword - he went down after some hacking.
You also have to make sure you dodge his fire spells. I had several potions of invisibility with me to repeat the run and drink process if necessary.
Your strategy would depend on the type of character that you have - but this worked for me.
Linear Mode
