I am trying to find Ahtar the Jailor who is supposed to be somewhere in Castle Dour. I have been all through the jail, but no luck.

Any ideas?

Finally found him in Jala's house in Solitude but he won't talk to me. He only tells me to get out or he will call the guard on me.

He is supposed to give a quest and then become available as a follower.

Any help?

http://www.uesp.net/wiki/Skyrim:Kill_the_Bandit_Leader

Read the Notes section, sounds like you may be experiencing the issue described in the first sentence.