Does anyone know how to get rid of the issue where it says you have a follower when you don't?

I never use a follower unless the game gives it to me for quests.

It might have glitched on me when I was doing the Purity quest for Vilkis, is the only time I can think of off hand. I took the quest, we left Jorvaskr and I see Vilkis turn around and head back in. I check my quest log and Purity was shown as completed even though we did not do the head in fire thing to cure him.

My current char is lvl 63 and did not think anything about the Purity thing again until I decided to finally do the Boethiah questline. Tried to get a follower and its telling me I have to get rid of my current one, though I have no clue as to who my current one is.