I completed the Dawnguard questline with the DG side several ingame weeks ago. And when i entered the Castle Volkihar Undercroft, then suddenly Serana was with me as a follower. And i was alone when exploring the castle beforehand and before that. And when i dismissed her she said she would go back to the castle, this isn't right isn't it?

Volkihar Castle is her home, so I would assume when she said "castle", that is where she was going. Followers always go "home" when you dismiss them.

Maybe she is just tagging along in case you need her?

No no, i had already completed the questline, with the Dawnguard side. So Castle Volkihar was empty. And i had no followers with me beforehand

I dismissed her after noticing she tagged along, and i couldn't recruit her again, must be a bug.

Slap her up, swear at her and generally give the slag a good cave of her hide... they always come back.