Is there anyway to keep them? I found one wandering around, rode him to a town, but he was gone when I came back out of the town. Can you do anything with them?

Unless you buy a standard horse, you can't keep them...they wander off.