Wandering around northern Solstheim I ran into two pirates having an argument. One of them was talking about a hidden treasure and the other one wanted to bring him back but he refused. So they got into a fight. After it was over I checked out the loser to see if he had a treasure map on him, but he didn't. Then I tried talking with the winner of the fight, but she didn't have any dialogue options. Next I checked her inventory to see if she had any map on her, but again no luck.

Is this just color? Or is there some quest supposed to start that I missed out on?

One of them should have the book that starts the Deathbrand quest on them (it's the map to find Deathbrand's treasure that they were talking about).

Ok, that explains it, I already read that book, so I already have that quest objective.