This happened to me too. I spent 11 hours of gameplay unable to unlock shouts. Pressing R just gave me the message "You need dragon souls to unlock shouts". Well, I had 2 dragon souls but it simply wouldn't work.
After spending forever pressing every key hundreds of times, killing more dragons just incase I had an insufficient number of souls to unlock shouts and generally searching online/trying random things in-game, mashing the keyboard for ten minutes eventually sorted the problem. It turns out that I needed to press Page Down, infact it turns out that I needed to press it twice. Then and only then the souls could be spent and souls could be unlocked.
Took about 11 hours of gameplay to figure this bizarre method out.
Oh. Thanks! I'm level 25 now, and I've collected many shouts from walls (I think I have about 8 in a list now), and my shouts page tells me I have 6 dragon souls. But to date I've only been able to use Unrelenting Force, and the Whirlwind Sprint - which the game "unlocks" for you as early quest things.
I was starting to worry that I'd missed some huge obvious explanation somewhere as to how to unlock these other shouts - like only certain souls worked to unlock certain shouts, or something.
Now I've read this thread, maybe I'll be able to suddenly become a bit of a beast. I know I've got frost breath, fire breath, and some stormcall thing which all sound quite mean, in my list of locked shouts.
Seems a bit crazy that none of this has been made either easy, or obvious, anywhere - not helped by how clunky it is just to navigate through the awful UI to reach your shouts in the first place!