Does difficulty setting affect how much XP you get?

Well, indirectly and partially. The higher the difficulty setting, the more damage you take from his attacks, while the less you deal. That's it.

But if I'm right, this could be a help developing certain skills, based on the more amount of damage you suffer / deal compared to lower settings.

Just by pure logic, probably Heavy / Light Armor and One / Two-Handed will increase faster in Expert on Master, just because the enemy hits you often and you have to perform way more successful attack time to get rid of him. But you can level up Restoration faster too, just because you have to restore health more.

My point is, if you need X point damage taken / damage dealt to reach next level in that skill, you'd get that more in that circumstances. This affects mostly combat skills like I mentioned, but as far as crafting skills, not likely. In those cases, you need Y piece of to proceed further, and price matters. So if you enchant something with a more valuable enchantment (thus, making it more expensive) brings more XP. From the logic of difficulty settings (just the damage multipliers change, nothing else) switching to a Expert or Master doesn't change a single thing.

If my logic wrong somewhere, just let me correct.

In short, no. Unless your counting a bunch of other factors in which this question is invalid.

Higher difficulty setting generally makes your attack skills to up faster, just because you need to hit your opponents more before they die.

Higher difficulty setting generally makes your restoration to go up faster, because you heal yourself more.

Lower difficulty setting makes it easier and safer to grind your armor and block skills, because you get less damage and still get the same XP each time you get hit.

Changing the difficulty setting only changes the damage multiplier. Everything else remains the same. If you hit someone once with your sword, you get same experience in Novice, Adept and Master. Your opponent just get more or less damage. If you get hit from the same opponent using the same attack, you get same armor experience in Novice, Adept and Master. You just get damaged less or more, depending on the difficulty setting.

I thought armor/block skills leveled depending on the damage you take?

Every time (it seems) I would become sub-orbital from a giant stomp, I received a message saying my light/heavy armor skill had increased, even if I wasn't exactly close to leveling those skills otherwise.

It works like this. Let's say a giant does 200 damage per hit. On adept, each time you get hit you get 200 XP points, and get damaged 200 health, before armor reduction. On Master, each time you get hit you still get 200 XP points, but now you get damaged 400 health, before armor reduction. On Novice, you still get 200 XP points, but you only get damaged 100 health. You can tolerate getting hit more times in Novice before you have to heal yourself or run away. So, if you are grinding your armor skill, you would put on your best armor, change difficulty setting to Novice, and get repeatedly hit by a giant.

Oooooh. I'm pickin' up what you're puttin' down.