I agree with what you're saying op but what you, and I, want is currently impossible due to technical limitations. Example given, Red Dead Redemption.
Rdr is in my opinion the next best thing in sand box style games and it has what your describe, an engrossing storyline told through in engine cutscenes. There are plenty of real characters and real character development, as well as plot twists and a great ending. Problem is, player choice and freedom is sacrificed to achieve this. You are John Marston period, you can be a dike to the people around you, you can kill them, fight the law or help them and the game notices this stuff and reacts. However when it comes to story you are still stuck with the most important decisions being made for you. Bummed you can't get a hoker? Bummed you can't start a romance? Wanna talk to essential npcs outside of a cutscenes, or kill them? Sorry you can't. Those decisions HAD to be made for you for the sake of the story. Can't have John's character develop into the tragic hero saving his family if you're out [censored] [censored]s.
The reason it has to be like this is simple technical limitations, if you have complete freedom of choice the game has to write a dynamic and engrossing storyline on the fly, otherwise you're back to being on rails and having major character decisions made for you, there just isn't an engine capable of writing a story around your decisions or hardware capable of running it.
Even games like those from Bioware with their detailed stories only have pretend choice. All of your character choice boils down to a generic good choice, generic bad choice, then a don't care or sarcastic choice where every possibility has to be predetermined and thus limited in scope to save dev time and investment as well as meet hardware capabilities. Bioware games are essentially "Choose your own adventure" books with a pretty Gui.
So the storyline you want and the freedom we expect won't happen until significant advances are made in both software and hardware.
^This.
Here's your challenge: write a single scene between the player and one of the main quest characters that is engaging, carries dramatic tension, and has convincing dialogue. Here's a small test sample of player characters that the dialogue must be written for:
1. A player who wants to play the part of a noble warrior (the Nord equivalent of a knight) who fully embraces his destiny as a Dragonborn. He's big and tough and wants to "save the world". (Here's your "straight" character.) How does your quest character address them? What does he or she say?
2. A player who wants to play the part of a mage who is physically weak, craves magickal power, has rarely ever stepped outdoors and is skeptical of her destiny as a Dragonborn. She is indifferent to the plights of the many and only concerned about accumulating knowledge and furthering her own development. She'd rather read a book than delve a dungeon.
3. A sleazy Bosmer cutpurse who has spent his entire life stealing from people he considers trusting fools. He thinks the whole Dragonborn thing is a load of crap, but he'll use his "destiny" to score some big gold from superstitious idiots who think he is some sort of hero. He couldn't care less who lives or dies as long as it isn't him.
4. A high-minded Altmer who has decided to spurn civilization and live in the forests as a druid. This character doesn't believe that man or mer have any more right to existence than the dragons. He is indifferent to civilization and views all things dispassionately from the perspective of natural balance and harmony. He has no interest in participating in the affairs of men and cares nothing for politics, seeing all men as equally foolish.
5. A simple Breton alchemist who wants to tend her garden and heal the sick. She's much too humble to believe that she could be the Dragonborn, and while she might go along with it, she always thinks that it's a mistake, that somehow the gods have chosen the wrong person. She spends most of her time trying to convince people that they have the wrong person.
6. A bloodthirsty Orc barbarian whose sole purpose in life is to prove that he is the most powerful warrior that ever lived. He cares only for his own glory and nothing for others. Being the Dragonborn simply confirms what he already believes. In fact, there's no reason why
he shouldn't be the high king of Skyrim since he's obviously been chosen by the gods.
Now, write a dialogue between this character (through dialogue options) and the main quest character they are conversing with. The dialogue options have to suit the player's character and the NPC's dialogue has to apply equally well to each example character. (All of them are equally valid RP options, so you can't exclude any of them.)
Now write a story arc that accommodates each of these player's RP preferences. Each quest in the story has to be able to appeal to all of these (and innumerable more) types. Has the plot changed at all based on the character?
Now try another experiment: pick any
one of these characters and write dialogue and a story arc that applies to this character alone. The player has no choice but to play the part of the character you have selected. Was this easier to do? Was the writing better? Was the story more engaging?
I encourage everyone reading this to give it a go.
Writing for sandbox games is not the same as writing a linear script. Each dialogue "chunk" has to stand alone and appeal to every possible play-style. Every quest has to appeal to every possible character type.
Unless you provide branching options. If you take a single narrative for a pre-made linear game and allow for even
one other character type written with the same degree of depth and conviction, you have effectively
doubled the length of the script. You have also doubled the cost of writing and voice acting and added additional testing. I guarantee you that no two lines of dialogue will be the same if you are writing convincing dialogue that effectively captures the player's choices. Every time you add another character type
and expect to achieve the same high quality as the first, you have added the full length of the script again, along with all of the different branching quest options, etc. And that doesn't even include cutscenes, which are almost anathema to a sandbox RPG.
The only way to reduce the workload is to remove choices from the player. The fewer choices the player has, the easier it will be to write, and the less it will be a sandbox RPG. The developers pick a point somewhere on the line between tightly scripted linear narrative and open, free-form sandbox and write to that point. The closer you are the first pole, the better the writing is and the fewer choices you have. The farther to the latter pole, the worse the writing is and the more choices you have. That doesn't mean sandbox games have to have bad writing, and that some aren't better written than others, only that they need to spend exponentially more time and money writing and producing it if they hope to provide the same quality as the linear narrative.