Scenes are the basic building blocks of stories. Like LEGOs, we snap them together to build bigger, more interesting structures. A scene is a focused event that usually takes place in a single location. Designing effective scenes is key to good stories, both in fiction and in RPGs.
In his book "Scene & Structure" Jack Bickham describes a scene as having three elements.
1) Statement of Goal
2) Introduction and development of conflict
3) Tactical disaster
The characters should have a clear goal. They should be moving towards something. This is specifically a short-term goal for the scene--the characters should also have long term goals. Every scene should be a stepping stone to reaching those long-term goals, but we're strictly interested in the short-term here. The character's want to convince the Duke to fund their expedition, or they want to find a way to sneak into the military base undetected. The goal is what gives the scene meaning. No goal, no point.
Obviously, the characters cannot just get whatever the want; where's the drama in that. We need conflict. Some obstacle gets in the characters way, preventing them from fulfilling their goal. The Duke's Minister of Finance argues strongly against loaning the characters money, or a guard comes across the characters as they are cutting through the barb-wire perimeter.
After introducing the conflict, both the players and GM need to develop it. Don't let yourself get stuck in "Yes, I am!" "No, you're not!" circular arguments--or repetitive, dice-based resolutions. The conflict needs to move forward. Both players and their opponents should bring up different points, try different tactics. If they are losing on one front, they should shift to another. This also applies to inanimate obstacles. So, the characters are nimble enough to cross the raging river on an old log, what else could go wrong?
You should spend most of the scene (75 to 90% of the time) developing and expanding the conflict.
Finally, the scene should end in disaster. Not only do the characters fail to reach their goal, they end up worse off than before. The Minister of Finance convinces the Duke that the characters are actually spies. Or the characters set off an alarm and can hear the telltale rumble of attack helicopters rapidly approaching.
I've heard similar advice from other writers. It basically boils down to "Never give the characters what they want." Of course, even a cursory glance at most stories will show you that characters do occasionally get what they want (though usually only after surviving considerable difficulty). Not all scenes end in disaster. Sometimes the characters do come out ahead. They find an important clue or a magical sword. Whatever. But, the basic advice is still sound. The characters should fail more often than they succeed. Their position should deteriorate over the length of the story. Things should get harder for them--tensions increase--until you reach the climax.
Traditional RPGs often run in the opposite direction. Characters become more powerful over time--either by gaining experience, or by stealing magical treasures off the dead. Sure, the conflict (and by this I mean combat) may have had a small toll on the party. Spells were spent; potions quaffed. But given a good night's rest, 90% of that cost has disappeared, and the characters have a few shiny new toys to try on the next band of orcs the run across.
Now, character advancement is a powerful motivator for the player. It is one of the more-significant sources of enjoyment for many. But, the way most games focus on character advancement makes creating good story arcs difficult. Worse yet, this idea of character advancement is so pervasive, it even infects many (if not most) of the independent story-focused games on the market.
Here are two more pieces of advice on scenes. A mentor of mine once said, "A scene should always have at least two things happening." If there is only one thing happening, it's not worth the effort. Better to cut the scene entirely.
Finally, never bring anything on stage unless you plan on using it later in the story.
Now, role-playing scenes seem to fail in several ways. For me, the worst is the "random encounter." This violates almost all the advice listed above. The characters have no goal, except to survive the scene. There is conflict (again, usually limited to mechanically-resolved combat), but it usually ends positively for the characters. Only one thing happens--someone attacks the characters. And, most importantly, these are random opponents--once slain, they disappear from the story. The resulting scene has no point beyond the immediate tactical challenge. It cannot create a story.
If you want to have bandits attack the party, fine. But make sure there is more meat to the scene. Perhaps all the bandits have a strange brand on the back of their left hand. They are part of a larger conspiracy, one that will crop up frequently throughout the rest of the story. Sure, the characters may win a temporary victory--but they have just uncovered a previously secret organization that wants them dead. That can't be good news.
Another common failure is the pointless-excuse-for-acting scene. Here, the characters interact with quirky locals--perhaps an unusual shopkeeper, tavern owner, or police officer. These scenes are often humorous--but there is no conflict, and the NPCs almost never play any further role in the story.
Let me back off a bit and say that there is a place for quieter, character-building scenes in both fiction and RPGs. These scenes often follow a major conflict. The momentum of the conflict carries us through the quieter scene, but we can catch our breath. Still, these scenes should be rare and they must be short. Nothing kills a story faster than bloated, sluggish scenes like these. If you want the characters to chat with a quirky shop keep, fine. Toss a bit of conflict into the mix; an old nemesis happens to enter the shop halfway through the discussion. Alternatively, make the character a more-important part of the story. Maybe the shopkeeper turns up missing the next day.
Another big failure is trying to build conflict through game mechanics. Let's take a character trying to scale a castle wall. The character has a goal (getting to the top in one piece) and a conflict (climbing is dangerous). In the most-degenerate case, the whole conflict boils down to a die roll. The character succeeds and reaches the top (and the conflict fades completely). Or he fails and falls to his death (again, conflict disappears, but not in a good way). Even if the fall would just injure the character, it is hard to build any real sense of tension this way. Either the injuries from a fall are minor, and the character ignores them. Or they will prevent him from climbing the wall entirely--which often brings the story to a screeching halt.
Ideally, conflict should not be something the characters roll their way out of. Instead, look for conflicts that make the characters to think. Force them to make hard decisions. Take our wall-climbing character. Imagine an experienced climber. The wall does not present a serious challenge. Instead, just after he reaches the halfway point, a teenage love-struck scullery maid appears at the top of the wall. She doesn't notice the climber, rather she stares dreamily off at the horizon, humming under her breath.
Now this presents a real problem for our character. She will probably notice if he tries to sneak past her. Even climbing back down would be risky. If he makes too much noise, she'll surely spot him. He could kill her to keep her from sounding the alarm--if he's the type to murder innocent young girls in cold blood.
This scenario also has room to grow. We can easily increase the tension by having her lover--a young knight--show up at a dramatically suitable moment. We can also have a disaster (the girl screams and alerts the guards) without it being instantly fatal.
From a scene-crafting point of view, the GM seems to have two important roles. First, they have to filter out the incidents that make good scenes from those that don't. The GM should focus game time on the first group, while de-emphasizing the second (keeping the scenes short, or handling the events offscreen). Keep the game focused on the exciting bits.
More important, the GM needs to constantly throw conflicts and complications at the characters. He needs to be mean. Chase them up trees, then throw rocks at them. This is hard, it means acting cruelly to your friends (or, at least, to your friend's imaginary constructs). In my experience, most GMs flinch and fail to provide an adequate level of real conflict in their games. Instead, they focus on mechanical conflict. It's not me, it's the dice.
When building conflicts, think about all the things that could go wrong. If you're planning things out between gaming sessions, actually list at least ten things. Then pick the most interesting. You usually have a good idea of what the players might do during the next game session--write down a few good complications. Often, you can refactor those complications and still use them when the players inevitably go off script.
If you are in the middle of a game, resist the temptation to grab the fist conflict that pops into your head. Those are often the most stereotypical. Most importantly, try to find something that opens possibilities, instead of shutting them. And don't be afraid of being mean.