J.J. Abrams has often talked about using the “mystery box” as a writing tool.* The underlying idea is that the mystery of what’s in the box is more interesting than what’s actually in the box. The longer you can hold that mystery, the more interesting the project is.
This idea is, I think, very flawed. Mostly because it encourages lazy writing.
Probably the best example of Mystery Box done well comes from Pulp Fiction. We never learn what’s in the case that Vince and Jules pick up. What it might be has fueled a ton of speculation. But what’s important of why it works is that not knowing what’s in the case doesn’t hurt the story. Whatever it is, it’s important to Marcellus Wallace, and it has a value that’s intrinsically recognizable even to Pumpkin and Honey Bunny.
When it doesn’t work is when the mystery-for-mystery’s sake gets in the way of logical storytelling. In fantasy tropes, this is the Enigmatic Wizard– the character who knows exactly what’s going on but refuses to say for no reason other than the author wants to keep people in the dark. In fact, more often than not, in those cases, keeping the secret is the very reason things go horribly wrong. The climax sequence in Harry Potter and the Order of the Phoenix hinges on deliberate poor communication between Dumbledore and Harry. But in that case, it’s actually theme: that keeping secrets is the weakness between them.
But more often than not, it’s the case of a writer coming up with a mystery that they really don’t know the answer to: so they prolong the mystery to spin their heels because they can’t think of anything that’s cool enough to match what they’ve built up. Sometimes they then answer the mystery box with a new one: this leads to a series of mystery-boxes-nesting-dolls, like X-Files eventually gave us: zero answers because questions had to keep spinning.
This is rarely satisfying.
This is especially troubling when, in an effort to hide the obvious answer of what’s in the box, the writer works furiously to lead you away from that answer to the point where the obvious answer that it turns out to be doesn’t make sense any more.
Case in point (SPOILERS FOR STAR TREK INTO DARKNESS): the movie (and the pre-press) worked furiously to not have you know that Benedict Cumberbatch’s character was actually Khan. Let alone how ludicrous him being Khan in the new timeline is and why trying to re-create Wrath of Khan doesn’t work in this context– in the effort to preserve the mystery they more or less stripped everything Khan-like from the character. Go back and watch “Space Seed”– is there any real connection between Montalban’s performance and character there and Cumberbatch’s? I’m not seeing it. But they wanted Who Is He? to be a mystery with a big reveal– but again it’s a reveal that plays all wrong, because it doesn’t mean anything to the characters. In fact, if you read between the lines of the screenwriters’ comments out there, you see that they didn’t want Khan, and they more or less wedged him into a plot that didn’t need him.
The best kind of mystery box is one where knowing the answer doesn’t render re-read fruitless. Where the foreshadowing pays off in a satisfying way. Else it’s just messing with your audience for the sake of confusing them. Why do that?
*- Here’s a link to his TED talk on the subject.