Fair warning, I'm going to be talking about magic a lot in this one. No, I'm not proud of myself.
JJ Abrams, now well on his way to fulfilling the universal dream of uniting two endlessly warring SF nations in a hands-across-the-fanverse exercise I never would have imagined possible, owns a box of magic tricks he claims he's never opened. It's his "Mystery Box" - and if I'm honest, the way he talks about it goes a long way toward solidifying in my mind why I've never really been excited by anything I've seen of his.
I'll qualify that. I've seen both his Star Trek films, but I've never been a Trek guy so they didn't do anything for me. I've seen the first two episodes of Lost, but only because they got Rifftraxed, so that's hard to judge by. I did like Almost Human, but I don't know what he actually contributed to the show beyond composing the theme music. Basically, my sample size is just way too small to draw conclusions from, and almost none of what I've seen was aimed at me - I'll also mention here that despite all of the above he's got me optimistic about the future of Star Wars now in a way I haven't been since before the turn of the century.
In his TED talk, Abrams performs a quick magic trick using (and implicitly exposing) a fairly standard sleight called a backpalm. In fact, whoever was in charge of the camera angles that day actually goes a step further and gives the game away outright with a carefully revealing shot at the critical moment. What caught my attention there wasn't the magic or the instant undermining it suffered, so much as the vital distinction that wasn't being made.
A secret isn't a mystery.
When Star Trek: Into Darkness was being promoted prior to release, the concept of the Mystery Box seemed to be all around it - at least for the length of time it took for someone to cut a hole in it. It was the second Trek movie of the new series, and we were told there would be a mystery villain. Only, the "mystery" lasted about five seconds because the immediate response from anyone who could possibly care about it was "is it Khan?"
What followed was an embarrassing mess of unconvincing denials and obvious question dodging. In the end [SPOILERS] it obviously was Khan and what should have been a jaw-dropping twist went off in cinemas like a sickly, wet fart. Iron Man 3 kinda went the same way for me with a twist that was purely cosmetic and contributed nothing but some pace-killing comic relief. The main bad guy and his boss basically switched places and nothing that actually mattered had changed. To me, it was a mildly irritating bit of sleight-of-hand that accomplished nothing concrete. To lock a frame around this for a moment, imagine if Return of the Jedi had pulled the same switch, revealing that Vader was the real head of the Empire and "the Emperor" was a hired stooge. What actually changed there, really? We already knew that Vader/Killian was a powerful bad guy doing powerful bad guy stuff. Now it just turns out that the story's mid-game mini-boss is actually the final boss and the whole third act just kinda deflates on itself.
Anyway, I'm drifting. My point is that if you're going to build a Mystery Box around your story, I think you need to be sure that what you have isn't just a poorly guarded or irrelevant secret. Every magic effect has a secret to it, and pretty much any magician will tell you that in general that's the least important or interesting thing about the trick. Knowing a secret, or revealing it, will eliminate the magic in a heartbeat - unless the secret itself poses further questions of its own. At that point it's possible you have a legitimate mystery to work with.
Example: there's a magic trick out there that I first encountered under the name of "Smash and Stab". It's possible you've seen some variation of it, and would probably recognise it as the "spike under the cup" trick. Essentially, the magician engineers a situation where he's slamming either his own or a spectator's hand down on a set of polystyrene cups, under one of which the spectator has hidden a spike or blade while the magician looked away. It's always struck me as a weird idea for a piece of magic, because it makes virtually no structural sense. There are literally only two possibilities for the spectator to consider: either the magician somehow knows where the spike is (in which case there's a legitimate magic trick at work, but no danger) or he doesn't (in which case there's very real danger but no actual magic). However, that trick alone has made the reputation of many magicians - crucially, even when they've injured themselves or others failing at it - because the secret is the least important part of the story it tells, and if it's the secret you're focusing on then you're - ahem - missing the point. The drama is undeniable whichever of the solutions is correct.
Counter-example: while the Sixth Sense and Unbreakable still arguably work as films when the twists are revealed, you'd be hard pressed to say the same about something like The Village. The Mystery Box they've built there amounts to nothing more than "what's the twist that we know is coming actually going to be?" - and that weak premise ends up being the only point of the story. It's the movie equivalent of a comedian saying, "I dreamed last night I was eating a giant marshmallow, and when I woke up YOU'RE A MORON!"
All I'm really saying is that a secret is nothing more than information withheld. It's up to the storyteller to make that information meaningful and, because you can only answer the question once, it's a fragile thing to build focus and tension around. A magic trick that, for example, leaves the spectator with an impossible object in their hand uses secrecy to create a mystery that they can take away with them - which to me is a much stronger storytelling principle. The greatest mysteries still have value once the secrets that enable them to exist are revealed. They can survive the revelation. Spoilers for good stories never bother me for precisely that reason. If the only thing keeping your story upright is the secret at its heart, then all you've actually got is a joke with a one-shot punchline that'll never be funny again.
The above is really just the tip of a much longer conversation that's permanently raging in my brain about the relationship between magic and storytelling. There's a lot more and, as anyone who knows me will affirm, if you catch me in person at a convention you'll be lucky to walk away without hearing at least fifteen minutes of it. I'll save the rest for then...