The ability to strike hard at an opponent with the simultaneous willingness to absorb some damage is usually more than enough to take out most people you will end up fighting, thus just about any martial art CAN be effective. The question is not 'Does your self-defense work,' but 'Are you able to make it work?' Or, as I heard it put so eloquently, "The martial art works just fine, you just have to have the guts and sense do it right."
The main issue, in my experience, is that effective use of any martial art requires some modicum of aggression/combative intent. To put it bluntly- the reason black belts get their butts handed to them by some random punk is because the martial artist has all the moves and none of the intent. Most people, in this day and age, who go to martial arts go precisely BECAUSE they are not comfortable with generating or acting upon combative intent- they are not comfortable getting in there and FIGHTING. They begin training in martial arts under the assumption that they are a collection of "tricks" that will allow them to protect themselves without having to generate or familiarize themselves with the rush and fear or combat.
This being said, there is a distinct difference between combat and "social combat." Many martial arts/self-defense arts are designed for only one or the other. If you try to use either in an inappropriate situation, or with inappropriate intent (ie- trying to use very lethal combat without the intent to actually kill your opponent) you will most likely have limited results. Most fighting in the civilian world is social combat- its purpose is not to kill or maim your opponent but to assert social dominance. This is why you see fights that go on for 5 minutes, countless punches thrown, and both people are able to get up and walk away. This is why untrained people, or trained people who lose their cool (perhaps after getting surprised or taking some damage) tend to instinctively resort of wide, open hand strikes to the top sides of the head and the upper ribs- the weakest strike directed at the most armored part of the opponent's body. We are not really engineered for natural killing of other humans, but rather to use social combat to assert dominance. This is why people tend to have to be trained to kill, animals do not.
To read a much more in depth scientific explanation of the differences and reasons for these things, there is an
article on the hoplology (the study of human combative behavior) website. Be forewarned, it gets pretty involved.
Thus, the ultimate question is not whether or not your self-defense works, but what are doing to condition your body, mind and heart to use it effectively.
Tyler