Master 1 Setup : How do I know if a setup works?

I read somewhere about mastering 1 setup and then moving on to another.
I am in the process of selecting a setup and paying 100% attention to mastering it.

But how would I know if that 1 setup works?
What if the setup in question has discretionary parameters so you can't really test it on Tradestation etc?

Or a newbie just have to experience failure before finding out the credibility of his setups? But then again, how do a newbie know if his setups' failure isn't caused by him?

You just have to experiment until you find something you are comfortable putting money on. It is not easy otherwise we would all be very,very rich by now. In many cases just having a setup that you think works will give you enough of an edge to get involved with confidence. If it gets you into the market and it only works 50% of the time you will make money if your money management skills are up to scratch. If I were you I would spend more time thinking about that aspect of the trade than the setup. If you get your money management right then setups will be easy.

The setup and the trader are in a symbiotic relationship in that even the most mechanical setup can be screwed up by a bad trader. What many traders do is expect the setup or the system or whatever to do the work, relinquishing responsibility for the trade to the setup/system (this is part of the distinction between setting a stop and waiting as an observer for it to be hit as opposed to taking an active role in the trade and employing pre-emptive stops if called for).

If one is inexperienced, there should be little or no discretion involved in the setup, though this does not mean that one is required to use indicators. If one is inexperienced, one has no basis on which to base discretionary judgment.

Therefore, go back X months and test your setup over Y charts until you have something that looks promising. Then, beginning where you left off, "forward" test the setup as if the charts you are working with were current (if you're trading intraday, scroll through the charts bar by bar). If the setup continues to perform as expected, then paper-trade it in real time. And so on, modifying as necessary.

If i may say, you don't need to test blindly your trading strategy. If you are using indicators or something (usually) computationaly simple then you can create backtesting function (in C/C++/C#, MATLAB or MT4 scripting language - or other programming language) so you'll get the best configuration for your strategy - but based on historical data!!! So there is no guarantee that it will work in future.

pardon my ignorance but what exactly does my 'setup' mean? Say I'm trading a system I wrote in Ninja Trader, mechanically taking orders from moving average cross-over entries with trailing stops, using end-of-day data from CRB Data, in IMM currency futures, using InteractiveBrokers, running linux on a big PC with 2 megs of RAM and a 3 GHz CPU - which bit of that is my 'setup'? Or does it just depend on the context?

Your setup occurs when the moving averages cross. I.e when this event 'sets up', you look at going long/short.

Another example of a 'setup' could be like TD's where he looks to reverse when a pinbar forms at levels of S/R confluence.

OK thanks. I'm slowly getting a handle on the lingo round here.