I had posted this on elite trader, but I think people won't understand what I mean by shell program, so I've copied those comments and put them here.

QUOTE:
Long + Short Long Only Short Only Buy & Hold
Starting Capital $100,000.00 $100,000.00 $100,000.00 $100,000.00
Ending Capital $315,627.29 $315,627.29 $100,000.00 $20,499.21
Net Profit $215,627.29 $215,627.29 $0.00 $-79,500.79
Net Profit % 215.63% 215.63% 0.00% -79.50%
Annualized Gain % 855.52% 855.52% 0.00% -95.55%
Exposure 6.82% 6.82% 0.00% 100.00%

Cash Interest $0.00 $0.00 $0.00 $0.00
Margin Loan Interest $0.00 $0.00 $0.00 $0.00
Total Commission $-96.00 $-96.00 $0.00 $-8.00
DividendsPaid $0.00 $0.00 $0.00 $0.00

Number of Trades 6 6 0 1
Avg Profit/Loss $35,937.88 $35,937.88 $0.00 $-79,500.79
Avg Profit/Loss % 25.39% 25.39% 0.00% -78.80%
Avg Bars Held 1.17 1.17 0.00 126.00

Winning Trades 5 5 0 0
Winning % 83.33% 83.33% 0.00% 0.00%
Gross Profit $220,598.67 $220,598.67 $0.00 $0.00
Avg Profit $44,119.73 $44,119.73 $0.00 $0.00
Avg Profit % 31.63% 31.63% 0.00% 0.00%
Avg Bars Held 1.20 1.20 0.00 0.00
Max Consecutive 5 5 0 0

Losing Trades 1 1 0 1
Losing % 16.67% 16.67% 0.00% 100.00%
Gross Loss $-4,971.38 $-4,971.38 $0.00 $-79,500.79
Avg Loss $-4,971.38 $-4,971.38 $0.00 $-79,500.79
Avg Loss % -5.77% -5.77% 0.00% -78.80%
Avg Bars Held 1.00 1.00 0.00 126.00
Max Consecutive 1 1 0 1

Max Drawdown $-21,394.72 $-21,394.72 $0.00 $-98,343.91
Max Drawdown Date 5/8/2009 5/8/2009 N/A 3/6/2009
Max Drawdown % -6.65% -6.65% 0.00% -95.64%
Max Drawdown % Date 3/17/2009 3/17/2009 N/A 3/6/2009

Wealth-Lab Score 11,717.8991 11,717.8991 0.0000 -186.9314
RAR 12,553.0688 12,553.0688 0.0000 -95.5487
Profit Factor 44.3737 44.3737 0.0000 0.0000
Recovery Factor 10.0785 10.0785 0.0000 0.8084
Payoff Ratio 5.4779 5.4779 0.0000 0.0000
Sharpe Ratio 2.4393 2.4393 0.0000 0.1779
Ulcer Index 1.4120 1.4120 0.0000 75.4648
WL Error Term 17.1130 17.1130 0.0000 39.3510
WL Reward Ratio 49.9926 49.9926 0.0000 -2.4281
Luck Coefficient 1.4859 1.4859 0.0000 0.0000
Pessimistic Rate of Return 7.5703 7.5703 0.0000 0.0000
Equity Drop Ratio 0.0002 0.0002 0.0000 -0.1984

Position Symbol Shares Entry Date Entry Price Exit Date Exit Price % Change Net Profit Bars Held Profit/Bar Entry Signal Exit Signal Cum Profit MAE % MFE % Delta Norm Series Actual Ratio Series
Long FAZ 1,560 3/11/2009 55.20 3/12/2009 52.02 -5.77 -4,971.38 1 -4,971.38 Buy FAZ -4,971.38 -6.97 9.24 2.15 0.06
Long FAZ 2,269 3/16/2009 38.00 3/17/2009 39.54 4.04 3,480.94 1 3,480.94 Buy FAZ -1,490.44 -8.42 18.47 2.13 0.13
Long FAZ 3,588 3/19/2009 24.05 3/20/2009 35.35 46.98 40,540.96 1 40,540.96 Buy FAZ Profit Target 39,050.52 -0.71 47.00 2.66 0.27
Long FAS 21,847 4/8/2009 6.34 4/13/2009 9.32 46.99 65,083.70 2 32,541.85 Buy FAS Profit Target 104,134.22 -7.57 47.00 -1.21 0.34
Long FAS 29,877 4/21/2009 5.68 4/22/2009 8.35 46.99 79,743.64 1 79,743.64 Buy FAS Profit Target 183,877.86 -1.41 47.00 -1.14 0.56
Long FAZ 52,008 5/7/2009 4.65 5/8/2009 5.26 13.13 31,749.43 1 31,749.43 Buy FAZ 215,627.30 -0.86 26.45 2.80 2.18

First off, I have no vested interest in this system. I'm aware that 6 trades in a 127 bar backtest has severe limitations. How long would you wait before you tried to trade it?

What'd I do?

I took the shell program for http://www.collective2.com/go/pairsqidqld and ran it on the watchlist FAS and FAZ, and after some optimization generated these results.

I think this will be a good example of both what not to do and what to do.

What not to do

Trade it, or even consider trading it.

What to do Wait several months or possibly a year before to get at least 25 trades and run a t test for statistical significance.

I've laid out the results of the program, but that doesn't mean I'm going to go committ tons of money to it. I'm not going to committ anything to it. All I did was take 20 minutes with the program I used to generate PTQQS and optimized on FAS and FAZ data with it.

The results are quite good, but have no statistical significance. I thought I'd create this thread so that people can see what a developer like myself goes through before committing dough to it. It'll be quite some time before the system becomes viable, as the data on FAS and FAZ are incredibly short. I had about 8 months of QID and QLD data when I created pairs trading QID QLD Scalper. On this I have only 5.5 months of data, which is hardly enough to do any kind of testing on.

Looks good, won't trade it though.

BGU and BGZ from the same shell program for their entire life of 129 bars to date.

Long + Short Long Only Short Only Buy & Hold
Starting Capital $100,000.00 $100,000.00 $100,000.00 $100,000.00
Ending Capital $180,812.41 $180,812.41 $100,000.00 $67,377.60
Net Profit $80,812.41 $80,812.41 $0.00 $-32,622.40
Net Profit % 80.81% 80.81% 0.00% -32.62%
Annualized Gain % 216.04% 216.04% 0.00% -53.56%
Exposure 15.44% 15.44% 0.00% 100.00%

Cash Interest $0.00 $0.00 $0.00 $0.00
Margin Loan Interest $0.00 $0.00 $0.00 $0.00
Total Commission $-256.00 $-256.00 $0.00 $-8.00
DividendsPaid $0.00 $0.00 $0.00 $0.00

Number of Trades 16 16 0 1
Avg Profit/Loss $5,050.78 $5,050.78 $0.00 $-32,622.40
Avg Profit/Loss % 4.19% 4.19% 0.00% -33.42%
Avg Bars Held 1.25 1.25 0.00 128.00

Winning Trades 11 11 0 0
Winning % 68.75% 68.75% 0.00% 0.00%
Gross Profit $107,662.36 $107,662.36 $0.00 $0.00
Avg Profit $9,787.49 $9,787.49 $0.00 $0.00
Avg Profit % 7.85% 7.85% 0.00% 0.00%
Avg Bars Held 1.27 1.27 0.00 0.00
Max Consecutive 6 6 0 0

Losing Trades 5 5 0 1
Losing % 31.25% 31.25% 0.00% 100.00%
Gross Loss $-26,849.95 $-26,849.95 $0.00 $-32,622.40
Avg Loss $-5,369.99 $-5,369.99 $0.00 $-32,622.40
Avg Loss % -3.87% -3.87% 0.00% -33.42%
Avg Bars Held 1.20 1.20 0.00 128.00
Max Consecutive 3 3 0 1

Max Drawdown $-14,872.89 $-14,872.89 $0.00 $-68,992.00
Max Drawdown Date 5/7/2009 5/7/2009 N/A 3/9/2009
Max Drawdown % -9.03% -9.03% 0.00% -68.99%
Max Drawdown % Date 3/17/2009 3/17/2009 N/A 3/9/2009

Wealth-Lab Score 1,272.6965 1,272.6965 0.0000 -90.5167
RAR 1,398.9732 1,398.9732 0.0000 -53.5627
Profit Factor 4.0098 4.0098 0.0000 0.0000
Recovery Factor 5.4335 5.4335 0.0000 0.4728
Payoff Ratio 2.0277 2.0277 0.0000 0.0000
Sharpe Ratio 4.7015 4.7015 0.0000 0.8040
Ulcer Index 2.5758 2.5758 0.0000 42.2085
WL Error Term 4.8570 4.8570 0.0000 13.8880
WL Reward Ratio 44.4793 44.4793 0.0000 -3.8568
Luck Coefficient 1.7945 1.7945 0.0000 0.0000
Pessimistic Rate of Return 2.1531 2.1531 0.0000 0.0000
Equity Drop Ratio 0.0043 0.0043 0.0000 -0.2044


I don't really have a point other than to make people see what it's like to have a hypothesis, because it means that you can take what you've already done and "tweak" for the data.