2 related questions here:

1. I have been running Monte Carlo optimizations on a multi symbol dataset (Dow30) in Raw Profit mode. I am selecting Preferred Values based on maximum Net Profit. After the sims I assign PVs, sometimes based on highest metric value per symbol, and sometimes based on average metric value for all symbols. The issue is that I get different results each time. I cannot recreate the results of any given optimization, even to verify that everything is working correctly. I am guessing that this is due to the Monte Carlo optimizer perhaps starting its random sequence with a new random seed for each run. But even in that case it would seem that with a large number of individual test cases within each Monte Carlo run, different optimization runs should produce quite similar results. What is actually going on here?

2. I have investigated whether perhaps I may be setting the PVs incorrectly. User Guide Section 7.7.2.1 > Preferred Values > Assign Preferred Values based on the highest (lowest) metric value per symbol >
"multiple rows may contain the same highest or lowest metric value (especially for integer metrics or infinite results), and in this case you should specify PVs by using one of the row selection options." I understand the problem, and perhaps that may be causing my differing results described above. However, wat are the "row selection options" referred to here? Does this mean the last 2 options in the menu "Assign Preferred Values ... from this Row"? If so, how is this used with the multisymbol optimization that I am doing? In other words, there is no single row to select? Is there a way to specify multiple metrics to be used for selecting the PVs, in order of priority, in case of equal values for the highest priority metric?
Thanks,
Ron

1 - Monte Carlo optimizer has random nature. It's not guaranteed to always discover the optimum values. Rather, it aims at find optimal ranges much more quickly than an exhaustive search for a large number of combinations.

It's not clear where do you "get different results each time" but if it's about running a backtest (with or without PVs applied) in Port.Sim, mode, then it has a simple explanation in the FAQ: Every time I run a Strategy I get a different result. What am I missing?

2 - The "row selection options" are nothing but standard Windows shortcuts for selecting multiple items: by using Shift / Ctrl or by dragging the mouse. If multiple rows contain the same highest/lowest metric value, you use keyboard/mouse to specify PVs.

Thanks Eugene. Clarifications:

1. I am running mullti-symbol Monte Carlo optimizations in Raw Profit mode. The link above, about getting different results across runs, seems to apply to Portfolio mode..? I am optimizing Strategy Params and selecting PVs based on maximum Net Profit, setting values individually per symbol. I then backtest the entire DataSet with the PVs (again in Raw Profit mode). Each time I do this I get a different total performance, in terms of Net Profit (all conditions are the same - time range, DataSet, Raw Profit mode). I therefore assume that each optimization run is producing a different best PV set.

2. Let's say we run an optimization on the Dow30 Dataset and we want to select PVs based on max Net Profit on a Per Symbol basis (each symbol gets its own PVs). Further assume that for many symbols there are many PV sets that have the same maximum Net Profit. Are you saying that I need to manually check all of the results, for each symbol, to see if there multiple PV sets with max Net Profit and then select the desired PV set, for every symbol in the Dataset? Or is there not some way to tell WLPro to first sort/select PV based on Net Profit and then (secondarily) use another metric, in the case of multiple PV sets with the same max Net Profit.

Thanks
Ron

Ron,

1. You get different results each time you do a MC-optimization-and-then-a-backtest cycle? Given MC's random nature, it's expected, don't you think?

Hi Eugene,

I suppose it depends on the details of Wealth Lab's Monte Carlo algorithm. The MC algorithms I am familiar with are not optimizers but rather ways of generating large numbers of sample cases based on given parameter distributions. In any case, yes, if each MC run produces a different best PV set then it does make sense that backtests using different PV sets would yield different results.

So then my question is, what is the best way to handle multiple "best" PV sets, as mentioned #2 above? Is there any automated way to have WLpro select PVs based on multiple sorting criteria, e.g. Net Profit first, ulcer index second?
Thanks,
Ron

Eugene - a follow-on question:

Since each MC run produces a different "best" PV set, is there a way to make Wealth-Lab automatically make many different PV runs, so I can then average the results? My underlying objective is to compare strategies. Each strategy though, first needs to have its parameters optimized, so that each strategy is providing its best performance, so that the comparison between strategies is balanced (all other variables are held equal - symbols, time frame, etc.). However, as discussed above I get performance variations even within a single strategy due to the variations in optimization results. Perhaps that variation can be mitigated by making many optimization runs and averaging the PVs. That might not make sense mathematically but let's assume it does. Is there a way to make Wealth Lab do this? Can Wealth Lab get/set average PVs based on multiple runs, using all of the symbols within a folder? Is there a way to have the multiple runs be all on the same symbol, as a way to average out the optimization variations?

Larger question: Given that this optimization variation exists, how does one compare strategies in Wealth Lab? In other words, how do you compare the performance of strategies A and B, if the performance of A and B can vary significantly based on choice of "best" PV set for each?

Only one set of Preferred Values is supported by Wealth-Lab.

Not trying for more than one set of Preferred Values, only one.

Is the following possible in Wealth Lab? If so I think it would address the problem.

1. Make a large number (let's say 100) of copies of a single stock symbol. Call them SYM0...SYM99.
2. Put SYM0...SYM99 into one Dataset.
3. Run an optimization on the Dataset, in Raw Profit mode. Presumably use genetic optimization to get the best results.
4. Assign Preferred Values based on highest average metric value for all symbols.

I would think that this would be equivalent to 100 optimizations on the same symbol, each one resulting in a best PV set, and then we pick the best PV set of the 100. BUT: How do you Step 1 -- how do you make a copy of symbol?

QUOTE:
how do you make a copy of symbol?

It depends on the provider and if you want to take the time to automated the process. For ASCII, you can just find the symbol in Explorer, hit Ctrl+C and then Ctrl+V 100 times. It would be pretty even easy to create ASCII DataSets of the same data by looping an export script that you can find in the Wiki.

For the life of me, however, I cannot understand the point of doing this!

Thanks Cone. The symbols I want to duplicate are the ones provided with Wealth Lab, e.g. those in the Dow30. By "Explorer" I assume you mean Windows Explorer? Could not find them. I found the Dataset xml files in C:/Fidelity Investments/ Wealth-Lab Pro 6/Data/DataSets. I also looked in ProgramData/Fidelity Investments/WealthLabPro/1.0.0.0 but no luck. Could you please point me at the symbol files?

The reason for doing is this is to compare strategies in an even-handed way. Since the performance of a strategy varies with its parameter values, to compare 2 strategies we would arguably want to compare their performance using the best param values for each strategy. So, we need to find the best parameter value set for each strategy. That, however, is problematic in Wealth Lab, because each time you run an optimization on a strategy to get the best performing parameter set (keeping all conditions constant) you get a different result. So, my theory is that if we run a large number of optimizations on the same strategy (and symbol(s)) then we should get a cluster or bell curve of results, from which we can somehow extract an average best parameter set, or perhaps the "best of the best" parameter set. Again the underlying objective is simply to compare the performance of different strategies, without biasing the performance of the strategies due to their parameter values.

For example, say we have strategies A and B. A might be a superior strategy but if the parameter values are not optimal then B could provide better performance. To compare them evenly I think we need to find the best param values for each.

Thanks,
Ron

If you always assign a non-random Position.Priority to every Position in your strategy and use Exhaustive method, you will arrive at the same result. Hope that helps.