Shifting Intraday DataSeries by exactly 1 day
Author: ronc
Creation Date: 2/6/2014 10:58 PM
profile picture

ronc

#1
1) I have 1-minute dataseries that I want to shift by exactly 1 day. I assume that each day comprises 390 bars. I am concerned that data anomalies could result in some days containing than fewer than 390 bars (e.g. data issues, suspensions in trading, etc.). How to handle this concern?

2) I tried using the series shift operator >> but that did not work. So I am trying the code below, but that is giving a divide by zero error apparently when it hits the Synchronize statement. Can't figure out why. Note that I have hardcoded the file path with my user name; you'll need to change to yours if you want to run this. Again my objective is to shift 1-minute data series to the right by integer numbers of days. Starting with 1 day = 390 bars.

Thanks!

CODE:
Please log in to see this code.

profile picture

Cone

#2
First you need to answer the questions where synchronization will fail, like the edge cases:

1. What should be done with the data after 1pm if you're shifting on to a day that closes early (like for Thanksgiving or Xmas)?
2. The opposite of #2. What do you do when you don't have data after 1pm to shift?
3. And of course, how do you handle the cases for zero-volume bars that don't exist?

Maybe you can sum it up simply by answering, what's the ultimate purpose of doing this?
profile picture

ronc

#3
My ultimate purpose:
I am training an intraday trading algorithm using 1-minute data. I have been using approximately 8 training days. Genetic optimization. Single symbol. RP mode. Optimization times about 24 hours per run. Previously I have done a lot of optimizations of other strategies and time frames using Portfolio Mode. It seems that Portfolio Mode is faster than RP mode for the same total amount of training data. My plan (that I want to test): Instead optimizing on days in RP mode on a single symbol, take 2-day chunks of data of that symbol (2 days is the shortest duration you can use WL Pro), time-shift them, and create new symbols for for each time-shifted series. Then optimize on the 2-day span of data in Portfolio Mode. So we have symbol BAC, BAC-shifted-by-2-days, BAC-shifted-by-4-days, etc. and we optimize on 2-day span across all these symbols in Portfolio Mode. If PS mode is in fact faster then this would provide optimization over the same total span of data but with lower training time and further would be more realistic since I only want to day-trading (i.e. the optimizations would be over 2 days rather than 8).

For questions 1,2,3 above, I am fine with just shifting the data as-is, with no modifications. I.e. if a given day has missing data or starts/ends early then that would just shift wholesale as it is, because it is just for training, e.g. its OK to trade on a holiday for the purposes of training. My training dates are arbitrary.
profile picture

Cone

#4
Maybe there's a discernable difference in optimization results between PS and RP modes due to post-processing, but the optimization would be completely difference since the results are combined in PS mode.

Anyway if you're sure you want to go through the trouble, the easy way (probably) to go about the date shift/new symbol task is to create ASCII files. The name you give to the file is the name of the symbol, so that's solved easily. To shift the dates before saving the data to the file, just subtract full days like this Date[bar].AddDays(-2); (For Monday and Tuesday, you'll have to subtract more, and don't forget about holidays.)

Make sure to turn on the ASCII Caching option for fast file reads.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).