GSB Forums - Powered by XMB 1.9.11

Peter: that session time ELD looks to be very useful. Do you implement it over the entire data range or only over more recent periods?

I normally do it over the full length of data you use in GSB.
I doubt its critical though

cotila1 - 22-3-2018 at 07:22 AM

Peter, I have read the manual, but to be very honest I dont fully get the point of the new methodology we should follow.
For instance I read ''make sure you update the methodology to included market analysis'' What you mean by that?
We need to choose the right session time based on your new script? Are the current 8.30-15 sessions still ok for ES and NG??
May be the main point is to try to avoid markets which look tough in showing decent performance over seen-data (such as CL). Please kindly clarify more.
If this new methodologies are crucial, better u do a video?

admin - 22-3-2018 at 07:17 PM

Peter, I have read the manual, but to be very honest I dont fully get the point of the new methodology we should follow.
For instance I read ''make sure you update the methodology to included market analysis'' What you mean by that?
We need to choose the right session time based on your new script? Are the current 8.30-15 sessions still ok for ES and NG??
May be the main point is to try to avoid markets which look tough in showing decent performance over seen-data (such as CL). Please kindly clarify more.
If this new methodologies are crucial, better u do a video?

830 to 1500 is still best for ES. The script is needed for other markets like, CL,S,NG etc
I am doing very long hours researching this, and its a massive topic. It's way too early for me to do a video.
Unless a market has been tested to see conceptually if GSB works on it, I don't recommend you live trade it.
Due to the massive CPU time needed to make CL markets, and that my tests showed unseen data failed in nearly all cases. I say avoid the CL market till this is fixed.
There are other complexities as sometime the market is just really hard to trade. Like ES in 2005, 2016, 2017.
Other factors like the range in 2007 2008 is so high, much of the profits are in these years and it makes the parameters shift to only trade well in this sort of high range market.

ES and Nat gas seemed to work well out of sample. ES is also a hot market at this instant.

Some of the things im working is is making multiple data sets of the same data, some with noise added. Others with the data shifted. ie 29 min bar and 31 minute bar.
The theory to test is will a system optimized in this modified data work better out of sample than a system optimized on the true data.
This all takes time to program and evaluate. I will keep you posted.

cyrus68 - 23-3-2018 at 12:29 AM

The spike in volatility that occurred last month may have been a prelude to greater generalised volatility in all asset classes, going forward. In other words, the policy-induced low-volatility era of the post-2008 period may now be over. This has implications for model construction and performance.

admin - 23-3-2018 at 12:31 AM

Ive made some progress in methodology that seems fairly simple, fast from a human time and cpu time on ES
I enclose a picture.
More probably next week. Still waiting on programmer to do some things

admin - 23-3-2018 at 12:33 AM

The spike in volatility that occurred last month may have been a prelude to greater generalised volatility in all asset classes, going forward. In other words, the policy-induced low-volatility era of the post-2008 period may now be over. This has implications for model construction and performance.

Totally agree, so im putting more weight into ES.
Beware chance of getting stopped a few times in succession is also higher.

cyrus68 - 23-3-2018 at 12:33 AM

In terms that Sun Tzu would understand: the terrain may have changed.

cotila1 - 23-3-2018 at 02:12 AM

And we are all with you Peter in this great journey. I am sure you will do a great job, as already done up to now with an excellent progress.

So summarizing, I would say to not trade live, for now, the markets you mentioned, may be the only reliable ones are ES and NG, and wait for new important future GSB releases that might incorporate the new methodology. Am I right?

Petzy - 23-3-2018 at 03:56 AM

Did some test with the new methadolgy. Looks good and stable. (I have only run gsb for maybe 1-2 hours at speed 45000/min)

Attachment: Login to view the details

admin - 23-3-2018 at 03:57 AM

[/rquote]

And we are all with you Peter in this great journey. I am sure you will do a great job, as already done up to now with an excellent progress.

So summarizing, I would say to not trade live, for now, the markets you mentioned, may be the only reliable ones are ES and NG, and wait for new important future GSB releases that might incorporate the new methodology. Am I right?
[/rquote]
Yes,
here is a performance report Out of sample Nov 1 2007

Attachment: Login to view the details

cotila1 - 23-3-2018 at 04:21 AM

[/rquote]

And we are all with you Peter in this great journey. I am sure you will do a great job, as already done up to now with an excellent progress.

So summarizing, I would say to not trade live, for now, the markets you mentioned, may be the only reliable ones are ES and NG, and wait for new important future GSB releases that might incorporate the new methodology. Am I right?
[/rquote]
Yes,
here is a performance report Out of sample Nov 1 2007

Well Done! looks great

cotila1 - 23-3-2018 at 08:01 AM

By the way, do you mind to share what kind of settings have u used? The usual default ''ESsystemsTight.gsbappset'' and ''managerES.gsboptset''?
secondary stream= spx and idx? WF with 200*200?
thanks

admin - 23-3-2018 at 03:46 PM

Its much more than settings alone. I will work on a video asap.

admin - 23-3-2018 at 08:15 PM

See top of this thread
New methodology video
Simulated data, 29,30,31 minute bars.
Simulated data, 26,30,34 minute bars.

admin - 25-3-2018 at 09:05 PM

TO create your own data, go here. (GSB paid members only)
http://www.trademaid.info/forum/viewthread.php?tid=118#pid16...
I have also added 29,30,31 min ES+IDX+SPX in the top of this thread too.

I'm also going to experiment with making systems with no Secondary filter. Amount of trades will be much higher - should be less curve fitting. Once this is done do a Walk forward with Secondary filter on. GSB needs some small tweaks to do this.

admin - 26-3-2018 at 05:48 PM

Here is a system I made today. On 29.30,301 es,spx,idx from 2000 to 10/2007
Results degraded a little with 2500 stop.
System in the member private forum soon.
Note the performance increase with this Truism added. I havnt test this apart form this system, but suspect its very worth while.
If someone has the time, would be worth testing on other GSB systems. Not that in this case it worked better for short than long.
Results shown in all other reports excluded this additional filter, and included $30 round turn slippage and commission
If decision = 1 And sfDecision = 1 and close >close[1] Then
Begin
Buy("Long entryM2.4") 1 contracts this bar on close;
End
Else If decision = -1 And sfDecision = -1 and close < close[1] Then
Begin
SellShort("Short entrym2.4") 1 contracts this bar on close;
End;

Attachment: Login to view the details

admin - 30-3-2018 at 05:54 PM

TESTS ON GOLD.

I did some tests on gold 30 minutes.
Put sessiontimecalcs on the data

....730
745, Rng,193.0418, abs movement,0.7407, Volume, 639.07
800, Rng,212.9424, abs movement,0.7305, Volume, 761.08
815, Rng,256.0768, abs movement,2.4897, Volume,1181.31
830, Rng,337.5720, abs movement,6.5775, Volume,2003.79
845, Rng,396.9856, abs movement,1.5329, Volume,2776.62
900, Rng,305.2675, abs movement,1.5638, Volume,1978.20
915, Rng,304.9040, abs movement,0.2881, Volume,2009.72
930, Rng,291.9444, abs movement,1.2791, Volume,1904.54
945, Rng,351.8930, abs movement,0.0583, Volume,2466.78
1000, Rng,334.2215, abs movement,1.2106, Volume,2418.63
1015, Rng,353.0281, abs movement,12.2462, Volume,2849.90
1030, Rng,304.3724, abs movement,6.0254, Volume,2177.36
1045, Rng,304.6982, abs movement,1.4335, Volume,2091.55
1100, Rng,283.9369, abs movement,1.7661, Volume,1893.64
1115, Rng,278.1516, abs movement,1.4849, Volume,1820.83
1130, Rng,258.6111, abs movement,5.0137, Volume,1648.02
1145, Rng,249.9897, abs movement,1.3100, Volume,1457.51
1200, Rng,235.0754, abs movement,6.5878, Volume,1310.24
1215, Rng,229.4925, abs movement,0.2126, Volume,1203.68
1230, Rng,214.4102, abs movement,2.2942, Volume,1093.11
1245, Rng,211.7284, abs movement,4.9657, Volume,1065.77
1300, Rng,209.3724, abs movement,4.1392, Volume,1028.39
1315, Rng,218.4328, abs movement,2.1571, Volume,1158.15
1330, Rng,227.7435, abs movement,5.4801, Volume,1421.82
1345, Rng,182.5652, abs movement,1.8621, Volume, 777.25
1400, Rng,172.2291, abs movement,1.4266, Volume, 595.25
1415, Rng,201.2791, abs movement,1.4986, Volume, 772.62
1430, Rng,185.2778, abs movement,2.7503, Volume, 642.51
1445, Rng,169.1152, abs movement,0.1783, Volume, 512.92
1500, Rng,158.2339, abs movement,1.6907, Volume, 471.74
1515, Rng,158.0178, abs movement,3.4979, Volume, 432.67
1530, Rng,148.3196, abs movement,0.3052, Volume, 385.09
1545,......
This shows 815 (bar starts at 800) to 1330 is where the volume and range are.

I then looked at the gold monthly chart to figure out what periods I didnt want in sample.
So after 7/29/2011 the range got bigger, though the last few years I feel gold systems will do poorly. (My own experience with all gold systems I have looked at)

SO I then build systems with gold over this period, and then later changed the gold dates to show data after 7/29/2011
(See trading periods on the left of main GSB screen. As of build 44.21 this can be changed at any stage.
Below was typical results.

24 in 25 systems (all with anchored stability of 80 or above) failed badly.
The one that passed was good till the 3/2016
I then built gold data from 2007 to 7/29/2011 using 30 min, then 29 min then 31 minute. In the data file there is no data after 7/29/2011.
I did a walk forward on the best systems.
Then i changed the data1 file to have data from 2007 to 2018 30 minutes.
This was typical of the results.

Systems profitable out of sample but still not great.
All systems with anchored stability of 60 of over were profitable out of sample. All but one struggled after march 2016.
The stability scores were much lower on 30,29,31 min bars than 30 minute. (no surprise as its much harder to trade 30 29 31 then just 30 min bars.

My conclusions.
1) Using 30,29,31 min bars is excellent for building systems, and especially for market validation.
2) Gold is a harder market, and from a systems perspective is a really hard market to trade the last few years.

The app to make the 30,29.31 minute date needed an update to cope with missing 1 minute bars in gold. I will update at some stage.

admin - 30-3-2018 at 06:19 PM

Unrelated, there is another assumption that needs to be challenged.
Changing to a related market will still give reasonable results if the system is a good one.
ie ES - mini Russell, nasdaq etc.
The proof would be.
Get a system you know is a curve fit and terrible out of sample.
Change the code so it trades the new data set, but all the indicators etc are on the original data set.
My strong suspicions is the system will still do well, till the out of sample date.
The other thing to look at is, make system system indicators run on the new data only, and see the results.
If anyone has the inclination to do this, please publish the results.

Carl - 31-3-2018 at 01:24 AM

Hi Peter,

More or less the same test results on RB.

Good looking equity lines and good metrics (Pearson, NP/DD, PF, PAS and so on) on IS dataset RB 28 min -35 min - 30 min.
But most of the systems failed in the out of sample tests.
And the only one that passed the out of sample test had only reasonable IS equity and metrics.

So I was not able to find any correlation between IS metrics and OOS performance.
My conclusion is the same as yours. I think we need more then only good IS metrics and good IS equity lines to choose robust systems. Testing on different sessions and related/correlated ticker might be a way to do this.

admin - 31-3-2018 at 03:40 AM

Hi Peter,

More or less the same test results on RB.

Good looking equity lines and good metrics (Pearson, NP/DD, PF, PAS and so on) on IS dataset RB 28 min -35 min - 30 min.
But most of the systems failed in the out of sample tests.
And the only one that passed the out of sample test had only reasonable IS equity and metrics.

So I was not able to find any correlation between IS metrics and OOS performance.
My conclusion is the same as yours. I think we need more then only good IS metrics and good IS equity lines to choose robust systems. Testing on different sessions and related/correlated ticker might be a way to do this.

Hi Carl,
Im not sure what your saying is the same as me.
Im saying Gold was terrible on 30 min bars, but acceptable on 30,29,31 min bars. Just a tough market last few years.
Are you saying it was not ok Out of sample on 30 ,28,35 (or was that 32 min?)
I think 35 is perhaps to far out form 30, but need lots of testing to verify this.

rws - 31-3-2018 at 04:31 PM

I have do this the hard way in Amibroker, optimize on different tickers and also optimize on different timeframes in the same optimize run. I could for example define the profit as the average of the 15, 30 and 60 min timeframe and to have parameters like that they all have at least >+10% return or do not 50% differ from each other etc.

I found scripts that have good performance on many different timeframes and although these do not have fantastic returns they are robust OOS.
It would be good if GSB could do this automatically.

I have many indicaters in a system and let Amibroker find a good combination and find out which combination is robust. I find that sometimes indicators don't have robust performance and adding things like high> last peak etc can improve. Also japanese candle sticks or optimized price patterns of the last few bars do have some value in combination with indicators. Please add these price pattern functions in GSB.

It doesn't have to be excactly 29-31 min, if a system based on 1 hour would at least have an adjustable positive return on 2 hours, 30 min and 15 min etc chance is much better it works OOS. If GSB just would work with 1 min data and would generate the timeframes automatically then it could find systems that work on all the timeframes that you set.
If GSB also would allow to run a system on a portfolio of several related indexes I am sure it would further improve OOS. I have seen that so many times when doing these tests in Amibroker. I often optimize on 10 tickers that are related to SP500 or Nasdaq 100 to find something that works.

Unrelated, there is another assumption that needs to be challenged.
Changing to a related market will still give reasonable results if the system is a good one.
ie ES - mini Russell, nasdaq etc.
The proof would be.
Get a system you know is a curve fit and terrible out of sample.
Change the code so it trades the new data set, but all the indicators etc are on the original data set.
My strong suspicions is the system will still do well, till the out of sample date.
The other thing to look at is, make system system indicators run on the new data only, and see the results.
If anyone has the inclination to do this, please publish the results.

Carl - 1-4-2018 at 03:19 AM

Hi rws,

For GSB systems on ES the original methodology delivered great results.
GSBSys1 has been profitable on ES and NQ in out of sample period.
GSBsys6 had also been profitable on ES out of sample.
My GSB generated system on ES (developed in June 2017) earned 13k after costs and slippage the last 12 months (9 months out of sample). And I can go on like this. Most of my GSB generated systems on ES have been profitable out of sample.

But on CL, GC and RB it seems it is much more difficult to develop systems that are profitable out of sample.
Maybe because prices behaved in a different way compared to earlier periods? Trend? Volatility?
Or maybe because the original methodology in GSB is not suitable and we need an other GSB methodology for these tickers?

rws - 1-4-2018 at 05:57 AM

Hi Carl,

I also found out after testing many markets (and about 200 of the highest volume ETF's ) that GSB works better on certain markets I think that is because the skeleton of the formula is more or less optimized for ES and the indicators don't matter that much. So far I have had more luck with other ways than GSB and I hope that GSB will improve because I bought it for trading. I have spent hundreds of hours in GSB and tested hundreds of tickers so it is not that I did not try. I don't think it does not work but I don't want to only trade ES as I have had better results on other (low volatile) markets and even simple stock/ETF picking for a couple of hours or days.

That said I did buy GSB as a replacement for searching systems in Amibroker and looking at the presale videos I thought it had great OOS performance.

I could make custom indicators that have pricepattern breakouts and try that in GSB too but I also find that exits/entries can often be improved by looking at leg lengths and fib. levels or support and resistance and that is something I will never be able to do in GSB.

I like the simplicity of GSB and hope it will have more ways to confirm OOS in the near future.

Hi rws,

For GSB systems on ES the original methodology delivered great results.
GSBSys1 has been profitable on ES and NQ in out of sample period.
GSBsys6 had also been profitable on ES out of sample.
My GSB generated system on ES (developed in June 2017) earned 13k after costs and slippage the last 12 months (9 months out of sample). And I can go on like this. Most of my GSB generated systems on ES have been profitable out of sample.

But on CL, GC and RB it seems it is much more difficult to develop systems that are profitable out of sample.
Maybe because prices behaved in a different way compared to earlier periods? Trend? Volatility?
Or maybe because the original methodology in GSB is not suitable and we need an other GSB methodology for these tickers?

Carl - 1-4-2018 at 10:21 AM

OF course it would be great if GSB could develop robust systems on all ticker symbols.

Hopefully with the new methodology and/or other secondary filter (or some other alternative approach) it will be possible to build robust systems on other ticker symbols than ES, NQ and ES.

And if not (worst case scenario) with GSB we are able to build robust systems on ES, NQ and NG.

zdenekt - 1-4-2018 at 01:30 PM

The most important point are EXITS from the trade.

You can have thousands of filters for trade entry but if you don´t have the option to try different exits then the variability of strategies is limited and conditions used in strategies are converging on the particular market regardless of the computing time and quantity of "unique systems" generated.

There is only few markets we can trade intraday because they have reasonable daily range for time frames 15 minutes and higher. All other markets are tradable with swing strategies but we have only intraday exit "market on close" in GSB...and Dollar stop loss / profit target.
If we will have more diffrent exits then we can find diffrent strategies with many diffrent conditions on many markets and maybe without secondary filter.

We can use a secondary filter on an existing strategy or a set of conditions that already generate a growing equity curve.
Otherwise it is just another condition...so we still try to find best entry conditions and it leeds to more and more conditions for less and less trades...and these multi-condition-combinations in conjunction with a low number of trades will usually fail in the future because they will only work at the appropriate volatility (only few times in decade).

In my opinion EXITS should have a higher priority than the secondary filter.

Sometimes even stupid trade entry is working when you don´t have stupid exit.

rws - 1-4-2018 at 02:36 PM

I think this concern is shared by more users

admin - 2-4-2018 at 12:18 AM

OF course it would be great if GSB could develop robust systems on all ticker symbols.

Hopefully with the new methodology and/or other secondary filter (or some other alternative approach) it will be possible to build robust systems on other ticker symbols than ES, NQ and ES.

And if not (worst case scenario) with GSB we are able to build robust systems on ES, NQ and NG.

[file]1033[/file]
This all just takes time to test. I had good profitability on VXX this month, and SQQQ, but I stopped soybeans as I did not have time to test the market. However Soybeans systems were fairly consistently profitably in March.

admin - 2-4-2018 at 12:39 AM

Need more testing, but soybeans looks fine on 30,29,31 min bars from 1/1/2011 to 2018/3/31
Out of sample was from 1/1/2007 to 12/31/2010 All systems looked bad in 2007, so im just assuming that was just a bad period.
Late note. 9 of 10 systems had good equity curves in the out of sample period. They were all poor from 1/2007 to 8/2008

Carl - 2-4-2018 at 02:06 AM

Great results on soybeans, Peter.

So one more successful ticker for GSB!

admin - 2-4-2018 at 02:47 AM

I did tests on sb30 with out of sample pre 2011.
All were poor first 18 months approx, and 13 out of 14 had fairly ok equity curves.
Note I put very little effort into choosing good system metrics in the in sample data.
So I think soybeans is good market for GSB. It MIGHT be wise to use 30 29 31 minute data though.
I would need more testing to know for certain if this is greatly needed.

John62 - 2-4-2018 at 06:20 AM

Quote: Originally posted by zdenekt

The most important point are EXITS from the trade.

You can have thousands of filters for trade entry but if you don´t have the option to try different exits then the variability of strategies is limited and conditions used in strategies are converging on the particular market regardless of the computing time and quantity of "unique systems" generated.

There is only few markets we can trade intraday because they have reasonable daily range for time frames 15 minutes and higher. All other markets are tradable with swing strategies but we have only intraday exit "market on close" in GSB...and Dollar stop loss / profit target.
If we will have more diffrent exits then we can find diffrent strategies with many diffrent conditions on many markets and maybe without secondary filter.

We can use a secondary filter on an existing strategy or a set of conditions that already generate a growing equity curve.
Otherwise it is just another condition...so we still try to find best entry conditions and it leeds to more and more conditions for less and less trades...and these multi-condition-combinations in conjunction with a low number of trades will usually fail in the future because they will only work at the appropriate volatility (only few times in decade).

In my opinion EXITS should have a higher priority than the secondary filter.

Sometimes even stupid trade entry is working when you don´t have stupid exit.

I agree. In some less volatile markets a like swing systems with good exits. Trades for a few days. Here a sample i'm trading since 2016 on TY:

Attachment: Login to view the details

admin - 2-4-2018 at 03:53 PM

Quote: Originally posted by John62

Quote: Originally posted by zdenekt

I agree. In some less volatile markets a like swing systems with good exits. Trades for a few days. Here a sample i'm trading since 2016 on TY:

Programmer says exits and secondary filter are going to be developed together, as the logic is related.
The TY system looks good. Its a market I have never nailed. System I build on TY worked much better when ported to ES. But i have done no work on this for years.

admin - 11-4-2018 at 08:00 PM

I have had great success with the Nth day feature in build 44.32 Basically GSB will not trade every Nth day. This forms a new out of sample period.
I tried nth 3 on ES, and 19 out of 20 systems had increased average trade out of sample compared to in sample

rws - 12-4-2018 at 11:25 AM

In case you have few data or want to test on current
market condition this is something that could be used.

http://www.financial-hacker.com/tag/oversampling/

In that case GSB should make it's data from the for example 1 minute
data and in case of 30 minuts have 29 different data sets of the
same period. This can also help to find if the system is robust.

admin - 12-4-2018 at 03:42 PM

Quote: Originally posted by rws

In case you have few data or want to test on current
market condition this is something that could be used.

http://www.financial-hacker.com/tag/oversampling/

In that case GSB should make it's data from the for example 1 minute
data and in case of 30 minuts have 29 different data sets of the
same period. This can also help to find if the system is robust.

In plain English, how does this work compared to making 29 & 31 min bars added to 30 min? (Entire data sequence used in series or parallel)
Im not sure how you get 29 other bars. Even if you offset the time by 0 to 29 minutes, the first bar of the day will be <30 min. Fine if its 29 min, but not fine if its 1 minute

rws - 12-4-2018 at 05:34 PM

suppose 1 minut base data.

first data set 30 min bars offset 1 min:
1
31
61

second data set 30 min bars offset 2 min:
2
32
62

29st data set 30 min bar offset 29 min
29
59
89

admin - 19-4-2018 at 12:26 AM

Copper (@HG) is a good market from what I can see, but the settings and data streams are critical to get really good systems. I will included them with the next beta in the private forum. Nth 2 worked very well with high % of systems working well out of sample only with the right data streams and settings. I will advise once I have finished my research.

admin - 24-4-2018 at 02:37 AM

Very incomplete video on nth feature. Wont get this finished till tomorrow.

Attachment: Login to view the details

admin - 25-4-2018 at 02:13 AM

Have lots of issues with nth and video. Bug fixes in 44.53. Only the manager needs to be upgraded from .52.
Video may be some time to I figure some things out.
Meanwhile enjoy nth and average of metrics. The metrics GUI will be enhanced before long.

admin - 27-4-2018 at 01:20 AM

Finally, new video on market testing, nth feature, date range * inverse date range, and most importantly- A method to validate if a market is good for GSB.
The method is fast & simple.
Please try this on other markets too.
The video is not as polished as I would like, but I hope it will be useful.
There is a minute of duplicated content that needs to be removed.

[file]1121[/file]
File is now fixed

admin - 27-4-2018 at 05:03 AM

I uploaded the wrong file. It was NOT nth2d-feature.mp4 but was Gsb_Market_testing1f.mp4

Petzy - 27-4-2018 at 08:37 AM

The file: Gsb_Market_testing1f.mp4 is damaged

admin - 27-4-2018 at 03:53 PM

Use the link for market validation video.

removed as upload now fixed

Dropbox link as file upload is failing

admin - 27-4-2018 at 05:06 PM

Finally, new video on market testing, nth feature, date range * inverse date range, and most importantly- A method to validate if a market is good for GSB.
The method is fast & simple.
Please try this on other markets too.
The video is not as polished as I would like, but I hope it will be useful.
There is a minute of duplicated content that needs to be removed.
File is now fixed

Attachment: Login to view the details

cotila1 - 28-4-2018 at 08:16 AM

Finally, new video on market testing, nth feature, date range * inverse date range, and most importantly- A method to validate if a market is good for GSB.
The method is fast & simple.
Please try this on other markets too.
The video is not as polished as I would like, but I hope it will be useful.
There is a minute of duplicated content that needs to be removed.
File is now fixed

Very good Video! thank u.
Just one confirmation: when I set ''Nth day'' to a choosen value (say 1 for instance), then this is applied to training +test periods together and not only to training period solely, am I correct?

admin - 29-4-2018 at 03:23 PM

Yes, it applies to all periods.

cyrus68 - 29-4-2018 at 11:02 PM

The videos were informative. I agree with the contention that one should focus on implementing a strategy that produces a high proportion of attractive systems, rather than doing extensive runs. As Big Dog mentioned on another thread, such a data-mining approach may not produce successful, robust systems.

admin - 29-4-2018 at 11:12 PM

The videos were informative. I agree with the contention that one should focus on implementing a strategy that produces a high proportion of attractive systems, rather than doing extensive runs. As Big Dog mentioned on another thread, such a data-mining approach may not produce successful, robust systems.

Thanks for the comments. I have spend much of the day testing this methodology on ES500, and doing minor tweaks.
ie 15 min vs 30 min. No data2 etc.
Likely I will publish the results in the private forum. Still fine tuning some details before I share results.

cyrus68 - 30-4-2018 at 02:04 AM

I was intrigued by some earlier references to the System Quality Number. Being unfamiliar with it, I searched the net and found the following reference:
https://www.aussiestockforums.com/threads/strategy-performan...

In it, Howard Bandy, who has done lots of work on performance testing, offers the sound suggestion to use the t statistic on OOS results.
I agree with him.

admin - 2-5-2018 at 06:40 PM

My tests for CL looked good. Long looked excellent, short failed, and long and short together were ok.
I haven't checked in detail, but my CL systems overall have been good out of sample.
No systems were terrible in results and some excellent.

admin - 2-5-2018 at 10:40 PM

I have 7 CL systems
The worst cl system had pl -$50 less slippage and commission.
All systems our of sample from December to Jan 23 2018
This was the best system

cyrus68 - 3-5-2018 at 12:57 AM

Good to hear that you have found some attractive CL systems. May I suggest an additional metric in GSB for ranking system performance.
The t statistic, calculated on the P/L, by trade, of the Trd curve would give the best overall measure of OOS system performance.

I think this would constitute an important addition to GSB's capabilities.

admin - 3-5-2018 at 01:01 AM

whats "The t statistic"

cyrus68 - 3-5-2018 at 01:09 AM

Check out the web link I gave in my earlier post.
In it, Howard Bandy describes what it is, and how to calculate it.
It is a better alternative to the System Quality Number.

I have a background in Econometrics. So this stuff comes naturally.

admin - 3-5-2018 at 01:53 AM

I will talk to programmer about T stats.
Here is a vix system I built. Zero slippage used.
Out of sample 20171020
I havnt dont market tests on vix

cyrus68 - 3-5-2018 at 05:17 AM

Why short only? Did you try long/short?
Alternatively, you may prefer directional bets on the short side

admin - 3-5-2018 at 03:34 PM

Why short only? Did you try long/short?
Alternatively, you may prefer directional bets on the short side

Long will not work well on vix. If you look at the chart you will see strong downward trend.

cyrus68 - 4-5-2018 at 12:10 AM

You're right. Over that period the trend has been down.
You don't hold overnight positions. So you are less exposed to upward spikes, which are more likely in the future.

admin - 8-5-2018 at 01:02 AM

44.64 is out with improved internal nth methodology and nth metrics now dont have to used via a mouse over.
http://trademaid.info/forum/viewthread.php?tid=39&page=6
I have found that ES works great (no surprise)
ES works better long than short, but both are ok, and long a little better than long and short.
CL works well (long & short), and even better if long only. Cl short was a fail.
I want to finish and finalize the video on this in < 1 week. I'm still doing more tests.

admin - 9-5-2018 at 12:00 AM

Here is the part2 draft video of updated validation method.
Its best to look at part1 as well, though the methodology has been refined somewhat.

Part 1 was on the forum a few weeks ago.
search for
Market_testing1f_r1.01.mp4

Attachment: Login to view the details

admin - 9-5-2018 at 04:10 AM

CORRECTION TO CONTENT OF VIDEO.
My CL session times were not correct, so Nth oos metrics have improved. Will publish them tomorrow

admin - 14-5-2018 at 06:36 PM

Update to market methodology with the new build of GSB features.
Please note.
The last section on crude oil needs a complete re vamp. Its not explained clearly.
This video says that SESSTIONTIMECALCS.ELD is explained in this video, but it is not. Its covered in the eld comments and the previous video in this thread. I am Unexpectedly over seas for roughly another week, so cant finish this off.
SESSTIONTIMECALCS.ELD is with all the TS code that comes with GSB.
Attachment: Login to view the details

cyrus68 - 16-5-2018 at 06:30 AM

The video proposes a useful test that I may adopt because it is quite fast. However, I would like to note that it is a test of the viability of a particular strategy applied to a given symbol. For example, in the case of ES, if you selected 15-minute data for the primary symbol, different secondary datasets, 5 indicators and 2 operators – this would constitute a different strategy. And let’s not forget different historical periods over which the test may be carried out. The test may produce different results compared with your first strategy.

One thing I noticed is that, for ES, you set the max # of unique systems at 600. Yet at the end you had 1735. Either GSB didn’t stop or you let it run further. I have the same problem with max # completed. Typically, I set it at 5 million. So, if GSB is processing 35,000 a minute, it should stop after 2 hours and 20 minutes. That doesn’t happen. So, I have to stop it manually.

Currently I use a rather biased, but simple method, to gauge the overall viability of a strategy. I look at the proportion of WF results with PAS scores above 50. If, out of 60 WF, there are 40 with scores above 50 - and most are bunched closer to 100 – that is a good sign. If there are only 10 or 15 with acceptable scores, that is not a good sign. The test is biased because WF is carried out on systems that are pre-selected on the basis of attractive metrics.

cyrus68 - 16-5-2018 at 07:03 AM

I would also like to propose a much simpler way of comparing the IS and OOS results. This could be done via the average t-value of the IS and OOS systems. This is a system quality number that I proposed for inclusion in GSB, a while back.

admin - 16-5-2018 at 01:38 PM

I would also like to propose a much simpler way of comparing the IS and OOS results. This could be done via the average t-value of the IS and OOS systems. This is a system quality number that I proposed for inclusion in GSB, a while back.

Is this it? If not for all the viewers of this thread can you re-state it?
What do we do or risk? I know its a minor point, but I personally use very big stops that rarely get hit, so not sure if the risk needs tweaking.
I have a question however regarding the system quality number Van Tharp uses to determine if a strategy is good or not. Basically it is a creative way of working out whether a strategy will perform well long term, based on x amounts of trades. The premise behind it is that it relates everything back to how much you are risking on each trade, in terms of "R" multiples (where "R" is the amount risked). You determine your average R profit over all trades, and also the std deviation of these trades. This is then used along with the number of trades to work out your SQN :

SQN = average of trades in terms of R multiples / stddev of r multiple trades * sqrt(number of trades).From Van Tharp, if your SQN > 1.7 you have a system that "statistically" should generate profits.

Carl - 16-5-2018 at 01:45 PM

I think I remember Tharp uses 100 trades in his calculation.

So if SQN = 1.7, then the average of trades in terms of R multiples / stddev of r multiple trades = 0.17

If the risk is the same for all trades (so a dollar stop), the SQN is equal to 1 divided by the number in the TS strategy report, tab "Trade analysis", "coefficient of variation"

admin - 16-5-2018 at 01:54 PM

I think I remember Tharp uses 100 trades in his calculation.

So if SQN = 1.7, then the average of trades in terms of R multiples / stddev of r multiple trades = 0.17

If the risk is the same for all trades (so a dollar stop), the SQN is equal to 1 divided by the number in the TS strategy report, tab "Trade analysis", "coefficient of variation"

shouldnt we use in sample number of trades & the out of sample number of trades to give us IS sqn and OOS sqn?

cyrus68 - 17-5-2018 at 01:01 AM

I gave a web link, in an earlier post, where Howard Bandy explains why we should use the t statistic instead of Tharp's SQN. Here it is again:
https://www.aussiestockforums.com/threads/strategy-performan...
Bandy's command of these issues is better than Tharp's. He also explains how to calculate it.

It is also possible to do a Wilcoxon test of paired sample values of IS and OOS. My preference is for the t test, because we can rank the systems according to their quality, as well as get the average for all systems.

Of course, you can implement both tests. But I suspect your programmer will have hiccups.

Carl - 17-5-2018 at 08:35 AM

Hi Cyrus68,

The web link in your post seems to be truncated

The full link is:
https://www.aussiestockforums.com/threads/strategy-performan...

admin - 17-5-2018 at 11:52 PM

The video proposes a useful test that I may adopt because it is quite fast. However, I would like to note that it is a test of the viability of a particular strategy applied to a given symbol. For example, in the case of ES, if you selected 15-minute data for the primary symbol, different secondary datasets, 5 indicators and 2 operators – this would constitute a different strategy. And let’s not forget different historical periods over which the test may be carried out. The test may produce different results compared with your first strategy.

One thing I noticed is that, for ES, you set the max # of unique systems at 600. Yet at the end you had 1735. Either GSB didn’t stop or you let it run further. I have the same problem with max # completed. Typically, I set it at 5 million. So, if GSB is processing 35,000 a minute, it should stop after 2 hours and 20 minutes. That doesn’t happen. So, I have to stop it manually.

Currently I use a rather biased, but simple method, to gauge the overall viability of a strategy. I look at the proportion of WF results with PAS scores above 50. If, out of 60 WF, there are 40 with scores above 50 - and most are bunched closer to 100 – that is a good sign. If there are only 10 or 15 with acceptable scores, that is not a good sign. The test is biased because WF is carried out on systems that are pre-selected on the basis of attractive metrics.

The reason for the over shoot in systems generated is GSB stops workers after this number is achieved, but then the workers finish of the current population and generations. This doesnt matter the least to me. You can always delete all systems after the 600 or so you chose. However the more systems the better. Using less workers will give less over shoot. I used 10 workers for these tests typically.
It would be good too compare the two methods. This is all fairly un explored territory, but worth investigating.

cyrus68 - 18-5-2018 at 03:02 AM

Thanks Carl.
I didn't notice the bad link. I copied and pasted from my earlier post.
Your link is correct.

edgetrader - 18-5-2018 at 06:31 AM

Other metrics can of course be added and tried. I like the way Peter compares IS and OOS in his videos. He's looking at the metrics that matter for trading. Net profit, average trade, and profit factor are also what I would look at when evaluating a system.

The only thing I'd do differently is to divide OOS by IS (not other way round) before forming the overall average. This would output percentages, like OOS is 60% as good as IS, and a higher number would be better.

A great benefit of Peter's approach is that it includes market direction, trading sessions and other settings. For example VXX is a market that trends down. Long-only systems will be much harder to make than short-only systems. Since Peter's IS and OOS systems share the same settings, like short-only, the metrics of the IS systems are indeed the best benchmark to compare metrics of OOS systems. Looking at OOS just by itself with no suitable benchmark wouldn't account for things like short bias.

cyrus68 - 19-5-2018 at 12:31 AM

Just to explain, for everybody's benefit, the t test is a statistically sound method of summarising system quality. It is done on the P/L, by trade, of the system. It allows us to rank systems by their overall quality. We would also be able to see the average value for all systems and compare the IS and OOS averages. Its primary use is to evaluate OOS systems.

The Wilcoxon test is a measure of the quality of a strategy. It is calculated on paired values of IS and OOS systems. For example, using NP or PF. It is also a statistically sound test.

In my view, these tests would greatly enhance GSB's capabilities.

admin - 19-5-2018 at 02:10 AM

Just to explain, for everybody's benefit, the t test is a statistically sound method of summarising system quality. It is done on the P/L, by trade, of the system. It allows us to rank systems by their overall quality. We would also be able to see the average value for all systems and compare the IS and OOS averages. Its primary use is to evaluate OOS systems.

The Wilcoxon test is a measure of the quality of a strategy. It is calculated on paired values of IS and OOS systems. For example, using NP or PF. It is also a statistically sound test.

In my view, these tests would greatly enhance GSB's capabilities.

I'm going to chat to the programmer about this. This last week I UN-expectanly went to New Zealand for family matters, and have done just the bare minimum. Hence no great changes of recent. There is a new build but it just has some bug fixes. Still waiting on the version with multi time frame / symbol support for the same system.

cyrus68 - 20-5-2018 at 09:20 PM

On Sunday, while imbibing my favourite scotch, a thought occurred to me. Implementing the Wilcoxon test would be a real pain in the butt for the programmer. There is a simpler way. Why not enable exporting the results of running a strategy (the metrics panel) as a csv file? We can easily import the data into Excel and carry out the Wilcoxon test in Excel, or in any number of statistical packages.

My main objective is to do the test on the NP of IS and OOS. But you could also do the test on PF or anything else. The whole process of exporting the data and running the test could be done in 5 minutes. In addition, you can do a graph of the data.

As for the t test, it should be relatively simple to do the programming.

admin - 20-5-2018 at 10:19 PM

On Sunday, while imbibing my favourite scotch, a thought occurred to me. Implementing the Wilcoxon test would be a real pain in the butt for the programmer. There is a simpler way. Why not enable exporting the results of running a strategy (the metrics panel) as a csv file? We can easily import the data into Excel and carry out the Wilcoxon test in Excel, or in any number of statistical packages.

My main objective is to do the test on the NP of IS and OOS. But you could also do the test on PF or anything else. The whole process of exporting the data and running the test could be done in 5 minutes. In addition, you can do a graph of the data.

As for the t test, it should be relatively simple to do the programming.

I think excel export is a great idea.

Carl - 20-5-2018 at 11:33 PM

In episode 143 of the Better System Trader podcast Bruce Vanstone mentions a couple of other statistical methods to quantity system performance.

http://bettersystemtrader.com/the-dna-approach-to-trading-wi...

Anova
T-test
Ledoit-Wolf
Diebolt-Mariano

cyrus68 - 22-5-2018 at 02:31 AM

Hi Carl

I haven't listened to the podcast yet. It looks to be useful.
The statistical methods that you mentioned are all available in a number of packages.

admin - 22-5-2018 at 07:31 PM

This short video is to show how to get session times on your chosen market.
Attachment: Login to view the details
This is critical for all markets. However S&P500,S&P400, Nasdaq, mini Dow, Russell 2000 you can just use the equivalent of 830 to 1500 central USA time. (don't use the last 15 minutes of the day)

Also explained is a little on exchange time or local time. Most users are confused over this. You should be using local time because exchange time will not allow other symbols to be added to your chart that are on different time zones.
If you use GSB supplied data (typically central USA time) and your own local time that is not central USA, you MUST NEVER mix different local times. Other wise the data will be forward or backward looking by the difference in the local times.

admin - 22-5-2018 at 10:28 PM

The above video is now on youtube.
https://youtu.be/NFC7ego_Y70

The market validation video has been slightly tweaked, though the last bit on crude oil still could have been improved.
https://youtu.be/iG7MVOC56zk
If you are new to this thread, both videos are essential to view. The first one is simple and short.

admin - 28-5-2018 at 01:02 AM

Export of system metrics can now be done, but the requested newer system quality metrics have not been done. Priority is muti time frame /data support.
Get the exe in the member only section of beta builds.
http://trademaid.info/forum/viewthread.php?tid=39

admin - 28-5-2018 at 09:55 PM

I did some interesting market validation today on sqqq 15 minutes (short only)
Market validation failed quite badly, though some of my systems on sqqq (but not all of them) have gone well.
This implies that failing market validation doesn't mean GSB cant make systems on the market, but that it is less likely to work out of sample.
My gut reaction is market validation is going work much better with 14,15,16 or 29,30,31 minute bars. This is what is being worked on in GSB now.
The system on the left I traded on live account, the system on the right I did not trade.

admin - 30-5-2018 at 05:02 AM

The csv export can also be used to determine what indicators are most used, and never used.
Many years ago with the earlier generation of GSB, I found that common indicators with the odd less common indicator was the better than just allowing only the common indicators. ( I will need to re verify this) But this is also going to be useful to see how often your custom indicators are used.
Long term I want GSB to have its own analytic's.

rws - 30-5-2018 at 09:48 AM

Something like this including multiple timeframe and multiple tickers confirmation is usefull.

I did some interesting market validation today on sqqq 15 minutes (short only)
Market validation failed quite badly, though some of my systems on sqqq (but not all of them) have gone well.
This implies that failing market validation doesn't mean GSB cant make systems on the market, but that it is less likely to work out of sample.
My gut reaction is market validation is going work much better with 14,15,16 or 29,30,31 minute bars. This is what is being worked on in GSB now.
The system on the left I traded on live account, the system on the right I did not trade.

admin - 31-5-2018 at 05:48 AM

This is a bit different to normal monte carlo usage. I think its better. The randomize history data and randomize parameters are what I think is the best options for GSB. I doubt there is much need for it once we have multi time frame and market features working.
These are being worked on right now, but progress is slower than I would have liked.
EWFO has nth best feature (not related to gsb nth feature) which is similar to randomize indicator variables.

admin - 25-6-2018 at 12:28 AM

GSB 46.13 should be out later this week. It is very significant in improvements in out of sample results.
If we leave every 2nd day out of sample on ES.30 minutes bars, my tests show a 25% drop in out of sample results.
All systems no matter how bad are included in these results.
Criteria of a system was all systems with PF

Ive got that down to 5.4% drop in out of sample results (best case) compared to in sample.
However my favorite method - that gave the highest fitness (Net profit* average trade) gave 9.4 % drop in out of sample results
but average net profit of $38900 compared to $28,300 out of sample of 30 min bars.
This is research you can objectively test your self, with minutes of human time, and an hour or so CPU time per market.
Note all GSB code will expire at the end of the month. This is needed as there are compatibilty issues between the versions on the cloud.
The 2.8% drop was 5 indicators, with es 29,30,31 min bars with data2 $spx 29 30 31
But the highest result.... will publish soon.

Let me explain an issue.
A child wants a pet mouse. He says to his parents, only thing I want for my birthday is a horse.
Parents are not happy. They compromise with a pet mouse. That's what the child wanted. (Smart kid)

So lets say we build systems on ES,EMD,ER,DOW,NASDAQ. (Same parameters on each indice)
Degradation pretend is 1% less results out of sample compared to in sample. Awesome you think!
but out of sample profit is $10,000

Lets say we build system on ES29, ES30 ,ES 31
Degradation out of sample is say 10% (still good figure)
But out of sample profit is $30,000
We are better of using the es29,30,31 to build systems.

admin - 25-6-2018 at 05:13 AM

I have now done tests on
es30
es 29 30 31 with and without $spx and with and without $idx
es 28 30 32 with spx 28 30 32,
as above but 27 30 33
as above but 26 30 34
as above but 25 30 35
as above but 27,28,28,29 30,31,32, 33
as above but 25,26,27,28,28,29 30,31,32, 33 ,34,35
es with er emd data1 and $spx $idx $rut data2
as above but also ym and nq added
The results are riveting. Will publish in private forum tomorrow

rws - 25-6-2018 at 02:36 PM

A test with 5 ES tickers with random offset in a portfolio would be interesting.

A test on a portfolio will often show less NP because it is not so fitted to the curve.
But if the system on the portfolio also has a good result if you run it on only ES you could have more robust real OOS.
You only know how good something is after you tried it live 10 times.

GSB 46.13 should be out later this week. It is very significant in improvements in out of sample results.
If we leave every 2nd day out of sample on ES.30 minutes bars, my tests show a 25% drop in out of sample results.
All systems no matter how bad are included in these results.
Criteria of a system was all systems with PF

Ive got that down to 5.4% drop in out of sample results (best case) compared to in sample.
However my favorite method - that gave the highest fitness (Net profit* average trade) gave 9.4 % drop in out of sample results
but average net profit of $38900 compared to $28,300 out of sample of 30 min bars.
This is research you can objectively test your self, with minutes of human time, and an hour or so CPU time per market.
Note all GSB code will expire at the end of the month. This is needed as there are compatibilty issues between the versions on the cloud.
The 2.8% drop was 5 indicators, with es 29,30,31 min bars with data2 $spx 29 30 31
But the highest result.... will publish soon.

Let me explain an issue.
A child wants a pet mouse. He says to his parents, only thing I want for my birthday is a horse.
Parents are not happy. They compromise with a pet mouse. That's what the child wanted. (Smart kid)

So lets say we build systems on ES,EMD,ER,DOW,NASDAQ. (Same parameters on each indice)
Degradation pretend is 1% less results out of sample compared to in sample. Awesome you think!
but out of sample profit is $10,000

Lets say we build system on ES29, ES30 ,ES 31
Degradation out of sample is say 10% (still good figure)
But out of sample profit is $30,000
We are better of using the es29,30,31 to build systems.

admin - 26-6-2018 at 02:40 AM

Quote: Originally posted by rws

A test with 5 ES tickers with random offset in a portfolio would be interesting.

A test on a portfolio will often show less NP because it is not so fitted to the curve.
But if the system on the portfolio also has a good result if you run it on only ES you could have more robust real OOS.
You only know how good something is after you tried it live 10 times.

Lets say we build system on ES29, ES30 ,ES 31
Degradation out of sample is say 10% (still good figure)
But out of sample profit is $30,000
We are better of using the es29,30,31 to build systems.

Hope to add random noise in later builds. Im interested if its better than say 20 30 31 min data

admin - 3-7-2018 at 10:15 PM

Market validation on CL.30 900 to 230pm exchange time gave market degradation of 42.8% with secondary filter of CLose-CloseD
I got it down to an acceptable -33% with tweaks.
Fascinating, 29 30 31 min bars did NOT improve results, which they clearly did for ES.
I will publish all my findings in the private forum when done.
For a benchmark, ES30 with secondary filter close-Closed was -14.8% degradation. With tweaks got it down to an amazing -1.2%
This is a whole universe of stats to explore on many markets.

admin - 5-7-2018 at 03:49 AM

Ive got CL down to -28.7 oos vs IS degradation which is very acceptable.
ES down to -1.2%!!!
CL made no improvement going from 30 min bars to 29,30,31
NG went from -39.1% to -12.4% using 29,30,31 min bars. More improvement possible.
Amazing how CL didnt improve but ES & NG did.

This is thrilling.

admin - 6-7-2018 at 01:56 AM

ng I got down to -11.3%, but the fitness was 2.64 times higher. Muti bar time frames and another tweak was used.

cyrus68 - 6-7-2018 at 06:09 AM

Peter
In your tests you reported that 5 indicators produced better results than 3.
However, it is not clear whether you stress-tested the given systems. One sort of stress-test is walk forward, which we can do in GSB or elsewhere.

Another test is Monte Carlo simulation, where random noise is introduced in the indicator parameters. It isn't possible to do this outside GSB for the systems that it produces, but it is highly likely that a 5-indicator system will produce poorer results than a 3-indicator system.

The general modelling principle is that parsimony and simplicity in model construction is more likely to produce robust systems.

admin - 6-7-2018 at 03:59 PM