BigDog
Junior Member
Posts: 20
Registered: 25-4-2018
Member Is Offline
Mood: No Mood
|
|
Tradestation WFO vs. GSB WFO
There are three great features that most impress me about GSB:
1) Speed (esp with multiple workers), which is on par with TSL
2) The ability to process multiple data streams, which opens up much richer possibilities for trading systems to be created by genetic programming
3) In-built WFO/Analysis
In this post I am focusing on the WFO feature.
TS incorporates a quite stringent WFO methodology that involves several rigorous tests. These tests are applied to the system using a range of
in-sample/OOS sample sizes and WF runs, in what is referred to as a "cluster". Only if the system passes the WFO tests for a sufficient number of
runs in the cluster is the system deemed sufficiently reliable to be worth trading.
The GSB approach is much more subjective, and invites the user to evaluate the WFO results heuristically, for example by comparing the slope of the
WFO equity curve to the equity curve of the non-optimized system.
This contrast in approaches can lead to very different outcomes. To illustrate this I have used the CL system that Peter developed in GSB, which
appears to show outstanding performance (see his post for details)
I assume Peter put the CL system through the usual WF analysis in GSB and was satisfied with the results.
What I did was run the system through the TS WFO cluster analysis, which examines the performance of the system on its five standard tests for various
combinations of OOS sample size and # runs. What I found was the system failed the tests, for every such combination.
Looking at the test results for one of the cluster analysis cells makes the reason for the failure clear. The system failed the "WF Efficiency" test,
producing profits OOS at less than 50% of the rate produced during the in-sample periods. That raises doubts about the reliability of the system
going forward.
The same outcome was seen for every OOS/#runs combination in the cluster analysis:
Now, you can certainly take issue with the criteria used for the tests in the TS WFO analysis. And you might certainly choose to overlook some level
of failure in testing. But to produce a fail in every cell of the cluster analysis - all for the same reason - raises serious questions about the
reliability of the system and its profitability, going forward. I, for one, would not be prepared risk my capital trading such a system - I have
rejected many other systems that demonstrated a much better success rate than this system.
Finally, a suggestion for Peter: whether you agree with the analysis in this specific case, or not, why not consider incorporating a set of standard
WFO tests in GSB and maybe even adopt the same cluster analysis approach as Tradestation?
|
|
|
BigDog
Junior Member
Posts: 20
Registered: 25-4-2018
Member Is Offline
Mood: No Mood
|
|
In some cases it helps if you exclude the weights from the TS WFO and optimize the remaining strategy parameters.
So, for example, using this method both the @ES and @NQ demo systems pass the WFO Cluster test with flying colors. The @S demo system, which
previously failed overall, now passes, although with some mixed results.
The CL system is still a bust on the WFO Cluster test, however.
|
|
|
BigDog
Junior Member
Posts: 20
Registered: 25-4-2018
Member Is Offline
Mood: No Mood
|
|
In fact, I find similar results for all the energy contracts: the sample systems fail the WFO Cluster test due to insufficient WF Efficiency.
In some cases the efficiency is as low at 15%, which means that OS performance is on average around only around 1/6 of IS performance, which is not an
encouraging sign.
Peter already pointed out how challenging it is develop a robust system in CL. What these results suggest to me is that we may be looking at a case
of "fooled by randomness" here. What I mean by that is simply that if you build enough systems (and with GSB it is easy to build a great number)
eventually you are certain to come across one with a great backtest performance, just by random chance. Such a system is likely to fall over more or
less immediately in live trading. I suspect, unfortunately, that may be the case with Peter's CL system and also the NG and RB systems, which
likewise failed the WFO Cluster test (for the same reason, i.e. poor Efficiency).
I have always found building systems for the energy markets far more challenging than for equity markets. Despite all its advantages, it doesn't look
as if GSB is going to be able to change that calculus.
|
|
|
admin
Super Administrator
Posts: 5069
Registered: 7-4-2017
Member Is Offline
Mood: No Mood
|
|
You have some good comments. I did put a comment on CL system that CL failed my market analysis testing & I stopped trading it shortly after release.
That was a few months ago and last week I made very significant advance on a very specific way to test a market. See the last video here.
http://trademaid.info/forum/viewthread.php?tid=117 Search for Attachment: Market_testing1f_r1.01.mp4 (126.6MB) (it will be updated later)
SO if the market analysis fails, do we over come this by more CPU power to find something that works, or do we avoid the market till GSB architecture
gets improved (Additional secondary filters) or other.
At the moment I'm thinking other. Short term GSB will have multi bar / market support.
My tests on gold showed that 20 30 31 min bars made very big difference to out of sample testing.
WF efficiency to me is almost useless. My EWFO (a big step up from ts WFO) has it though.
Its basically the ratio of the first period profits to last. It doesn't take into account how good the markets truly are in the first period. IN your
CL tests, the first period is 2007 to 2008. This is extremely lucrative period and years after this are not. This is why efficiency failed. the nq and
es start at 2001 and would likely fail the same test if starting at 2007.
I personally see little use for cluster analysts. The reason is if you hit high parameter stability, the clusters are all the same. SO I think this
approach is simpler and better.
Last week I did testing on 100's of systems I walk forwarded. Only criteria was a fixed performance metrics.
The OOS results were in direct proportion to WF stability or parameters results.
Your comments are excellent and might be better of it this section.
http://trademaid.info/forum/viewthread.php?tid=117
People love black and white pass / fail concept. Well trading has black , white and grey.
When you judge any great persons life, you will see the good, the bad and the in-between. Same with trading systems.
I also agree that CL is a hard market to crack. I suspect multi time frame bars are going to fix this issue.
You should also look at EWFO. It has many more features, but not cluster analysis.
http://trademaid.info/tools-ewfo.htm
|
|
|
|
![[*]](images/davis/default_icon.gif){kind=icon}
