Python: TestGenerator: unmatched data error (low value X at Y is not reached from the least timeframe, low price Z mismatches)

[kenorb]

After fixing #85, this error start happening at M30 timeframe when backtesting:

TestGenerator: unmatched data error (low value 1.08294 at 2015.07.20 09:15 is not reached from the least timeframe, low price 1.08391 mismatches)

Similar error at TestPeriod M5 (plus volume error, similar as per: #70):

TestGenerator: unmatched data error (low value 1.08391 at 2015.07.20 09:15 is not reached from the least timeframe, low price 1.08474 mismatches)
TestGenerator: unmatched data error (volume limit 590 at 2015.07.20 09:30 exceeded)

And M15:

TestGenerator: unmatched data error (low value 1.08294 at 2015.07.20 09:15 is not reached from the least timeframe, low price 1.08391 mismatches)

---

This can be tested using Docker container, e.g.

docker run ea31337/ea-tester run_backtest -v -t -T M30 -e MACD -y 2015 -m 7 -D5 -b DS -M 4.0.0.1010

• Try testing also with M5 and M15 (-T). And all months: -m 1-12. Remove -b and -M params to avoid re-downloading the data when running it again.
• You can login to Docker container by: docker run -it ea31337/ea-tester bash.
• Add -x for debug.

After fix in convert_csv_to_mt.py, clone and convert the CSV data e.g. via (see: Makefile as example):

find . -name '*.csv' -print0 | sort -z | xargs -r0 cat | tee all.csv > /dev/null
./convert_csv_to_mt.py -v -i all.csv -f hst4 -t M1,M5,M15,M30,H1,H4,D1,W1,MN

and move generated files and replace with the existing HST files in ~/.wine/drive_c/Program Files/MetaTrader 4/history/default and re-run the test.

To read the converted HST files, you can use convert_mt_to_csv.py tool, e.g.

./convert_mt_to_csv.py -i EURUSD15.hst -f hst4

---

The problem is with low value mismatch as per error. The fix shouldn't break the fixed logic in dbb096e, which wasn't working before as per #85. The solution should include the proper calculation of OLHCV values (open/low/high/close/volume) across the different timeframes to avoid any unmatched data errors.

See also:

• Generated FXT/HST files.
• Failing CI tests.

Requirements

Basically these commands needs to work with no data error after complete testing:

• docker run ea31337/ea-tester run_backtest -v -t -M4.0.0.1010 -d 1000 -p EURUSD -m 1-12 -y 2015 -s 10 -b DS -D5 -e TestFXTHeader (see: Build #1087).
• docker run ea31337/ea-tester run_backtest -T M1 -v -t -e MA -y 2015 -m 7 (this works)
• docker run ea31337/ea-tester run_backtest -T M5 -v -t -e MA -y 2015 -m 7
• docker run ea31337/ea-tester run_backtest -T M15 -v -t -e MA -y 2015 -m 7
• docker run ea31337/ea-tester run_backtest -T M30 -v -t -e MA -y 2015 -m 7
• docker run ea31337/ea-tester run_backtest -T M30 -v -t -e MA -y 2015 -m 1-12 (test all months just in case)

Resources

• THE ALGORITHM OF TICKS' GENERATION

--

Est. 10h

[paulohrpinheiro]

I'm very sorry, and I want to apologize, but I'm not able to create the environment to run the tests. I would take a long time to get set up, and a whole day is gone, so I'm giving up the task. Once again, I'm sorry, thank you.

[kenorb]

I'm sorry about it. If you're still interested, I can give you the access to the remote Linux where the issue can be easily reproduced. Or this can be tested using Travis CI.

[ghost]

Is this still an issue?

[kenorb]

@Ryu0 I think it is. If the steps aren't clear, I can double check them this evening. I think now the approach is to use Docker container, but syntax using run_backtest.sh in order to reproduce issue should be the same.

[kenorb]

@Ryu0 I've tested and the problem can be still reproduced. I've updated description to use docker syntax.

Here is the example:

docker run ea31337/ea-tester run_backtest -v -t -M4.0.0.1010 -d 1000 -p EURUSD -m 1-12 -y 2015 -s 10 -b DS -D5 -e TestFXTHeader

The main error which appears:

TestGenerator: unmatched data error (low value 1.08294 at 2015.07.20 09:00 is not reached from the least timeframe, low price 1.08391 mismatches)

Please check also other timeframes (use -T). For example this timeframe works fine (it doesn't generate data error):

/opt/scripts/run_backtest.sh -T M1 -v -t -e MA -y 2015 -m 7

In summary, the error happens, because of some problem in the data which has been converted from CSV into FXT and MT4 complains about it, and it doesn't happen for other years for some reason. Basically I need to find the source of this error and fix it.

To test your fixes, you can login to Docker container by:

docker run -it ea31337/ea-tester bash

then you can run:

/opt/scripts/run_backtest.sh -v -t -M4.0.0.1010 -d 1000 -p EURUSD -m 1-12 -y 2015 -s 10 -b DS -D5 -e TestFXTHeader

Once data has been downloaded first time, second run should use the existing files.

To replace FXT data, you can generate using script and replace in the platform dir.

$ ls ~/".wine/drive_c/Program Files/MetaTrader 4/tester/history"
EURUSD30_0.fxt

To test only specific month, change it using -m param.

If you've any questions, please ask. For quicker answer, contact me at http://t.me/kenorb

[ghost]

Hi. Is this still an issue, right?

[kenorb]

@brunofarina I think it's still the issue.

[artskeem]

Lol so is this the code challenge? Or we haven't gotten there yet

[ghost]

@kenorb I'm able to working on this issue.