[md2orchestra] Generated XML omits the datatypes **MultipleStringValue** and **MultipleCharValue**

[outofphase]

XML generated by md2orchestra omits the datatypes MultipleStringValue and MultipleCharValue

Generating an XML file from a minimal Markdown file should include all required elements from the reference file specified as a parameter to md2orchestra, however although most Datatypes are included in the generated XML neither MultipleStringValue nor MultipleCharValue are.

Problem confirmed to exist in released v1.0.1 of tablature.

To verify/reproduce:

• see itiviti.orig.md which is a minimal Markdown ROE containing a field for each problem Datatype
• see itiviti.md which is generated from the original file by a couple of roundtrips md->xml-md (this file is stable during the round trip conversion and includes the fields, codesets, etc. pulled in from the reference file)
• generate itiviti.xml from itiviti.md using md2orchestra and confirm that the problem Datatypes are missing.

ubuntu@orchestra:~/ROE$ grep "Multiple" itiviti.xml
        <fixr:codeSet type="MultipleStringValue" id="1031" name="CustOrderHandlingInstCodeSet">
        <fixr:codeSet type="MultipleCharValue" id="18" name="ExecInstCodeSet">

ubuntu@orchestra:~/ROE$ grep "datatype name=" itiviti.xml
        <fixr:datatype name="String" added="FIX.4.2">
        <fixr:datatype name="Length" baseType="int" added="FIX.4.3">
        <fixr:datatype name="SeqNum" baseType="int" added="FIX.4.3">
        <fixr:datatype name="UTCTimestamp" baseType="String" added="FIX.4.2" updated="FIX.5.0SP2" updatedEP="256">
        <fixr:datatype name="data" baseType="String" added="FIX.2.7" updated="FIX.5.0SP2" updatedEP="208">
        <fixr:datatype name="NumInGroup" baseType="int" added="FIX.4.3">
        <fixr:datatype name="int" added="FIX.2.7" issue="SPEC-370" updated="FIX.5.0SP2" updatedEP="206">

itiviti.md
itiviti.orig.md

[kleihan]

Tested with Tablature v1.0.2, error confirmed. No reference file used, only the datatype name was added:

    <fixr:datatypes>
        <fixr:datatype name="Length"/>
        <fixr:datatype name="String"/>
        <fixr:datatype name="Qty"/>
        <fixr:datatype name="SeqNum"/>
        <fixr:datatype name="UTCTimestamp"/>
        <fixr:datatype name="MonthYear"/>
        <fixr:datatype name="Price"/>
        <fixr:datatype name="char"/>
        <fixr:datatype name="Exchange"/>
    </fixr:datatypes>

Markdown source included ### Codeset ExecInstCodeSet type MultipleCharValue (18) and code set was provided in XML as <fixr:codeSet type="MultipleCharValue" id="18" name="ExecInstCodeSet">. Only the datatype was not added.

[outofphase]

Since the original issue was opened I have found more about the issue. The generated XML includes datatypes of Fields in the Markdown but does not include the following:

• Union datatypes
• datatypes of Codesets (such as MultipleStringValue and MultipleCharValue)
• base datatypes.

I currently patch the XML after generation to include the missing datatypes before using it to generate our mapping files. For example currently I need to add:
{'Reserved1000Plus', 'Reserved4000Plus', 'MultipleCharValue', 'Pattern', 'Reserved100Plus', 'MultipleStringValue', 'Tenor'}

[kleihan]

Just to confirm that the datatypes are missing with and without using the "-r" option for a reference file. To be investigated, sorry for dropping the ball on this @outofphase.

[outofphase]

Using build #95 of the tools with my ROE I find that some of the missing datatypes are now present, but some are still absent. It appears the Union Datatypes ('Reserved1000Plus', 'Reserved4000Plus', Reserved100Plus', 'Tenor') are now added, but the Datatypes only used by Codesets are still not included.

I still need to patch the generated XML: Adding datatypes: {'Pattern', 'MultipleCharValue', 'MultipleStringValue'}

[donmendelson]

Tested itiviti.md markdown file, and it produces MultipleStringValue type when used with a reference file. That small file has no union datatypes. Other tests pick up union types from the reference file even when it is not explicit in the markdown. There are many code paths to discover types that are not explicit in input.