Toolbox stores *a lot* data about export mappings

[spine-o-bot]

In GitLab by @ererkka on May 27, 2020, 12:49

In the export mapping setting in project.json file, there is a key "records" which seems to hold a lot of the data itself, see example below:

"settings": {
            ...,
            "records": {
              "grid": [
                        [
                            "ammonia"
                        ],
                        [
                            "bld_heat"
                        ],
                        [
                            "elec"
                        ],
                        [
                            "fuel"
                        ],
                        [
                            "hydrogen"
                        ],
                        [
                            "ind_heat"
                        ],
                        [
                            "natGas"
                        ],
                        ...
                      ]
            }
...
}

I wonder why the actual record items are stored here, shouldn’t the data be in the Data Store of the project?

Also under key "indexing_settings" there is long array of arrays having the index values.

"settings": {
    ...,
    "indexing_settings": {
        "ts_unit": {
            "indexing_domain": {
                "name": "t",
                "description": "",
                "indexes": [
                    [
                        "t000000"
                    ],
                    [
                        "t000001"
                    ],
                    [
                        "t000002"
                    ],
                    [
                        "t000003"
                    ],
                    [
                        "t000004"
                    ],
                    [
                        "t000005"
                    ],
                    ...
                  ],
                "pick_list": [
                    true,
                    true,
                    true,
                    true,
                    true,
                    true,
                    true,
                    true,
                    true,
                    true,
                    ...
                   ]
                 },
           "index_position": 3
        },
        ...
}

The sub-key "pick_list" in this case has more than 10,000 rows of true. I wonder if this is because of the specific case or if it’s because how the exporter stores its settings. In this case there is a lot of data also, but the way the exporter is designed makes it store a lot of data into the settings.

The source Data Store database file in this case is about 9.3 MB, but the project.json has swollen to almost 19 MB!

See also #714.

[spine-o-bot]

In GitLab by @PekkaSavolainen on Aug 25, 2020, 15:58

assigned to @soininen

[soininen]

This issue is outdated. Exporter's specification file looks nowadays completely different, and doesn't scale with input database size.