Task and Dataset definition

[sebffischer]

I wanted to start a general discussion on the metadata of tasks and datasets.

One thing I have noticed is that it would be great to have more fields available in the task. There are a couple of concrete examples that I have in mind:

Multiple target variables

There is a datasets where different target variables are possible (y1, y2).
Currently, one has (at least I don't know how else to do it) to upload the dataset twice (once ignoring y1 and once ignoring y2). It would be nice if one could set a field "features" in a task. That way one would only have to upload the dataset once, and simply exclude y1 and y2 respectively in the "features" field.

Group column

In general I think there are cases, where there are columns that I want to have available via download, but that are neither features nor the target variable. One example is a group id.
Let's say I have a dataset with the column "group", that I don't want to use as a feature, but I want to e.g. use it to locally split my data (not using an OpenML task). If I include it in the dataset (i.e. don't set it as ignore column) then it is included in every OpenML task that is created for the dataset afaik.
If I don't include it, it is not available as download and I cannot use it to create the datasplits.

Positive class

For classification problems it would be nice to be able to set a "positive class".

Other fields / properties that would be possible but not so important imo are e.g. "weights" or "stratum".

[PGijsbers]

Multiple target variables

I think the procedure you describe here is correct, and I agree that tasks should be able to declare which features to use. Given the schema below, it would make sense to add this to task.task_description.input.data_set. It makes sense to add an ignore_features attribute, which are by default populated with the dataset.ignore_attributes and dataset.row_id but may be specified by the user directly.

Group column

I believe we discussed this during the OpenML Hackathon or one of the engineering meetings and agreed this is something we want.

[sebffischer]

What needs to be done do make this possible? :)

[PGijsbers]

@joaquinvanschoren or @janvanrijn might be better able to answer that.

[joaquinvanschoren]

My thoughts about this are:

• We should start using official schema to define metadata (for datasets, tasks, runs, flows,...). E.g. we could use JSON Schema. Then people could make concrete proposals and we could discuss them easily. Ideally we also have a convenient tool yet that helps you browse JSON schemas online. Any ideas?
• When we have consensus, the REST API needs to be updated to return the extra information and to update the creation of tasks. An interesting question here is whether we do that in the current PHP API or whether we hold off until we have the new Python version. If it's a minor change, we can probably do it in PHP. Next, the language bindings (Python, R,...) need to be updated to work with the new schema.
• We should have proper metamodel versioning, or maybe just API versioning. I.e. changes would be done in v.2 of the API, or maybe v.2.0.

We haven't had a proper procedure in place, which is why it's so hard to change things. We should fix this soon.

When defining new versions of the metadata schema, it would be good to also look at standardization initiatives that have emerged in recent years:

• For datasets, we should probably make the data schema compatible with schema.org's definition of a dataset, and maybe even with DCAT-AP, a metadata model for data catalogs. We could create a small group around this to work on that.
• For tasks, I'm not sure whether there are any emerging standards. MLCommons has a working group around this - they used PapersWithCode to extract about 100 different machine learning tasks. It's more of a taxonomy than a schema of all inputs and outputs, but it may be helpful.

[sebffischer]

And probably I am blind, but I cannot find a definition of a dataset on schema.org, can you send a link to what you are referring to?

And a general question: would these changes still be backwards compatible with the current schemata?

[sebffischer]

Regarding the browsing of schemata: Maybe just use pull reuqests and other people review / comment on those PRs?

[joaquinvanschoren]

We plan to make a roadmap next week.
Here is the link: https://schema.org/Dataset

Browsing: I don't mean reviewing, but simply understanding a schema.
We have the OpenAPI explorer, see Pieter's screenshot above, or https://www.openml.org/apis > REST API > scroll to bottom
The format used by schema.org is also not bad, although slightly overwhelming, and I don't know how they generate that :)

[sebffischer]

I prefer the OpenAPI explorer to schema.org. Is there any problems with simply going for OpenAPI explorer? :)

[joaquinvanschoren]

No. Creating an OpenAPI description is quite a bit of work though.
We currently export these automatically from the PHP implementation. We should discuss how to do this going forward.