More concise syntax for "Output a Field when Present"

[Shadow-Devil]

Hi I have an idea for a new language syntax: I found a common pattern in our code base:

 {
   (key: value) if (value != null)
 }

which can become quite long e.g.

 {
   (key: payload.very.deeply.nested.value) if (payload.very.deeply.nested.value != null)
 }

I know that there is the writer property skipNullOn='objects' but this treats every entry the same, so no fine-grained control is possible what should be skipped and what not.
My idea would be to add the following syntax, which is a shortcut to the above syntax:

{
   key?: payload.very.deeply.nested.value
 }

If you need a key with a question mark at the end, it already has to be wrapped into a string, so the question mark could be after the string like this: "key?"? : payload.value.
I think this is quite intuitive, since in RAML and for dataweave types this syntax specifies an optional key.

What do you think about this feature? 🙂

Idea was originally posted on slack: https://dataweavelanguage.slack.com/archives/C044FKW9XUK/p1670859340600209?thread_ts=1670859340.600209&cid=C044FKW9XUK
and https://dataweavelanguage.slack.com/archives/C03ULMDEHQX/p1725648135173439?thread_ts=1725648135.173439&cid=C03ULMDEHQX

[ryan-slack-big]

This would also help reduce clutter when we have more than a few "optional" fields:

Currently we have:

payload map {
    name: $.name,
    (street: $.address.street) if ($.address.street?),
    (unit: $.address.unit) if ($.address.unit?),      
    (region: $.address.state default $.address.province) if  ($.address.state? or $.address.province?),
    (postalCode: $.address.zip) if ($.address.zip?),      
    (country: $.address.country) if ($.address.country?)
}

Would be easier to read/write this though:

payload map {
    name: $.name,
    street?: $.address.street,
    unit?: $.address.unit,
    region?: $.address.state default $.address.province,
    postalCode?: $.address.zip,
    country?: $.address.country
}

[ryan-slack-big]

Optional field syntax must clearly distinguish it's behavior between a null value and "key not present".
i.e. equivalent to behavior of (insurance: $.insurance) if($.insurance != null) vs (insurance: $.insurance) if($.insurance?)

The example I provided uses "key not present" mostly to match Output a Field When Present.

In practice however, the "not null" behavior @Shadow-Devil provided is likely the more common use case, and is certainly more intuitive. While "key not present" is more technicaly correct, a concise easy to read/write syntax should follow the principle of least astonishment.

[machaval]

So If I understand correctly key?: value is a sugar syntax over (key: value) if value != null. Whatever the value and even if the key has attributes or not. On the key side it reads ok as we are saying the key may or many not be present. What is weird to me is the value semantics, as it is only if it is null, though I understand this is a common case that may be enough to have it is special syntax.

Again I think the idea is good my only problem is adding more syntax to DW (that is quite convoluted with special characters), vs just a function that is removeNullKeyValues.

[Shadow-Devil]

The problem with a function like removeNullKeyValues is that you loose the fine grained control on which fields it should apply. IMO this syntax is fine, since it already exists in RAML and DW Types (to denote optional fields) so it would feel familiar for the average Mulesoft Developer. The special treatment for null values is quite similar to the default keyword so I also think this is reasonable.

[ryan-slack-big]

The (key: value) if value != null syntax results in an optional field that is also nullable. e.g. Union<String, Null>? This complicates an otherwise simple type of just String, pushing developers away from trusting the type-system to be help them, and confusing the tooling.

A (mostly) real example:

1. Developer adds a Mule flow component that consumes a payload with an optional "name" field is using the expression { ... (name: payload.name) if (payload.name != null) ... }.
2. This is consumed by a final transform at the end of the API flow, which at development time picks up the output type from RAML to be a json object with an optional "name" property. There are quite a few other fields as well, and it's mostly a 1:1 mapping, so the developer does the smart thing and uses the drag-and-drop field mapping in Studio. This will map the name using the code {... name: payload.name as String ...}, which looks about right to the developer and they move on.
3. Eventually a payload comes along with no "name" field (it is optional after all), and Mule produces error messages about "Cannot coerce Null to String".
4. Chaos Ensues, (ok, maybe not....)

TLDR:
The tooling infers the wrong type signature and generates bugged code that looks ok to many developers.

With the name?: payload.name syntax however, the tooling could infer the type accurately and generate the correct code name?: payload.name.

[machaval]

I agree that we should fix the
Developer adds a Mule flow component that consumes a payload with an optional "name" field is using the expression { ... (name: payload.name) if (payload.name != null) ... }.
Without the need of a new syntax. This should work and it shouldn't infer nullable value, though it is easier to say than done as this can be simple to fix with simple expression complicated to do with very big and nested expressions.