Observations are null even when they are included in 'rawMessage'

[brewin]

rawMessage shows the wind direction and speed as 320 degrees and 14 knots, but the windDirection and windSpeed values are null. I have noticed this happens very frequently. I have seen other instances where the temperature is in rawMessage, but temperature is null.

            "properties": {
                "@id": "https://api.weather.gov/stations/KIXD/observations/2024-06-06T17:53:00+00:00",
                "@type": "wx:ObservationStation",
                "elevation": {
                    "unitCode": "wmoUnit:m",
                    "value": 331
                },
                "station": "https://api.weather.gov/stations/KIXD",
                "timestamp": "2024-06-06T17:53:00+00:00",
                "rawMessage": "KIXD 061753Z 32014KT 10SM CLR 28/07 A2996 RMK AO2 SLP136 60000 T02780072 10283 20200 58003",
                "textDescription": "Clear",
                "icon": "https://api.weather.gov/icons/land/day/skc?size=medium",
                "presentWeather": [],
                "temperature": {
                    "unitCode": "wmoUnit:degC",
                    "value": 27.800000000000001,
                    "qualityControl": "V"
                },
                "dewpoint": {
                    "unitCode": "wmoUnit:degC",
                    "value": 7.2000000000000002,
                    "qualityControl": "V"
                },
                "windDirection": {
                    "unitCode": "wmoUnit:degree_(angle)",
                    "value": null,
                    "qualityControl": "Z"
                },
                "windSpeed": {
                    "unitCode": "wmoUnit:km_h-1",
                    "value": null,
                    "qualityControl": "Z"
                },
                "windGust": {
                    "unitCode": "wmoUnit:km_h-1",
                    "value": null,
                    "qualityControl": "Z"
                },
                "barometricPressure": {
                    "unitCode": "wmoUnit:Pa",
                    "value": 101460,
                    "qualityControl": "V"
                },
                "seaLevelPressure": {
                    "unitCode": "wmoUnit:Pa",
                    "value": 101360,
                    "qualityControl": "V"
                },
                "visibility": {
                    "unitCode": "wmoUnit:m",
                    "value": 16090,
                    "qualityControl": "C"
                },
                "maxTemperatureLast24Hours": {
                    "unitCode": "wmoUnit:degC",
                    "value": null
                },
                "minTemperatureLast24Hours": {
                    "unitCode": "wmoUnit:degC",
                    "value": null
                },
                "precipitationLastHour": {
                    "unitCode": "wmoUnit:mm",
                    "value": null,
                    "qualityControl": "Z"
                },
                "precipitationLast3Hours": {
                    "unitCode": "wmoUnit:mm",
                    "value": null,
                    "qualityControl": "Z"
                },
                "precipitationLast6Hours": {
                    "unitCode": "wmoUnit:mm",
                    "value": 0,
                    "qualityControl": "C"
                },
                "relativeHumidity": {
                    "unitCode": "wmoUnit:percent",
                    "value": 27.199250760371999,
                    "qualityControl": "V"
                },
                "windChill": {
                    "unitCode": "wmoUnit:degC",
                    "value": null,
                    "qualityControl": "V"
                },
                "heatIndex": {
                    "unitCode": "wmoUnit:degC",
                    "value": 26.835711042744446,
                    "qualityControl": "V"
                },
                "cloudLayers": [
                    {
                        "base": {
                            "unitCode": "wmoUnit:m",
                            "value": null
                        },
                        "amount": "CLR"
                    }
                ]
            }

[dwhitemv25]

I've seen it the other way as well, where there are weather elements but the METAR is missing. I've also seen where the METAR parser doesn't handle the gust indicator in the wind word and the parsed observations are wrong or missing. I have workaround code in one of my weather templates to parse the wind word myself.

I suspect the problem is upstream somewhere, either in the METAR ingestion or in MADIS. I'd have to set up some logging to see if there is any common factor in either issue.

[brewin]

Yep. Here is an empty rawMessage. I don't know how the Json is generated, but something doesn't seem right here.

            "properties": {
                "@id": "https://api.weather.gov/stations/AW020/observations/2024-06-06T19:45:00+00:00",
                "@type": "wx:ObservationStation",
                "elevation": {
                    "unitCode": "wmoUnit:m",
                    "value": 10
                },
                "station": "https://api.weather.gov/stations/AW020",
                "timestamp": "2024-06-06T19:45:00+00:00",
                "rawMessage": "",
                "textDescription": "",
                "icon": null,
                "presentWeather": [],
                "temperature": {
                    "unitCode": "wmoUnit:degC",
                    "value": 23.890000000000001,
                    "qualityControl": "S"
                },
                "dewpoint": {
                    "unitCode": "wmoUnit:degC",
                    "value": 14.390000000000001,
                    "qualityControl": "S"
                },
                "windDirection": {
                    "unitCode": "wmoUnit:degree_(angle)",
                    "value": null,
                    "qualityControl": "Z"
                },
                "windSpeed": {
                    "unitCode": "wmoUnit:km_h-1",
                    "value": 0,
                    "qualityControl": "S"
                },
                "windGust": {
                    "unitCode": "wmoUnit:km_h-1",
                    "value": null,
                    "qualityControl": "Z"
                },
                "barometricPressure": {
                    "unitCode": "wmoUnit:Pa",
                    "value": 101090,
                    "qualityControl": "S"
                },
                "seaLevelPressure": {
                    "unitCode": "wmoUnit:Pa",
                    "value": null,
                    "qualityControl": "Z"
                },
                "visibility": {
                    "unitCode": "wmoUnit:m",
                    "value": null,
                    "qualityControl": "Z"
                },
                "maxTemperatureLast24Hours": {
                    "unitCode": "wmoUnit:degC",
                    "value": null
                },
                "minTemperatureLast24Hours": {
                    "unitCode": "wmoUnit:degC",
                    "value": null
                },
                "precipitationLast3Hours": {
                    "unitCode": "wmoUnit:mm",
                    "value": null,
                    "qualityControl": "Z"
                },
                "relativeHumidity": {
                    "unitCode": "wmoUnit:percent",
                    "value": 55.310492588237999,
                    "qualityControl": "S"
                },
                "windChill": {
                    "unitCode": "wmoUnit:degC",
                    "value": null,
                    "qualityControl": "S"
                },
                "heatIndex": {
                    "unitCode": "wmoUnit:degC",
                    "value": 23.778773973137223,
                    "qualityControl": "S"
                },
                "cloudLayers": []
            }

[sullynole]

There are known issues with the backend observations ingest, mentioned on the Updates tab. Retrieving the data directly from MADIS is an option until those issues are resolved.

[scadergit]

Most of my investigation the past few weeks shows data missing from MADIS as the cause. It's become a hot topic for the beta weather site, so I look forward to getting this sorted out.