Fix metar response: Invalid field name(s) found: raw_text #79

rbuj · 2020-05-29T10:03:13Z

Old URI

https://aviationweather.gov/adds/dataserver_current/httpparam?dataSource=metars&requestType=retrieve&format=xml&hoursBeforeNow=3&mostRecent=true&stationString=LEDA&fields=raw_text

New URI

https://aviationweather.gov/adds/dataserver1_3/httpparam?dataSource=metars&requestType=retrieve&format=xml&hoursBeforeNow=3&mostRecent=true&stationString=LEDA&fields=raw_text

Test: Update weather info on mateweather-applet

msg->response_body->data contains raw_text element now

(gdb) set print elements 0
(gdb) print "%s", msg->response_body->data
$3 = 0x16464e0 "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<response xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:xsi=\"http://www.w3.org/2001/XML-Schema-instance\" version=\"1.2\" xsi:noNamespaceSchemaLocation=\"http://aviationweather.gov/adds/schema/metar1_2.xsd\">\r\n  <request_index>15318</request_index>\r\n  <data_source name=\"metars\" />\r\n  <request type=\"retrieve\" />\r\n  <errors />\r\n  <warnings />\r\n  <time_taken_ms>13</time_taken_ms>\r\n  <data num_results=\"1\">\r\n    <METAR>\r\n      <raw_text>LEDA 291130Z AUTO 18005KT CAVOK 27/12 Q1016</raw_text>\r\n    </METAR>\r\n  </data>\r\n</response>\r\n\r\n"
(gdb)

Closes #80

vkareh

LGTM, thanks for the quick fix!

muktupavels · 2020-05-29T10:57:21Z

I would not consider this as fix without having more info... That parameter was used to limit returned data. Now it returns more data, that wont be used...

vkareh · 2020-05-29T11:12:23Z

@muktupavels has a good point. Regardless of the amount of data, dataserver_current seems like a break in API, whereas dataserver1_3 seems like the actual locked-in version of the API that we've been using up to now (or compatible enough).

I would estimate that dataserver1_3 won't change but dataserver_current can change spec without notice, as it's likely a pointer to whatever new version of their API. Maybe it makes sense to lock in the 1.3 version

rbuj · 2020-05-29T12:01:08Z

The current server of AWC is TDS 1.3 (January 2010), which uses metar1_2.xsd. It should be available at the following URLs:

The 1st server doesn't accept fields=raw_text in Fields constraint, by removing it from the soap request, the response contains the METAR element, what includes raw_text.

	<xsd:element name="METAR">
		<xsd:complexType>
			<xsd:sequence >
				<xsd:element ref="raw_text" minOccurs="0" />
				<xsd:element ref="station_id" minOccurs="0"/>
                                <xsd:element ref="observation_time" minOccurs="0"/>
                                <xsd:element ref="latitude" minOccurs="0"/>
				<xsd:element ref="longitude" minOccurs="0"/>
				<xsd:element ref="temp_c" minOccurs="0"/>
				<xsd:element ref="dewpoint_c" minOccurs="0"/>
				<xsd:element ref="wind_dir_degrees" minOccurs="0"/>
				<xsd:element ref="wind_speed_kt" minOccurs="0"/>
				<xsd:element ref="wind_gust_kt" minOccurs="0"/>
				<xsd:element ref="visibility_statute_mi" minOccurs="0"/>
				<xsd:element ref="altim_in_hg" minOccurs="0"/>
				<xsd:element ref="sea_level_pressure_mb" minOccurs="0"/>
				<xsd:element ref="quality_control_flags" minOccurs="0"/>
				<xsd:element ref="wx_string" minOccurs="0"/>
				<xsd:element ref="sky_condition" minOccurs="0" maxOccurs="4"/>
				<xsd:element ref="flight_category" minOccurs="0"/>
				<xsd:element ref="three_hr_pressure_tendency_mb" minOccurs="0"/>
				<xsd:element ref="maxT_c" minOccurs="0" />
				<xsd:element ref="minT_c" minOccurs="0" />
				<xsd:element ref="maxT24hr_c" minOccurs="0" />
				<xsd:element ref="minT24hr_c" minOccurs="0"/>
				<xsd:element ref="precip_in" minOccurs="0" />
				<xsd:element ref="pcp3hr_in" minOccurs="0"/>
				<xsd:element ref="pcp6hr_in" minOccurs="0"/>
				<xsd:element ref="pcp24hr_in" minOccurs="0"/>
				<xsd:element ref="snow_in" minOccurs="0"/>
				<xsd:element ref="vert_vis_ft" minOccurs="0"/>
				<xsd:element ref="metar_type" minOccurs="0" />
				<xsd:element ref="elevation_m" minOccurs="0" />
			</xsd:sequence>
		</xsd:complexType>
	</xsd:element>

I have no objection to change the URI base of the request keeping the fields constraint to get only that element:

	<xsd:element name="raw_text" type="xsd:string"/>

vkareh

LGTM, this works as well

raveit65 · 2020-05-29T21:28:21Z

Big thank you for fast reaction to @rbuj
Guys, weather data is back! Do we need the PR?

rbuj · 2020-05-30T12:53:31Z

The AWC SOAP WS is back with the same conditions in both URIs, ie no changes need to be made in the request.

rbuj · 2020-06-02T13:09:02Z

Apparently it's still necessary for getting the weather info from METAR database

raveit65 · 2020-06-02T17:09:27Z

@rbuj
Do you think that your proposed code change would work with old and new behaviour from AWC ?
Sadly, we don't know if this is a temporary issue by AWC or a change of their API.

vkareh · 2020-06-02T17:44:08Z

I still think that moving to dataserver1_3 might be a good idea. I don't have any inside information, but from poking around at the website I can only assume that dataserver_current is the equivalent to a symlink to the latest version, which used to be 1.3, but now it's not?

If we want to be more paranoid, we can do both and change to dataserver1_3 and drop the fields=raw_text from the request...

N0rbert · 2020-06-03T08:15:06Z

What is really bad - this fix is needed for all Ubuntu versions.
Did you tried to contact aviationweather.gov admins to ask them changing URL back?
It will be safer to not change the code of libmateweather internals, I think.

What should we do if they change the URL again?
And anyway it would be good to have some configuration place for weather URL (like dconf, gsettings) to change it without recompilation.

raveit65 · 2020-06-03T08:41:57Z

What is really bad - this fix is needed for all Ubuntu versions.

Wrong, it is really bad that all Distros need this!

N0rbert · 2020-06-03T14:03:03Z

I tried to write an e-mail to NOAA and AviationWeather about reverting their API changes. Will inform you on any reply.
The contact form https://aviationweather.gov/contact does not work, shows error about their proxy server. The API seems to be documented at https://aviationweather.gov/dataserver .

raveit65 · 2020-06-03T14:27:51Z

Thank you.
I am testing this PR since one day now, currently they didn't fallback to old behaviour.

N0rbert · 2020-06-03T17:04:02Z

Got answer from NOAA, they are currently running on their backup system.
I invited Daniel from NOAA to this pull-request. And wrote to him that we want to have old URL/API back.

devietor · 2020-06-03T17:13:27Z

I'm the lead web developer at the Aviation Weather Center. I'm trying to get a handle on the change that cause the weather download to fail.

I've noticed that our backup ADDS Dataserver is having troubles with the field specifier. I'm looking into that problem. It appears from the URLs listed above that datasever1_3 works but dataserver_current doesn't. That's good information.

devietor · 2020-06-03T22:35:25Z

I'm not sure why the dataserver_current and dataserver1_3 are working differently. They're symlinks to the same Tomcat .war file. But they are different. I did notice that the war file on that server is older than we're using on the production server. But while it's operational, I can't change it to test this theory.

The dataserver_current is the preferred link but we keep dataserver1_3 and dataserver1_3_0 active. The latter two work fine on the backup server.

We will be on the backup server until probably next Monday or Tuesday. The server work at the Weather Service web farm will be going through this Friday. We might switch back on Friday but it's looking more like Monday.

Here are direct links:

current production:
https://aviationweather.cp.ncep.noaa.gov/adds/dataserver_current/httpparam?dataSource=metars&requestType=retrieve&format=xml&hoursBeforeNow=3&mostRecent=true&stationString=LEDA&fields=raw_text

current backup:
https://aviationweather-mo.weather.gov/adds/dataserver_current/httpparam?dataSource=metars&requestType=retrieve&format=xml&hoursBeforeNow=3&mostRecent=true&stationString=LEDA&fields=raw_text

We are in the process of upgrading our servers from Redhat 6 to 7. The new RH7 servers are:

primary:
https://aviationweather-cprk.ncep.noaa.gov/adds/dataserver_current/httpparam?dataSource=metars&requestType=retrieve&format=xml&hoursBeforeNow=3&mostRecent=true&stationString=LEDA&fields=raw_text

backup:
https://aviationweather-bldr.ncep.noaa.gov/adds/dataserver_current/httpparam?dataSource=metars&requestType=retrieve&format=xml&hoursBeforeNow=3&mostRecent=true&stationString=LEDA&fields=raw_text

satoshi · 2020-06-05T13:41:57Z

What's holding up merging this change in? There are two items "Pending" but if you look at the Details, they're green.

vkareh · 2020-06-05T13:57:09Z

@satoshi - the pipeline has nothing to do with it. If you read the thread, you will notice that there's still an issue in the API that we use to fetch weather data, and they are performing work on their servers that will extend possibly until Monday/Tuesday.

satoshi · 2020-06-05T14:21:59Z

@vkareh I read the thread and understand that the server work is still ongoing; however I thought this change would address the issue regardless.

you will notice that there's still an issue in the API that we use to fetch weather data

I don't know what you're talking about. You said yourself "I still think that moving to dataserver1_3 might be a good idea" which this change would do.

raveit65 · 2020-06-05T14:41:52Z

I don't know what you're talking about. You said yourself "I still think that moving to dataserver1_3 might be a good idea" which this change would do.

Nope, fixing all distros is a bad idea!
I prefer that it should work like before.

If you can't wait a few day, you need to patch your installation for yourself.
....... and revert it some days later for yourself :)

colibris79 · 2023-10-18T12:10:41Z

I confirm that the method provided by @devietor is working. Tested on Linux Mint 21.2 Victoria x64, MATE 1.26.0 I've made a patch which can be applied to libmateweather.so.1.6.9 with SHA-256 sum d3f66b2ff019991eaffe1c8d5ed984173971030f5fb6b0dd506216f105ea7690. Download patch, unpack and execute bspatch /usr/lib/x86_64-linux-gnu/libmateweather.so.1.6.9 ~/libmateweather.so.1.6.9 libmateweather.so.1.6.9.patch, then replace the library with file from your home directory to sudo mv -f ~/libmateweather.so.1.6.9 /usr/lib/x86_64-linux-gnu/

Tried it but it didn't work for me either. Guess because my libmateweather.so.1.6.9 has a different sha-256sum?

/usr/lib/x86_64-linux-gnu$ sha256sum libmateweather.so.1.6.9
585de2de62fac7fadd26cb8a9fbab6910139cbdf9afda5456a63b9c785181970  libmateweather.so.1.6.9

pepa65 · 2023-10-18T12:52:32Z

If you do strings libmateweather.so.1.6.9, does https://www.aviationweather.gov/adds/dataserver_current/httpparam appear?

Does grep 'https://www.aviationweather.gov/adds/dataserver_current/httpparam' libmateweather.so.1.6.9 give a match??

raveit65 · 2023-10-18T16:34:09Z

Fix is merged.
master
0cc07f7
1.26 branch
b75056f

devietor · 2023-10-18T18:56:39Z

We're in a transition period. This is probably a stop gap solution. It took me a while to find this email list. So I'll use it in the future to give you heads up on changes we make with the dataserver code. Dan. Web Developer Aviation Weather Center NWS

…

On Wed, Oct 18, 2023 at 4:56 AM Mike Ovchinnkov ***@***.***> wrote: I confirm that method provided by @devietor <https://github.com/devietor> is working. Tested on Linux Mint 21.2 Victoria x64, MATE 1.26.0 I've made a patch <https://github.com/mate-desktop/libmateweather/files/13003210/patch.zip> which can be applied to libmateweather.so.1.6.9 with SHA-256 sum d3f66b2ff019991eaffe1c8d5ed984173971030f5fb6b0dd506216f105ea7690. Download patch, unpack and execute bspatch /usr/lib/x86_64-linux-gnu/libmateweather.so.1.6.9 ~/libmateweather.so.1.6.9 libmateweather.so.1.6.9.patch, then replace the library with file from your home directory to sudo mv -f ~/libmateweather.so.1.6.9 /usr/lib/x86_64-linux-gnu/ — Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFZNPHZWLSQZSVS72MO47Y3X76RT5AVCNFSM4NN43HIKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZWHAYDSOBSGM3A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

colibris79 · 2023-10-18T22:03:38Z

If you do strings libmateweather.so.1.6.9, does https://www.aviationweather.gov/adds/dataserver_current/httpparam appear?

Thanks for the answer @pepa65 . It does not. I have the new server:
https://www.aviationweather.gov/cgi-bin/databin/databin.php

I have tried reinstalling with 'sudo apt-get install --reinstall libmateweather1' but it didn't work either

Here are the websites from 'strings libmateweather.so.1.6.9' :

https://www.aviationweather.gov/cgi-bin/databin/databin.php
http://www.metoffice.gov.uk/weather/europe/uk/%s.html
http://www.bom.gov.au/fwo/%s.txt
http://www.weather.gov/forecasts/xml/sample_products/browser_interface/ndfdBrowserClientByDay.php?&lat=%.02f&lon=%.02f&format=24+hourly&startDate=%04d-%02d-%02d&numDays=7
http://tgftp.nws.noaa.gov/data/forecasts/zone/%s/%s.txt
http://image.weather.com/web/radar/us_%s_closeradar_medium_usen.jpg

MikePooh · 2023-10-19T04:25:56Z

You may try to use HEX editor such as okteta. It allows to find the string in binary file and easily change it.

pepa65 · 2023-10-19T05:34:39Z

But https://www.aviationweather.gov/cgi-bin/databin/databin.php is the wrong link.

What does apt-cache policy libmateweather1 say?

msz59 · 2023-10-19T08:04:40Z

It should read https://www.aviationweather.gov/cgi-bin/data/dataserver.php

I was able to fix the problem on my Ubuntu 22.04, libmateweather1:amd64 1.26.0-1 doing:

cd /usr/lib/x86_64-linux-gnu
perl -p -i.orig -e 's+adds/dataserver_current/httpparam+cgi-bin/data/dataserver.php\0\0\0\0\0\0+' libmateweather.so.1.6.9

raveit65 · 2023-10-19T08:12:03Z

Why not recompiling your libmateweather package with given commits in our git?
If you're not able to do this open a report at your distro and ask the package maintainer to build a new distro release with a patch from our git.

raveit65 · 2023-10-19T08:14:43Z

@N0rbert
Can you inform ubuntu-Mate about the fix in git please?

msz59 · 2023-10-19T08:45:52Z

Thanks @raveit65.
Recompiling packages is probably not an everyday task for a typical Linux user. I did it many times in RHEL family (rpmbuild) but never with Debian packages.

N0rbert · 2023-10-19T09:13:49Z

@devietor, please if possible provide server-side solution with redirect or maybe symlink in server filesystem. It is impossible to patch all mateweather packages on all distros for all users.

msz59 · 2023-10-19T09:21:54Z

So it would be another reason for making that URL a configurable parameter.

colibris79 · 2023-10-19T09:53:40Z

But https://www.aviationweather.gov/cgi-bin/databin/databin.php is the wrong link.

What does apt-cache policy libmateweather1 say?

It says

apt-cache policy libmateweather1
libmateweather1:
  Installed: 1.26.0-1
  Candidate: 1.26.0-1
  Version table:
 *** 1.26.0-1 500
        500 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
        100 /var/lib/dpkg/status

pepa65 · 2023-10-19T10:05:47Z

@colibris79 If you can confirm that your newly installed libmateweather1 had the string https://www.aviationweather.gov/cgi-bin/databin/databin.php, then the following might work:

sed -i 's|https://www.aviationweather.gov/cgi-bin/databin/databin.php|https://www.aviationweather.gov/cgi-bin/data/dataserver.php|' /usr/lib/x86_64-linux-gnu/libmateweather.so.1.6.9

It will replace the wrong URL with the right one.

Edit: This will indeed work. The original lib file with your wrong URL has the exact sha256 sum that you reported.

pepa65 · 2023-10-19T11:00:57Z

I have a tiny patchscript at https://good4.eu/plw that should be safe to use (it only replaces a non-functional URL -if it exists- with a functional one -- if the binary /usr/lib/x86_64-linux-gnu/libmateweather.so.1.6.9 exists).

For quick application: wget -qO- good4.eu/plw |sh

(But you'll probably want to check out the 2-line code first!)

colibris79 · 2023-10-19T11:11:53Z

I have a tiny patchscript at https://good4.eu/plw that should be safe to use (it only replaces a non-functional URL -if it exists- with a functional one -- if the binary /usr/lib/x86_64-linux-gnu/libmateweather.so.1.6.9 exists).

For quick application: wget -qO- good4.eu/plw |sh

(But you'll probably want to check out the 2-line code first!)

Thank you so much! I can always reinstall or make a backup of the file.

pepa65 · 2023-10-19T11:16:55Z

Thank you so much! I can always reinstall or make a backup of the file.
You can always just reinstall like: sudo apt-get --reinstall install libmateweather1

If you use the plw utility, make sure you have the original file, it won't work on your modified one..!

colibris79 · 2023-10-19T13:57:24Z

Thank you so much! I can always reinstall or make a backup of the file.
You can always just reinstall like: sudo apt-get --reinstall install libmateweather1

If you use the plw utility, make sure you have the original file, it won't work on your modified one..!

Reinstalled (again) and patched it with the 'sed' command and now is working again. I have no clue why first two times it didn't. Will keep your patchscript just in case. Thanks again

msz59 · 2023-11-02T10:07:51Z

My weather applet stopped working again, 2 or 3 days ago. Has the URL changed once again?

MikePooh · 2023-11-02T10:09:22Z

My weather applet stopped working again, 2 or 3 days ago. Has the URL changed once again?

Mine is working fine.

cwendling · 2023-11-02T10:12:27Z

My weather applet stopped working again, 2 or 3 days ago. Has the URL changed once again?

Did you hotpatch the binary? If so, possibly your system had an upgrade/reinstall of the library, overriding your changes

MikePooh · 2023-11-02T10:25:13Z

If your package manager is APT you can check the last installed packages with grep install /var/log/dpkg.log | awk '{print $5}' | cut -d ":" -f1 maybe libmateweather is among that

kikislater · 2023-11-02T10:38:47Z

I have a tiny patchscript at https://good4.eu/plw that should be safe to use (it only replaces a non-functional URL -if it exists- with a functional one -- if the binary /usr/lib/x86_64-linux-gnu/libmateweather.so.1.6.9 exists).

For quick application: wget -qO- good4.eu/plw |sh

(But you'll probably want to check out the 2-line code first!)

Thanks, it works in manjaro and archlinux as well !
Nice discussion here about strings, okteta and editing a binary with sed, I was never thinking it will be possible to use sed in a binary file ...

msz59 · 2023-11-02T10:56:20Z

Did you hotpatch the binary? If so, possibly your system had an upgrade/reinstall of the library, overriding your changes

No, there were no updates to libmateweather on my Ubuntu 22.04, still version 1.26.0-1. Just in case, I have reinstalled the package and hotpatched again. No success.

kikislater · 2023-11-02T11:03:07Z

Even with killall mate-panel ? Panel will reload by itself

Info about URL change: mate-desktop/libmateweather#79 (comment) https://gitlab.gnome.org/GNOME/libgweather/-/issues/232 https://gitlab.gnome.org/GNOME/libgweather/-/issues/236

vkareh approved these changes May 29, 2020

View reviewed changes

rbuj force-pushed the weather-metar-soap branch from 6d54cca to 01319a7 Compare May 29, 2020 12:16

vkareh self-requested a review May 29, 2020 12:41

vkareh approved these changes May 29, 2020

View reviewed changes

rbuj closed this May 30, 2020

rbuj deleted the weather-metar-soap branch May 30, 2020 12:54

rbuj restored the weather-metar-soap branch June 2, 2020 13:08

rbuj reopened this Jun 2, 2020

Fix metar response: Invalid field name(s) found: raw_text

14270e7

rbuj force-pushed the weather-metar-soap branch from 01319a7 to 14270e7 Compare June 2, 2020 14:00

satoshi mentioned this pull request Jun 4, 2020

Clock applet stopped showing weather (Fedora) #80

Closed

MikePooh mentioned this pull request Oct 18, 2023

Dataserver url changed #124

Merged

acoul mentioned this pull request Nov 19, 2023

libmateweather.so.1.6.9 not working void-linux/void-packages#47306

Closed

procule mentioned this pull request Dec 8, 2023

New hardcoded dataserver url points to a 308 Permanent Redirect #131

Merged

Fix metar response: Invalid field name(s) found: raw_text #79

Fix metar response: Invalid field name(s) found: raw_text #79

Conversation

rbuj commented May 29, 2020 • edited Loading

vkareh left a comment

Choose a reason for hiding this comment

muktupavels commented May 29, 2020

vkareh commented May 29, 2020

rbuj commented May 29, 2020 • edited Loading

vkareh left a comment

Choose a reason for hiding this comment

raveit65 commented May 29, 2020

rbuj commented May 30, 2020

rbuj commented Jun 2, 2020

raveit65 commented Jun 2, 2020

vkareh commented Jun 2, 2020

N0rbert commented Jun 3, 2020

raveit65 commented Jun 3, 2020

N0rbert commented Jun 3, 2020

raveit65 commented Jun 3, 2020

N0rbert commented Jun 3, 2020

devietor commented Jun 3, 2020

devietor commented Jun 3, 2020

satoshi commented Jun 5, 2020

vkareh commented Jun 5, 2020

satoshi commented Jun 5, 2020 • edited Loading

raveit65 commented Jun 5, 2020

colibris79 commented Oct 18, 2023 • edited Loading

pepa65 commented Oct 18, 2023

raveit65 commented Oct 18, 2023

devietor commented Oct 18, 2023 via email

colibris79 commented Oct 18, 2023 • edited Loading

MikePooh commented Oct 19, 2023 • edited Loading

pepa65 commented Oct 19, 2023

msz59 commented Oct 19, 2023 • edited Loading

raveit65 commented Oct 19, 2023

raveit65 commented Oct 19, 2023

msz59 commented Oct 19, 2023

N0rbert commented Oct 19, 2023 via email

msz59 commented Oct 19, 2023

colibris79 commented Oct 19, 2023

pepa65 commented Oct 19, 2023 • edited Loading

pepa65 commented Oct 19, 2023

colibris79 commented Oct 19, 2023 • edited Loading

pepa65 commented Oct 19, 2023

colibris79 commented Oct 19, 2023 • edited Loading

msz59 commented Nov 2, 2023

MikePooh commented Nov 2, 2023

cwendling commented Nov 2, 2023 • edited Loading

MikePooh commented Nov 2, 2023

kikislater commented Nov 2, 2023

msz59 commented Nov 2, 2023

kikislater commented Nov 2, 2023

rbuj commented May 29, 2020 •

edited

Loading

rbuj commented May 29, 2020 •

edited

Loading

satoshi commented Jun 5, 2020 •

edited

Loading

colibris79 commented Oct 18, 2023 •

edited

Loading

colibris79 commented Oct 18, 2023 •

edited

Loading

MikePooh commented Oct 19, 2023 •

edited

Loading

msz59 commented Oct 19, 2023 •

edited

Loading

pepa65 commented Oct 19, 2023 •

edited

Loading

colibris79 commented Oct 19, 2023 •

edited

Loading

colibris79 commented Oct 19, 2023 •

edited

Loading

cwendling commented Nov 2, 2023 •

edited

Loading