Ha historical sensor (and MFA support)

[shtrom]

Fix for #6 (historical sensor), and also #10 (mfa)

[shtrom]

@LeighCurran I'm still letting this soak on my HA instance (see #6), but I think this is now pretty much good to go.

[shtrom]

Hum, no... I think my Throttle handling has broken something...

[slothking87]

Im not sure if its my system, or if its the last commit to show only available tariffs - But I seem to have lost my tariff sensors this morning, and no amount of removing and adding the integration seems to be able to get them back.

[shtrom]

Dammit. Likely my changes, sorry. I noticed that the API tends to return no data for a while for the previous day once it ticks over to the new day. That includes the tariffs. When that's the case, the integration should refuse to start, and get retried later by HA. Do you have something in your logs that says as much? -- Olivier Mehani ***@***.***> Sent from my mobile, please excuse my brevity.

[slothking87]

No need to be sorry - thats the joys of testing new things!

I cant see anything related to the component in the logs anywhere - Did I need to change the logging level somewhere? I tried in the config.yaml file, but that hasnt seemed to have helped - But I might have the code wrong

[slothking87]

I rolled back to the previous version, then tried to roll forward once the data for yesterday was available. Same thing happened where it marked all sensors apart from kilowatt hour, estimated balance and dollar value usage.

[shtrom]

Ok, the data's in now. It looks like it takes most of the next day before the API has the next-day data. Try restarting your instance now, it should now have the some data, including tariffs. I have an idea to make this more resilient. That'll require adding a feature to the AuroraPlus Python module. I'll have a go at this later today. -- Olivier Mehani ***@***.***> Sent from my mobile, please excuse my brevity.

[shtrom]

Cool. I think you're good to go now! I've had it running for about a week with no new changes required, so I think it's pretty solid now. The only issue seems to be data in Aurora being slow to show up (up to multiple days). Let's see how you fare. Barring install issues that you had due to running forked code, the rest should be smooth now. -- Olivier Mehani ***@***.***> Sent from my mobile, please excuse my brevity.

[slothking87]

Yep. All looking good so far - Thanks heaps for your help (and patience). Out of interest how far back should it be grabbing historical data?

[shtrom]

tl;dr: it only looks as far as the most recent day with data. The Aurora+ API lags a bit before it has data, so it tells you whether data is available or not for each day. If data is not available, the AuroraPlusHA historical sensors will try the day before, all the way to the latest day that has some data available. If the AuroraPlus API makes multiple days available at once (or within an hour, which is the refresh period), the integration will only look at the latest, and not go backwards any further. The HistoricalState then only insert data that is newer than the last recorded data, and ignores older data, even if it was previously missing. There might be a way to play further with recorder to 1) find gaps in statistics, 2) query those from the Aurora+ API and 3) backfill the data in he recorder. However, I don't think it's natively supported by the HistoricalState, so would be tricky to implement. I concluded I could live with some occasional gaps (; -- Olivier Mehani ***@***.***> Sent from my mobile, please excuse my brevity.

[shtrom]

Ok. @LeighCurran I think we're in a good place. I've been running this for a few weeks with no issue and have been getting data reliably from the historical sensors and into the Energy dashboard.

This is still running with my patched version of AuroraPlus 1.1.6 from, as 1.5.0 didn't appear to be in the repo. The patch LeighCurran/AuroraPlus@1021685 is simple enough that I think it should be fine to cherry-pick onto 1.5.0 (give or take some automatic linting) if need be.

If you're happy with merging LeighCurran/AuroraPlus#1 there, I can then point the dependency back to the next PyPi version here, and we can merge this here, too.

[shtrom]

@slothking87 I just pushed an interim fix to support MFA https://github.com/shtrom/AuroraPlus/tree/oauth-mfa-token to this branch.

Instructions are in this commit 7e277e5

You might have to redo the dance to trash your previous state and pull a new version of the dependency library

[slothking87]

I feel like Im missing something very obvious - But I cant get it to work.

When loading the URL mentioned in a web browser, I do get a generic error page, but with this message which makes me think Im missing something somewhere

"Sorry, but we're having trouble signing you in.
We track these errors automatically, but if the problem persists feel free to contact us. In the meantime, please try again.
Correlation ID: 2ba71a8f-221c-4b05-8d0d-1e1d6ade01c9
Timestamp: 2023-11-29 06:14:55Z
AADB2C99059: The supplied request must present a code_challenge"

[shtrom]

Yeah, the url should have been generated with a code challenge... Can you show me the code you used, and the url (this one is safe to share as it's protected by the code_challenge so it's locked to your code, but feel free to edit parts out if you want) -- Olivier Mehani ***@***.***> Sent from my mobile, please excuse my brevity.

[slothking87]

Excuse the noob questions - So the process is to run the commands as they are listed and that opens the correct link?
The code I got was identical to yours so guessing I missed something or something didnt want to work.
EDIT: Nvm - Got the code

Edit 2: So to make this work again - I update the details in the HA config file with username: '' and password being the big long string above?

[shtrom]

Ok, so this URL should be pasted back into the api.redirect_uri(<redirect_uri>), and you'll get an access_token.

Then, yes, update your HA secrets to have username='', and password=<access_token>, and restart HA. If you're in luck, you'll start getting data again.

[shtrom]

PS: be careful about posting redirect_uris (I was talking about posting the first URL that is returned by oauth_authorize), as they do contain secrets.

In this case, the secret is protected by the code_challenge that only your client api instance has, but best be prudent.

[shtrom]

If you're in luck, you'll start getting data again.

I am!

[slothking87]

so close! I got the extra information, but getting the usual authentication error inside home assistant. How much of that final string is whats needed for the access token?

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 359, in _async_setup_platform
await asyncio.shield(task)
File "/config/custom_components/auroraplus/sensor.py", line 98, in async_setup_platform
AuroraPlus = await hass.async_add_executor_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/auroraplus/sensor.py", line 96, in aurora_init
session.getmonth()
File "/usr/local/lib/python3.11/site-packages/auroraplus/init.py", line 89, in getmonth
self.month = self.request("month", index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/auroraplus/init.py", line 63, in request
+ self.serviceAgreementID
^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'api' object has no attribute 'serviceAgreementID'

[shtrom]

Ok, same thing here on my non-test HA deployment. Hold tight.

[shtrom]

You need to redo what you did there #9 (comment) to get the new version of the auroraplus lib.

Ah! Here's the command to run #9 (comment)

[shtrom]

Ok, that didn't work. But I realised existing instances do need an update in the config/custom_components/auroraplus/manifest.json to point to the new branch: shtrom@7e277e5, then a restart.

I also had to delete and remove and reinstall the previous install of the auroraplus:

rm -rf /usr/local/lib/python3.11/site-packages/auroraplus*
pip install --upgrade --no-cache-dir   git+https://github.com/shtrom/AuroraPlus@oauth-mfa-token 

You can check with

cat /usr/local/lib/python3.11/site-packages/auroraplus/__init__.py

that it corresponds to the version at https://github.com/shtrom/AuroraPlus/blob/oauth-mfa-token/auroraplus/__init__.py

[slothking87]

Hmmm, Im still getting the same error as before with the no service agreement ID. Ill have a better play over the weekend - Im probably doing something wrong somewhere

[shtrom]

As usual, Aurora had a wobble just when I was testing, but it came good today, so it works on mine.

If you still have the serviceAgreementID, it's likely you're not on the updated branch yet. I'm not sure how you managed to update it last time, but if you can ssh into the system that runs it, check the __init__.py and compare it to the branch. If it still is different, this means that the lib is not updated.

Worse case scenario, just delete the auroraplus directory (the Python lib, not the HA integration), and it should get reinstalled, hopefully to the correct version, on restart.

[slothking87]

Bingo - That worked!!!!!!

IF anyone uses supervised HA on a RPI like I do, You can run the pip commands via an add on (https://github.com/home-assistant/addons-development). Just add the command (pip install git+https://github.com/shtrom/AuroraPlus@oauth-mfa-token in this case) under the pypi option, save, then start the add on and it does the rest for you. keep an eye on the logs tab to see when its done.

[shtrom]

Awesome!

I still need to work out how to make HA fetch and record the token, rather than having to put it in manually, but that's a good functional workaround in the meantime.

[shtrom]

Ok, it looks like the access_token has a three-week expiry. Data stopped coming on the 20th, I just had to put a new one in my secrets.yaml and restart. I got data for yesterday straight away, as expected. Let's see if that expires around the 17th.

If so, that'll make the need for a way to refresh the token more pressing... If that's indeed something that the Aurora+ Web UI supports. Otherwise, we'll have to at least streamline the process of obtaining a new token.

[shtrom]

@slothking87 re: LeighCurran/AuroraPlus#3 (comment)

I just updated the ha-historical-sensor as per ldotlopez/ha-historical-sensor#12, and pushed it.

This should fix your issue on 2024.1.2, but you'll have to do the usual dance of trashing the old python dep to and make sure the new one is installed.

[shtrom]

Ok, it looks like the access_token has a three-week expiry. Data stopped coming on the 20th, I just had to put a new one in my secrets.yaml and restart. I got data for yesterday straight away, as expected. Let's see if that expires around the 17th.

It lasted until the 25th! Sound like a... 29 day lifetime? I'm renewing it today, so I guess we'll see on the 26th of Feb?

[shtrom]

(psst, have a try at #11 for a UI-based configuration of the access token).