Gitlab
@@ -646,6 +646,8 @@
+
+
@@ -845,6 +847,27 @@
+
+
+
+
+
+
+
+
+
+
+
+ Omplir forats de dades de senyals
+
+
+
+
+
+
+
+
+
diff --git a/search/search_index.json b/search/search_index.json
index 8eeec3a..8759a6f 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Sobre Som Energia Jardiner","text":"
Documentaci\u00f3 per eines desenvolupades en el context del projecte somenergia-jardiner de Som Energia.
"},{"location":"#repositoris","title":"Repositoris","text":"
- Jardiner
- Mirror a Github
- Plant reader per la lectura de la API del prove\u00efdor de dades.
- Plantmonitor Legacy project per a les tasques d'ingesta de solargis i meteologica.
"},{"location":"#accessos-directes","title":"Accessos directes","text":"
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/","title":"Com processem les alarmes en DBT?","text":""},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#context-and-problem-statement","title":"Context and Problem Statement","text":"
Alarmes decomptadors venen en batches que poden venir molt tard.
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#decision-drivers","title":"Decision Drivers","text":"
- Necessitem un hist\u00f2ric d'alarmes
- Cal suportar un batch arbitrari (rebre les lectures de les \u00fatlimes 24h o dels \u00faltims 3 mesos)
- Alarma \u00e9s pr\u00f2pia d'un punt en el temps i un aparell o un conjunt (planta)
unsupported ara mateix
- No suportarem out of the box updates de les lectures (el plantmonitor ni ho considera, ara per ara)
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#considered-options","title":"Considered Options","text":"
- Calcular l'alarma de les \u00faltimes N hores (rang definit per l'alarma)
- Calcular per cada row si en aquell moment hi havia alarma o no
pseudocode-ish idea: Energy last 24h is 0 -> alarm
select sum(energy) from select lag(energy, partition by device order by date desc) from (select * from registry where device1 and date > last_date_1 or device2 and date > last_date_2) group
on last_date = select max(date) from processed group by device
- Union amb jinja de 2
select '' as lectura_date, '' as device, '' as alarm
calculated_alarms:
{% foreach row in meters%} UNION opci\u00f3 2 select max(energy) > 0 from registry where device = row['device'] and date > row['date'] - 24h
select from {{ this }} order by date desc limit 1
materialitzada: select max(date) from calculated_alarms group by device
- incremental de dbt + foreach device
{{ config(materialized='incremental') }}
select date, max(energy) > 0 from registry --> caldr\u00e0 un lag o algo
from raw_app_data.events
{% if is_incremental() %} where event_time > (select max(event_time) from {{ this }})
quan entri un device nou hem de comprovar que where event_time > NULL torna totsels resultats
- without lag: left join with itself on row_number()-24 < row_number() < row_number()
source
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: triem l'opci\u00f3 2. per suportar calcular alarmes en el passat
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#positive-consequences","title":"Positive Consequences","text":"
- Suporta la c\u00e0rrega de batches arbitraris
- Mant\u00e9 un hist\u00f2ric d'alarma
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#negative-consequences","title":"Negative Consequences","text":"
- Requereix materialitzar una taula amb la \u00faltima lectura processada (o tenir un staging on s'inserten les noves)
- Incrementa la complexitat de la query SQL
"},{"location":"adr/2022-10-14-meter_alarm_logic/","title":"Meter alarm logic","text":"
- Status: Unknown
- Deciders: [list everyone involved in the decision]
- Date: [YYYY-MM-DD when the decision was last updated]
Technical Story: [description | ticket/issue URL]
"},{"location":"adr/2022-10-14-meter_alarm_logic/#context-and-problem-statement","title":"Context and Problem Statement","text":"
Les alarmes de comptadors s'han de notificar.
"},{"location":"adr/2022-10-14-meter_alarm_logic/#decision-drivers","title":"Decision Drivers","text":"
coberts: * moxa ens arriben les lectures cada 24h * hi ha comptadors sense lectures durant temps * la notificaci\u00f3 diuen que la volen un cop al dia al llegir del moxa
- tamb\u00e9 volem notificacions de quant s'ha arreglat una planta que estava en alarma
"},{"location":"adr/2022-10-14-meter_alarm_logic/#considered-options","title":"Considered Options","text":"
- fer un post per cada planta un cop cada 24h amb el resum de l'estat de les alarmes [OK|NOK]
- tenir una taula de canvis d'estat i notificar-la
- guardar la taula i fer un diff al dia seguent
"},{"location":"adr/2022-10-14-meter_alarm_logic/#decision-outcome","title":"Decision Outcome","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#negative-consequences","title":"Negative Consequences","text":"
- rebran un e-mail cada dia si hi ha un comptador que no va o reporta 0 --> poden desubstriure's de l'alarma
podem
"},{"location":"adr/2022-10-14-meter_alarm_logic/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#guardar-taula-i-fer-diff","title":"guardar taula i fer diff","text":"
- guardar la taula i fer un diff al dia seguent
ho farem amb un pandas/sql f\u00e0cil per no posar dbt a producci\u00f3 encara
--> ho acabar\u00e0 fent dbt a mig termini
"},{"location":"adr/2022-10-14-meter_alarm_logic/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026
"},{"location":"adr/2022-10-14-spines/","title":"Spines","text":"
- Status: Unknown
- Deciders: [list everyone involved in the decision]
- Date: [YYYY-MM-DD when the decision was last updated]
Technical Story: [description | ticket/issue URL]
"},{"location":"adr/2022-10-14-spines/#context-and-problem-statement","title":"Context and Problem Statement","text":"
[Podem fer un spine amb un left join sense condici\u00f3 contra plant i meter per tenir el carret buit i despr\u00e9s fer un left join amb les lectures
dbt_spine left join plant left join meter on plant.id = meter.plant
"},{"location":"adr/2022-10-14-spines/#decision-drivers","title":"Decision Drivers","text":"
- [driver 1, e.g., a force, facing concern, \u2026]
- [driver 2, e.g., a force, facing concern, \u2026]
- \u2026
"},{"location":"adr/2022-10-14-spines/#considered-options","title":"Considered Options","text":"
- [option 1]
- [option 2]
- [option 3]
- \u2026
"},{"location":"adr/2022-10-14-spines/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].
"},{"location":"adr/2022-10-14-spines/#positive-consequences","title":"Positive Consequences","text":"
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2022-10-14-spines/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2022-10-14-spines/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-14-spines/#option-1","title":"[option 1]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2022-10-14-spines/#option-2","title":"[option 2]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2022-10-14-spines/#option-3","title":"[option 3]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2022-10-14-spines/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026
"},{"location":"adr/2022-10-26-real-time_alarms/","title":"Real time alarms","text":"
- Status: Unknown
- Deciders: [list everyone involved in the decision]
- Date: [YYYY-MM-DD when the decision was last updated]
Technical Story: [description | ticket/issue URL]
"},{"location":"adr/2022-10-26-real-time_alarms/#context-and-problem-statement","title":"Context and Problem Statement","text":"
Necessitem alertes d'incid\u00e8ncies dels diferents dispositius de les plantes. Cada dispositiu t\u00e9 especificitats: granularitat temporal de les lectures, lag natural en al c\u00e0rrega de lectures i casu\u00edstiques concretes que es consideren errors.
Dispositius: - Meter ip Granularitat lectura: 1 hora Cada quan arriben les lectures: 20 min (per\u00f2 ERP sincronitza cada 2 hores) Valors historics updatable: No actualment (a l'erp s\u00ed) Poden apar\u00e8ixer lectures antigues: No actualment (a l'erp s\u00ed)
-
Meter moxa Granularitat lectura: 1 hora Cada quan arriben les lectures: 20 min (per\u00f2 ERP sincronitza cada 24 hores) Batch: 24h Valors historics updatable: No actualment (a l'erp s\u00ed) Poden apar\u00e8ixer lectures antigues: No actualment (a l'erp s\u00ed)
-
Inverter Granularitat lectura: 5 minuts Cada quan arriben les lectures: 5 minuts Valors historics updatable: No Poden apar\u00e8ixer lectures antigues: Rarament
-
String Granularitat lectura: 5 minuts Cada quan arriben les lectures: 5 minuts Valors historics updatable: No Poden apar\u00e8ixer lectures antigues: Rarament
-
Plant
Casu\u00edstica: Meter \u00e9s variable enla cad\u00e8ncia d'enviar lectures (sovint tarda a reconnectar)
- Meter que no conecta Granularitat lectura: 1 hora Cada quan arriben les lectures: Valors historics updatable: No Poden apar\u00e8ixer lectures antigues: No
"},{"location":"adr/2022-10-26-real-time_alarms/#decision-drivers","title":"Decision Drivers","text":"
- Velocitat d'execuci\u00f3 si cal exeutar-ho sovint
- Poder recalcular les alarmes hist\u00f2riques en el futur
- Necessitat de notificar a diversos agents diverses alarmes
- Real-time \u00e9s necessari en algunes alarmes per evitar p\u00e8rdues econ\u00f3miques
- Necessitem un hist\u00f2ric
- Hauriem de tenir un from to de duraci\u00f3 de les alarmes (tot i que conf\u00f3n una mica a projectes)
"},{"location":"adr/2022-10-26-real-time_alarms/#considered-options","title":"Considered Options","text":"
- HI - Calcular realtime a partir de l'hist\u00f2ric individual de cada alarma
- RT - Calcular una taula real-time i una hist\u00f2rica m\u00e9s lenta
- ~~RH - Calcular la real-time i guardar-ho com a hist\u00f2ric~~
- ~~ST - Trobar la manera de fer stream enlloc de batch~~
"},{"location":"adr/2022-10-26-real-time_alarms/#decision-outcome","title":"Decision Outcome","text":"
Hem decidit atacar la opci\u00f3 HI i veure si podem reduir a 5 minuts d'execuci\u00f3 amb incremental. Probablement no ser\u00e0 possible i necessitarem un fast-track (opci\u00f3 RL) per les alertes real-time.24h
Aquest fast track s'executar\u00e0 cincminutalment o podem veure si es podria fer en streaming EL -> T -> N(otificaci\u00f3) Ser\u00e0 exclusiu per alertes i amb poca consolidaci\u00f3 a base de dades i no revisitar\u00e0 el passat. Nom\u00e9s NOW()-ish.
"},{"location":"adr/2022-10-26-real-time_alarms/#positive-consequences","title":"Positive Consequences","text":"
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2022-10-26-real-time_alarms/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2022-10-26-real-time_alarms/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-26-real-time_alarms/#hi-calcular-la-taula-historica","title":"HI - Calcular la taula hist\u00f2rica","text":"
[example | description | pointer to more information | \u2026]
-
- Una \u00fanica l\u00f2gica si no tenim incremental
-
- No hem fet incremental amb DBT mai encara
-
- l'incremental pot reescriure dels dos darrers dies
-
- pots regenerar la taula sempre
-
- fa m\u00e9s complexe les alarmes individuals (over partitions)
DAG:\n(production_inverter) -> (alarm) -> (alrm2)\n\nwith ( Taula production_inverter (registry gapfilled, clean, alarms )) as foo\n\ntime | inverter_id | energy | n_readings | alarm_no_reading | alarm_zero_daylight\n10:00:00 16 34 1 FALSE FALSE\n09:55:00 16 0/NULL 0 TRUE NULL\n09:50:00 16 0 1 FALSE TRUE\n\nselect last row of foo where alarm = TRUE\n
"},{"location":"adr/2022-10-26-real-time_alarms/#rt-calcular-una-taula-real-time-i-una-historica-mes-lenta","title":"RT - Calcular una taula real-time i una hist\u00f2rica m\u00e9s lenta","text":"
[example | description | pointer to more information | \u2026]
-
- Logica m\u00e9s senzilla i r\u00e0pida que podem executar m\u00e9s sovint a airflow
-
- Permet amb un date_trunc() sobre les lectures fer l'agregaci\u00f3 i no cal rolling
-
- Igualment s'ha d'escriure la l\u00f2gica HI per tenir l'hist\u00f2ric correcte
-
- Podries tenir la taula d'estat actual m\u00e9s f\u00e0cilment
-
- La unificada s'hauria de fer unint la taula hist\u00f2rica amb la real-time
-
- requereix molt lag en escriure la hist\u00f2rica, o posar-hi la l\u00f2gica que necessites igualment a la opci\u00f3 HI
-
- Pot generar incongru\u00e8ncies entre el Real-Time (alarmed) que no surtin a l'hist\u00f2ric
-
- A projectes no li agrada tenir una taula estat actual i hist\u00f2ric separat
RT:
real-time, per les alarmes que necessiten notificaci\u00f3 immediata, i per cada tipus d'alarma
with foo as (\nselect\n count(energy),\n max(energy)\nfrom registry\nwhere NOW() - device.alarm_offset < time\ngroup by device\n)\nselect\n now() as time,\n count = 0 as no_reading_alarm,\n max = 0 as no_energy_alarm\nfrom foo\n
notify OK/KO en aquella execuci\u00f3 fent un diff de la taula anterior com ara
hist\u00f2ric: com tenim ara, al final del dia o d'una setmana, re-calcular les alarmes hist\u00f2ricament
"},{"location":"adr/2022-10-26-real-time_alarms/#st-trobar-la-manera-de-fer-stream-enlloc-de-batch","title":"ST- Trobar la manera de fer stream enlloc de batch","text":"
- Dolent perqu\u00e8 no podem regenerar-ho a posteriori
"},{"location":"adr/2022-10-26-real-time_alarms/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/","title":"Pipeline de dades revisited","text":"
- Status: proposed
- Deciders:
- Date:
Technical Story:
Com ser\u00e0 la obt? horaria calculada un cop cada hora? I quan passem a comptadors 15-minutals, segurem amb dbt run 15-minutal? Quina frescor de les dades cal? --> posem-ho a un adr diferent
"},{"location":"adr/2023-06-13-obt_plantes/#context-and-problem-statement","title":"Context and Problem Statement","text":"
Si les dades cincminutals s\u00f3n horaries, la obt de tots els indicadors tamb\u00e9?
Passarem a comptadors 15minutals en algun moment proper.
"},{"location":"adr/2023-06-13-obt_plantes/#decision-drivers","title":"Decision Drivers","text":"
- \u00das real de la obt
- Frescor necess\u00e0ria
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#considered-options","title":"Considered Options","text":"
- [option 1]
- [option 2]
- [option 3]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].
"},{"location":"adr/2023-06-13-obt_plantes/#positive-consequences","title":"Positive Consequences","text":"
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-06-13-obt_plantes/#option-1","title":"[option 1]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#option-2","title":"[option 2]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#option-3","title":"[option 3]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/","title":"Pipeline de dades revisited","text":"
- Status: proposed
- Deciders: Diego, Roger, Pol
- Date: 2023-06-12
Technical Story:
https://trello.com/c/rmCTOJrH/205-decidir-com-farem-lobt-i-el-cincminutal-r%C3%A0pid
Passem a una macrotaula de dades llargues clau-valor per a les dades cincminutals?
Com ser\u00e0 la obt? horaria calculada un cop cada hora? I quan passem a comptadors 15-minutals, segurem amb dbt run 15-minutal? Quina frescor de les dades cal? --> posem la discussi\u00f3 de la obt a un adr diferent
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#context-and-problem-statement","title":"Context and Problem Statement","text":"
Ara que sabem molt m\u00e9s dbt i que incorporarem un prove\u00efdor de dades cincminutals amb dades llargues que deprecar\u00e0 l'esquema antic, ens plantegem si haur\u00edem de basar en una macrotaula de dades llargues per les dades cincminutals.
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#decision-drivers","title":"Decision Drivers","text":"
- \u00das real del cinc-minutal
- Frescor necess\u00e0ria
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#considered-options","title":"Considered Options","text":"
- normalitzar dset i fer un multisource normalitzat com est\u00e0 ara
- desnormalitzar lo que hi ha ara i fer un multisource com ens envia dset
- fer una pseudo-obt llarga a partir de dset i fer le multisource al mart
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: opci\u00f3 3
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-1","title":"[option 1]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-2","title":"[option 2]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-3","title":"[option 3]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026 numbers of links can vary -->
"},{"location":"adr/2023-09-12-redash_vs_superset/","title":"Pipeline de dades revisited","text":"
- Status: proposed
- Deciders: Dades
- Date: 2023-09-12
"},{"location":"adr/2023-09-12-redash_vs_superset/#context-and-problem-statement","title":"Context and Problem Statement","text":"
En el seu moment vam triar redash perqu\u00e8 podia tenir apis com a sources i el dev \u00e9s molt m\u00e9s c\u00f2mode perqu\u00e8 no est\u00e0 tant layeritzat (dataset/query - chart - dashboard).
A la pr\u00e0ctica no hem fet servir mai la lectura directa de les apis perqu\u00e8 les aut\u00e8ntificacions no estaven suportades, ens agradava tenir una c\u00f2pia, t'obligava a treballar en queries de l'sqlite de redash.
A m\u00e9s a m\u00e9s, redash delega la gesti\u00f3 d'usuaris a la db, cosa que implica que no pots limitar que un usuari pugui veure uns valors d'una query i no uns altres, per exemple filtrar per plantes la producci\u00f3, els permisos es fan a nivell de db.
La macrofase de plantmonitor i l'evoluci\u00f3 de l'equip de dades cap a superset amb els altres equips ens porta a replantejar-nos-ho.
Iniciem un spike de superset per a fer els nous dashboards.
"},{"location":"adr/2023-09-12-redash_vs_superset/#decision-drivers","title":"Decision Drivers","text":"
- \u00das real del cinc-minutal
- Frescor necess\u00e0ria
- \u2026
"},{"location":"adr/2023-09-12-redash_vs_superset/#considered-options","title":"Considered Options","text":"
- superset
- redash
- metabase -- en Diego comenta que en la versi\u00f3 free no tens la capa sem\u00e0ntica
"},{"location":"adr/2023-09-12-redash_vs_superset/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: ?
"},{"location":"adr/2023-09-12-redash_vs_superset/#positive-consequences","title":"Positive Consequences","text":"
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-09-12-redash_vs_superset/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-09-12-redash_vs_superset/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#redash","title":"redash","text":"
\ud83d\udc4d
mustache
definir parametres a nivell de default de charts, de chart especific o dashboard
\ud83d\udc4e
no apte per heavy duty (cache regular)
"},{"location":"adr/2023-09-12-redash_vs_superset/#superset","title":"superset","text":"
\ud83d\udc4d
cohesi\u00f3 cooperativa -- tota la cooperativa fa servir superset
\ud83d\udc4e
No pots fixar filtres a nivell de chart al dashboard, cal fer-ne una c\u00f2pia. Implica que si volem canviar com s'ensenya un chart de planta, els haurem de canviar tots un a un.
El jinja \u00e9s funky. No \u00e9s clar com fer servir el \"filter_values\" a dins d'una query. Les queries din\u00e0miques s\u00f3n molt menys directes de fer que al redash.
"},{"location":"adr/template/","title":"Template","text":"
^-- [short title of solved problem and solution]
- Status: [proposed | rejected | accepted | deprecated | \u2026 | superseded by
[ADR-0005](0005-example.md)] - Deciders: [list everyone involved in the decision]
- Date: [YYYY-MM-DD when the decision was last updated]
Technical Story: [description | ticket/issue URL]
"},{"location":"adr/template/#context-and-problem-statement","title":"Context and Problem Statement","text":"
[Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question.]
"},{"location":"adr/template/#decision-drivers","title":"Decision Drivers","text":"
- [driver 1, e.g., a force, facing concern, \u2026]
- [driver 2, e.g., a force, facing concern, \u2026]
- \u2026
"},{"location":"adr/template/#considered-options","title":"Considered Options","text":"
- [option 1]
- [option 2]
- [option 3]
- \u2026
"},{"location":"adr/template/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].
"},{"location":"adr/template/#positive-consequences","title":"Positive Consequences","text":"
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/template/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/template/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/template/#option-1","title":"[option 1]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/template/#option-2","title":"[option 2]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/template/#option-3","title":"[option 3]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/template/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026
"},{"location":"deprecated/TODOs/","title":"Views de plantmonitor a migrar a dbt","text":"
~ marca les que segurament no cal fer, al gust
view_availability.sql\n~view_q_hourly_debug.sql~\nview_q_monthly.sql\n~view_cos_phi_hourly_debug.sql~\nview_cos_phi_monthly.sql\nview_expected_power.sql\nview_hd_daily.sql\nview_ht_daily.sql\nview_ht_monthly.sql\nview_pr_hourly.sql\nview_target_energy.sql\n
"},{"location":"deprecated/TODOs/#agregacions","title":"Agregacions","text":"
No cal fer-les perqu\u00e8 son agraupacions que segurament farem servir m\u00e8triques de dbt tipus:
view_inverter_energy_daily.sql\nview_inverter_energy_monthly.sql\nview_meter_export_energy_daily.sql\nview_meter_export_energy_monthly.sql\nview_average_yearly_production_plant.sql\nview_satellite_irradiation.sql\n
A m\u00e9s a m\u00e9s les m\u00e8triques de dbt resolen el problema seg\u00fcent:
sum(a/x) -- horaria
sum(a/x) + sum(b/y) ... mensual wrong
sum((a+b+c)/(x+y+z)) -- mensual correct
"},{"location":"deprecated/TODOs/#views-per-limitar-lacces-a-la-info-dels-tecnics","title":"views per limitar l'acc\u00e9s a la info dels t\u00e8cnics","text":"
No cal fer-ho, limitarem l'acc\u00e9s a la wide table o farem servir els permisos del superset o ja veurem
view_plants_energes.sql\nview_plants_energetica.sql\nview_plants_ercam.sql\nview_plants_exiom.sql\nview_meterregistry_energetica.sql\nview_inverter_energy_daily_exiom.sql\nview_inverter_intensity_energes.sql\nview_inverter_intensity_ercam.sql\nview_inverter_intensity_exiom.sql\nview_inverter_power_energes.sql\nview_inverter_power_ercam.sql\nview_inverter_power_exiom.sql\nview_inverterregistry_energes.sql\nview_inverterregistry_ercam.sql\nview_inverterregistry_exiom.sql\nview_inverter_temperature_exiom.sql\nview_sensorirradiationregistry_energes.sql\nview_sensorirradiationregistry_ercam.sql\n
"},{"location":"deprecated/TODOs/#no-shan-de-fer-obsoletes","title":"no s'han de fer (obsoletes)","text":"
zero_inverter_power_at_daylight.sql\nzero_sonda_irradiation_at_daylight.sql\n
"},{"location":"desenvolupadors/2023-02-08-novu_notifications/","title":"Novu helpers and quickstart","text":""},{"location":"desenvolupadors/2023-02-08-novu_notifications/#swagger-api","title":"Swagger API","text":"
Novu api
"},{"location":"desenvolupadors/2023-02-08-novu_notifications/#overview","title":"overview","text":"
Features in novu get to the api waaaay before the get to the UI, so keep it at hand since most of the changes (adding removing subscribers, topics, etc.) will be done via python, react or postman via api rest calls.
"},{"location":"desenvolupadors/2023-02-08-novu_notifications/#helpers","title":"helpers","text":"
We have a handful of scripts to interact with novu via airflow or command-line scripts/notify_alert.py and scripts/novu_interface.py.
The notify_alert is ran by airflow to periodically check the alarm tables and trigger a notification on change.
novu_interface is more of a command-line util to do novu stuff manually, like adding subscribers or topics.
For example, to add a topic of a plant we'd do
$ python -m scripts.novu_interface --help\n\nUsage: python -m scripts.novu_interface [OPTIONS] URL API_KEY TOPIC_KEY\n TOPIC_NAME\n\nUsage: python -m scripts.novu_interface add-topic [OPTIONS] URL API_KEY\n TOPIC_KEY TOPIC_NAME\n\n\u256d\u2500 Arguments \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 * url TEXT [default: None] [required] \u2502\n\u2502 * api_key TEXT [default: None] [required] \u2502\n\u2502 * topic_key TEXT [default: None] [required] \u2502\n\u2502 * topic_name TEXT [default: None] [required] \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\u256d\u2500 Options \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 --help Show this message and exit. \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
$ python -m scripts.novu_interface add-topic {novu_base_url} {api_key} {topic_key} {topic_name}\n$ python -m scripts.novu_interface add-topic http://moll3.somenergia.lan:3000/v1 {api_key} asomada-topic-1 \"asomada general topic\"\n
and to get the topics
$ python -m scripts.novu_interface get-topic {novu_base_url} {api_key} {topic_key} {topic_name}\n\n2023-01-27 16:39:29,080 [INFO] {\"page\":0,\"totalCount\":1,\"pageSize\":10,\"data\":[\n {\"_id\":\"63d3ef030d600f1df1a3829e\",\"_environmentId\":\"6349479f3043d373c3d5fc07\",\"_organizationId\":\"6349479f3043d373c3d5fc02\",\"key\":\"asomada-topic-1\",\"name\":\"asomada general topic\",\"subscribers\":[]}\n]}\n
Let's add some subcribers to it:
$ python -m scripts.novu_interface add-subscriber http://moll3.somenergia.lan:3000/v1 {api_key} pol_recipient\n$ python -m scripts.novu_interface add-topic-subscriber http://moll3.somenergia.lan:3000/v1 {api_key} asomada-topic-1 pol_recipient\n
"},{"location":"desenvolupadors/2023-06-05-download_solargis_manually/","title":"Manually download a solargis reading","text":"
To download a specific day you just need to call
python -m scripts.solargis <dbapi> <apikey> 2023-05-13 2023-05-13 1 2 3 4\n 5 6 7 9 13 14 15 16 17 22 40\n
See python -m scripts.solargis --help for info on each parameter (essentially time range and plant id list) and the wiki or the pyxis server for credentials.
Note that to download one day you have to put the same from and to date. The from is start day and to date is end day. The script DOES NOT check for duplicates. You'll have to clean up yourself using the query time if you made a mistake.
"},{"location":"desenvolupadors/2023-06-10-overview-devs/","title":"Overview for devs","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#quickstart","title":"quickstart","text":"
To fully deploy the dbt models to the production schema you can do
dbt build -s tag:jardiner,tag:legacy --store-failures --full-refresh --target prod\n
You will need to have the prod profile in your ~/.dbt/profiles.yml file.
"},{"location":"desenvolupadors/2023-06-10-overview-devs/#current-state-of-the-project","title":"Current State of the project","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#pieces","title":"Pieces","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#repos","title":"repos","text":"
Plantmonitor
Fa d'ORM de les plantes, cont\u00e9 la api i \u00e9s codi de les raspberrypis. A m\u00e9s a m\u00e9s tamb\u00e9 t\u00e9 el codi d'ingesta d'apis de tercers.
Warning
La meitat del plantmonitor acabar\u00e0 deprecated pel prove\u00efdor de dades
Plant Reader
Lectura remota de les plantes. Actualment nom\u00e9s Asomada es llegeix remotament i es guarda a plant_lake.
Acabar\u00e0 deprecated pel prove\u00efdor de dades.
Actualment tamb\u00e9 allotja la lectura de la API del prove\u00efdor de dades.
Jardiner
dbt de les plantes. Actualment tot views.
"},{"location":"desenvolupadors/2023-06-10-overview-devs/#data-sources","title":"Data sources","text":"
- rPIs
- plant_reader
- Irradiation satellite data provider (SAT)
- Meteo forecast provider (METEO)
- Plant data provider (PLANT)
- Price provider (Planned) (PRICE)
"},{"location":"desenvolupadors/2023-06-10-overview-devs/#visualization","title":"Visualization","text":"
- redash
- novu notifications -> helpscout/mail
- superset (other \u00e0mbits of the cooperative)
"},{"location":"desenvolupadors/2023-06-10-overview-devs/#project-tructure","title":"Project tructure","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#actual","title":"Actual","text":"
flowchart LR\n\ndw[db/plants]\n\nplants -- pull 5' \\n plant_reader/dags --> lake[db/plant_lake]\nlake -- push 5' \\n airbyte --> dw\n\nplants -- pull 5' \\n plantmonitor/main.py --> rPIs -- push 5' --> dw\nplants -- pull 2h/12h \\n meter \\n ERP's import_tm_data_click.py --> ERP -- pull 20'\u00ba\\n meter data --> dw\nSAT -- pull daily\u00ba\\n irr/expected energy --> dw\nMETEO <-- pull daily\u00ba\\n meter data + irr/kWh forecast --> dw\nplants -- pull 15'\\n plant_reader/dags --> PLANT -- pull 15'\\n devices data --> db/plant_lake\n\ndw -- dbt views --> prod --> alarms\nprod --> alerts\nprod --> datasets
\u00ba: plantmonitor does it. Update rate defined at conf/startup_configuration.py and each task is run as a function.
"},{"location":"desenvolupadors/2023-06-10-overview-devs/#planned-scheme","title":"Planned scheme","text":"
flowchart LR\n\ndw[db/plants]\n\nplants -- pull 2h/12h \\n meter \\n import_tm_data_click.py --> ERP -- pull 20' \u00ba\\n meter data --> dw\nSAT -- pull daily\\n irr/expected energy --> dw\nMETEO <-- pull daily\\n meter data + irr/kWh forecast --> dw\nplants -- pull 15' --> PLANT -- pull 15'\\n devices data --> dw\n\n\ndw -- dbt fast --> fast[dbt_prod/fast_] --> union\ndw -- dbt --> obt --> alarms\nobt --> union\nunion -- notify.py --> alerts\nobt --> datasets
Despr\u00e9s de l'ADR d'estructura de pipeline
See Roadmap
"},{"location":"desenvolupadors/2023-07-10-dbt_cmds_utils/","title":"dbt commands","text":"
- Generar i servir la documentaci\u00f3 per a poder veure el directed graph.
$ dbt docs generate --project-dir dbt_jardiner --target pre\n\n$ dbt docs serve --project-dir dbt_jardiner --target pre --port 8010\n
- Pujar tots els CSVs de seeds
dbt seed --project-dir dbt_jardiner\n
- Podem passar variables per command line (en comptes de definir-les a dbt_project.yml i fer-les accesibles per diferents resources):
dbt run --vars '{\"variable\":\"value\"}'\n
Per fer testing amb un csv determinat:
dbt run --project-dir project_x --target testing --vars '{\"test_sample\":\"nom_del_csv\"}' -m model_x\n
"},{"location":"desenvolupadors/2023-07-13-codi_reciclable/","title":"Reusuable sqls","text":""},{"location":"desenvolupadors/2023-07-13-codi_reciclable/#dbt-sql-for-rolling-window","title":"dbt sql for rolling window","text":"
{%- set plant_alarm_ranges = dbt_utils.get_query_results_as_dict(ref('plant_alarm_ranges')) -%}\n\n{% for plant in plant_alarm_ranges %}\n\n {% if var('plants_code') == 'SomEnergia_Riudarenes_ZE' %}\n {%- set time_to_sum = '1 hour' -%}\n {% elif var('plants_code') == 'SomEnergia_Matallana' %}\n {%- set time_to_sum = '4 hour' -%}\n {% else %}\n {{ ref('accounts') }} accounts\n {% endif %}\n\n SELECT\n *,\n SUM(export_energy_wh) OVER (\n PARTITION BY plant_name\n ORDER BY \"time\"\n RANGE BETWEEN interval {{time_to_sum}} PRECEDING AND CURRENT ROW\n ) as export_energy_wh_sum\n FROM (\n select * from {{ref('meter_registry_cleaned')}} where codename = {{plants_code}}\n ) as meter_registry_cleaned_current_plant\n UNION ALL\n\n{% endfor %}\n
"},{"location":"desenvolupadors/2023-08-28-alta_planta/","title":"Context","text":"
Tenim un pipe de transformaci\u00f3 de dades que acaba en una serie de dashboards de superset que mostra dades de plantes des de el pipe de transformaci\u00f3 de dades arribant del nostre prove\u00efdor de dades.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#protocol-dalta-duna-planta-fotovoltaica","title":"Protocol d'alta d'una planta fotovoltaica","text":"
Per afegir una nova planta a la visualitzaci\u00f3, cal seguir un protocol documentat al Formulari d\u2019inscripci\u00f3 de noves plantes a Superset
El equip de gestio d'actius ha d'omplir aquest formulari amb les dades de la planta i ens ho han de comunicar per a que puguem seguir amb el proc\u00e9s nosaltres.
Obres en curs amb jardiner-admin
Aquesta tasca est\u00e0 en proc\u00e9s de ser portada al jardiner-admin. Per tant, aquest protocol pot canviar en el futur.
Necessites permisos d'administrador
Aquesta tasca \u00e9s per a desenvolupadors que treballen amb el pipe de transformaci\u00f3 de dades. Si no ets un desenvolupador, no cal que facis aquesta tasca.
Aqui sota ens dediquem a documentar t\u00e9cnicament els passos a seguir des de la perspectiva de desenvolupament.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#checklist-general-per-it","title":"Checklist general per IT","text":"
- Tenir acc\u00e9s als seg\u00fcents documents
- Mapeig de senyals
- Dades fixes de planta
- Info api solargis
- Rendiment de planta
- Producci\u00f3 actual de planta
- Omplir dades al excel de mapeigs d\u2019airbyte
- Afegir planta segons l\u2019estructura de plantmonitor
- Dades de Solargis de planta nova s\u00f3n al script a plantmonitor
- Dades apareixen al superset correctament
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#descripcio-de-dades","title":"Descripci\u00f3 de Dades","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-mapeig-de-senyals","title":"Mapeig de senyals de Planta","text":"
Per cada planta, GdA genera fitxers amb informaci\u00f3 de la planta per la seva gesti\u00f3. Aquests fitxers contenen informaci\u00f3 de senyals, aparells, plantes, etc. per\u00f2 sense el context del pipe de Jardiner. Com IT, portem aquesta informaci\u00f3 al fitxer de mapeig de senyals per a que el pipe de Jardiner pugui fer integrar les dades de la planta.
Un exemple d'aquest fitxer pot ser aquest.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-dades-fixes-de-planta","title":"Dades fixes de planta","text":"
Aquest fitxer cont\u00e9 dades fixes de la planta, com ara la pot\u00e8ncia de la planta, la seva ubicaci\u00f3, etc. Es trova a google drive i es transforma amb airbyte aqui
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-info-api-solargis","title":"Info API SolarGIS","text":"
GdA ha de crear una entrada per la planta nova al fitxer excel per aix\u00f2
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-rendiment-de-planta","title":"Rendiment de planta","text":"
T\u00e9 un nom tipus \"C\u00e0lcul Rendiment de Planta <nom planta>\". El podeu buscar al cercador de drive.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-produccio-actual-de-planta","title":"Objectius de producci\u00f3 de planta","text":"
Es troba al fitxer de google sheets aqui i es transforma amb airbyte aqui.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-unificacio-noms-projectes-generacio","title":"Unificaci\u00f3 Noms Projectes Generaci\u00f3","text":"
Cont\u00e9 els llistat de noms unificats en un sol fitxer. Ves a Unificaci\u00f3 Noms Projectes Generaci\u00f3.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-modelos-contadores-inversores-scadas","title":"Modelos contadores_inversores_SCADAS","text":"
Modelos contadores_inversores_SCADAS
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#descripcio-de-tasques","title":"Descripci\u00f3 de tasques","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-mapeig-airbyte","title":"Afegir dades al mapeig d'airbyte","text":"
Tenim un document on consolidem relacions entre UUIDs de senyals, aparells i plantes que inserim amb airbyte
Aquest mapeig es una traducci\u00f3 del fitxer de mapeigs.
Un cop heu acabat de fer el mapeig, cal engegar la tasca de sync de l'airbyte d'uuids per que les dades arribin a la base de dades. Heu de tenir especial cura d'omplir les columnes updated_at i aquelles que posen UUIDs, ja que amb elles se creuen les dades dins del pipe.
update_at s'actualitza automagicament
Al fitxer de google sheets hi ha un script que actualitza la columna updated_at quan hi ha hagut un canvi a nivell de fila. Es molt limitat i heu de supervisar que s'ha actualitzat correctament, ja que de vegades google sheets no ho fa. Podeu mirar a Extensions > Apps Script per veure el codi.
UUID s\u00f3n versi\u00f3 4
Els senyals porten UUID4 (han de ser versio 4) \u00fanics al mapeig de la planta indicada. Es poden generar UUID4 amb https://www.uuidgenerator.net/ O ben v\u00e9 amb un script de python que podeu afegir a ~/.bash_aliases:
#\u00b7function\u00b7to\u00b7genenerate\u00b7random\u00b7uuid4\u00b7using\u00b7python$\nfunction pyuuid4() {\npython -c \"import uuid$\nfor _ in range(${N:-1}):\n print(uuid.uuid4())\" ; }\n
i el podeu executar amb N=<n> pyuuid4 per a generar n UUID4.
$ N=10 pyuuid4\nbf98707e-688c-471c-8700-9ffeaf436531\n8f6bfcc0-48bf-4b7c-895f-0b2efa37dd63\nee5dcfd1-80ba-48fb-800d-5bc8c04d217f\nd32b0a23-c479-4ae9-bb54-43390fedf8e4\neced945e-7a91-4451-bbd5-974a82c9c0d4\nfbacbf3d-23f6-4361-b5ec-b48b390d9c09\nb9c4ea66-672e-4dca-ad08-fddacdfb3627\nae3e242e-9a57-4359-a72b-313a72b48348\n9e11a595-c32c-42e1-8549-a5161c0ea624\n172061ed-96b4-4aa3-b5bd-0c66e850383b\n
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-meteologica-plantmonitor","title":"Afegir planta per
meteologica amb
plantmonitor","text":"
Qu\u00e8 \u00e9s meteologica?
meteologica \u00e9s un servei que ens permet tenir dades meteorol\u00f2giques de les plantes fotovoltaiques. Aquest servei es comunica amb plantmonitor per a obtenir dades de les plantes. Es una API tipus SOAP i teniu la documentaci\u00f3 disponible aqui.
Qu\u00e8 \u00e9s plantmonitor?
Tots els scripts en aquest apartat fan refer\u00e8ncia al repositori de github plantmonitor. \u00c9s un projecte previ a jardiner que ens permet monitoritzar plantes fotovoltaiques quan l'obtenci\u00f3 de dades la gestionem nosaltres.
A plantmonitor tenim un script que ens permet afegir plantes noves. Aquest script s'ha de cridar amb un fitxer yaml que contingui les dades de la planta.
S'ha de crear la planta a la base de dades tant a la raspberrypi com a plantmonitor (no calen els dispositius, que els crea autom\u00e0ticament). Haureu d'afegir plantlocation, plantparameters i moduleparameters. Trobareu un exemple a plantmonitor
Un cop teniu el fitxer yaml, per exemple data/plant-nova.yaml, podeu cridar l'script addPlant.py amb el seg\u00fcent comandament:
PLANTMONITOR_MODULE_SETTINGS='conf.settings.prod' python addPlant.py data/plant-asomada.yaml\n
Haureu de tenir un fitxer .env.prod amb les variables d'entorn necess\u00e0ries per a que l'script funcioni correctament. Mireu a la documentaci\u00f3 de plantmonitor per a m\u00e9s informaci\u00f3.
De moment manualment cal afegir els registres de plantestimatedmonthlyenergy. L'any \u00e9s irrellevant. Si tenim hist\u00f2rics d'energia objectiu, els podeu afegir.
INSERT INTO\n public.plantestimatedmonthlyenergy(plantparameters, \"time\", monthly_target_energy_kwh)\nVALUES\n (33, '2022-01-01 00:00:00+01', 473630),\n (33, '2022-02-01 00:00:00+01', 453370),\n (33, '2022-03-01 00:00:00+01', 592040),\n (33, '2022-04-01 00:00:00+02', 617000),\n (33, '2022-05-01 00:00:00+02', 667210),\n (33, '2022-06-01 00:00:00+02', 648940),\n (33, '2022-07-01 00:00:00+02', 703030),\n (33, '2022-08-01 00:00:00+02', 644500),\n (33, '2022-09-01 00:00:00+02', 574580),\n (33, '2022-10-01 00:00:00+02', 524110),\n (33, '2022-11-01 00:00:00+01', 446490),\n (33, '2022-12-01 00:00:00+01', 442060);\n
-
Executar l\u2019script d\u2019alba i posta per a generar totes les albes i postes dels propers 10 anys
python -m scripts.sun_events -p Alcolea -s 2021-12-13 -e 2021-12-14\n
-
Afegir les dades al diccionari de solargis (Veure com aqui). Quan deixi de ser hard-coded ja no caldr\u00e0 aquest pas.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-solargis","title":"Afegir planta al script de SolarGIS","text":"
S'ha de modificar el codi de plantmonitor per incloure la nova planta. Podeu trobar un exemple de modificacions github
Amb la MR aprovada i fusionada, s'ha de fer git pull des de el servidor de producci\u00f3 i reiniciar el servei de plantmonitor.
ssh someone@somewhere.lan\nsource /home/<someone>/Envs/plantmonitor/bin/activate\ncd /home/<someone>/somenergia/plantmonitor\ngit pull\npython -m scripts.solargis --help\npython -m scripts.solargis [OPTIONS] DBAPI SOLARGIS_API_KEY FROM_DATE TO_DATE [PLANT_IDS]\nsudo supervisorctl restart plantmonitor\n
Aqui, someone \u00e9s el nom de l'usuari que t\u00e9 acc\u00e9s a somewhere.lan, i PLANT_IDS s\u00f3n els ids de les plantes que vau afegir a l'API de SolarGIS amb la PR anterior. Demaneu acc\u00e9s a somewhere.lan a l'equip de dades.
La API de SolarGIS requereix un token d'acc\u00e9s i es crida remotament amb supervisord, amb codi al repositori de plantmonitor.
L'API de SolarGIS \u00e9s SOAP
L'API es del tipus SOAP i no REST. \u00c9s a dir, demana un XML amb les dades de la planta. Podeu trobar informaci\u00f3 sobre com cridar-la a la seva documentaci\u00f3 d'usuari.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#dades-requerides-per-aparells","title":"Dades requerides per aparells","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-comptadors-a-lerp","title":"Afegir Comptadors a l'ERP","text":"
Les lectures a l'ERP s\u00f3n necessaries mentre metelogica faci servir les lectures de comptadors que plantmonitor copia de l'ERP. Tamb\u00e9 s\u00f3n necess\u00e0ries per al generationkwh. Per aix\u00f2 cal agafar les dades de Modelos contadores_inversores_SCADAS i afegir-les a l'erp Infrastructura -> Registradors -> tots els registradors.
- ip lan, port, porta d\u2019enlla\u00e7 i contrasenya
- tel\u00e8fon, si s'escau
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#serveis-de-somenergia-addicionals","title":"Serveis de SomEnergia addicionals","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#nagios","title":"nagios","text":"
Preguntar a Gesti\u00f3 d'Actius si encara fan servir el nagios. Si si, afegir-la al nagios o modificar-la en cas de canvi d\u2019ip p\u00fablica
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#opendata","title":"opendata","text":"
Cal afegir la info de la nova planta a taula literal que es fa servir a somenergia-opendata/som_opendata/queries/plantpower.sql. Un cop aquesta info estigui a plantmonitor, l\u2019opendata podr\u00e0 agafar-la directament per sql i no caldr\u00e0 fer aquest pas.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#generationkwh","title":"generationkwh","text":"
Si la planta est\u00e0 inclosa al generation, \u00e9s important afegir-la a l\u2019erp abans de la mitja nit del dia de posada a producci\u00f3. Si no, cal descartar c\u00e0lculs de drets del generation per a que incloguin els resultats. Si, a m\u00e9s, els drets s\u2019han comen\u00e7at a gastar (15 dies de marge de facturaci\u00f3), cal reperfilar els nous drets amb la producci\u00f3 afegida perqu\u00e8 no superi els drets ja atorgats.
La nova planta es crea fent servir l\u2019script a somenergia-generationkwh/scripts/genkwh_plants.py
Un exemple complert de com afegir una planta amb aquest script el podeu trobar a https://github.com/Som-Energia/somenergia-generationkwh/blob/master/scripts/genkwh_migrate_plantmeter_1_7_0_newplant.sh
Es recomana provar primer a un ERP de testing o local, i, despr\u00e9s aplicar-ho a producci\u00f3.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#reperfilacio-de-dades","title":"Reperfilaci\u00f3 de dades","text":"
Un exemple d\u2019script de migraci\u00f3 en la que vam haver de reperfilar: https://github.com/Som-Energia/somenergia-generationkwh/blob/master/scripts/genkwh_migrate_plantmeter_1_7_2_fontivsolarfix.sh
"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/","title":"Incremental Dedup","text":"
Exemple de com fer un incremental amb deduplicaci\u00f3 en dbt.
{{\n config(\n materialized = 'incremental',\n unique_key = ['signal_uuid','ts'],\n incremental_strategy = 'merge',\n incremental_predicates = [\n \"DBT_INTERNAL_DEST.queried_at > dateadd(day, -1, current_date)\"\n ]\n )\n}}\n\nwhere\n ts < now() - interval '1 hour' {#- select only freshly ingested rows #}\n\n {% if is_incremental() -%}\n and queried_at > coalesce((select max(queried_at) from {{ this }}), '1900-01-01') and queried_at > now() - interval '2 hour'\n {%- endif %}\n
source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00 energia_comptador 03:00 null 04:00 potencia_inversor 03:00 50 +1d 04:00 potencia_inversor 03:00 50 04:00
Son ara les 15:04. en la materialitzada tindrem
source senyal ts valor potencia_inversor 14:00 67 energia_comptador 14:00 null
o bien, la \u00faltima fila no hi ser\u00e0:
source senyal ts valor potencia_inversor 14:00 67"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/#explicacio","title":"Explicaci\u00f3","text":"
where max(ts) = 14:00:
source senyal ts valor potencia_inversor 14:00 67 energia_comptador 14:00 null
where max(ts) - 24h:
source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00 potencia_inversor 03:00 50 +1d 04:00 potencia_inversor 03:00 50 04:00 potencia_inversor 14:00 67 energia_comptador 14:00 null
unique (signal_uuid, ts):
source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/#prova-practica","title":"prova pr\u00e0ctica","text":"
Abans d'executar l'incremental tenim aquest tres \u00faltims registres a materialized one hour late
group_name queried_at ts signal_code signal_device_type signal_device_uuid signal_frequency signal_id signal_is_virtual signal_last_ts signal_last_value signal_type signal_tz signal_unit signal_uuid signal_uuid_raw signal_value materialized_at SE_vallehermoso 2024-01-09 13:24:42.488123+01 2024-01-05 23:15:00+01 ce_eactexp meter 08bc8d88-4ea2-4ee9-b817-00e1f07debba 15 minutes 983894 false 2024-01-08 23:45:00+01 0 absolute Europe/Madrid kWh 646d1ec2-0ca2-4c22-9739-8161aafb224e 646d1ec2-0ca2-4c22-9739-8161aafb224e 0.0 2024-01-09 13:36:52.925381+01 SE_asomada 2024-01-09 13:24:42.488123+01 2024-01-05 23:00:00+01 s8102 meter c5ed9fab-e73c-40a5-acb3-113e22052fd6 5 minutes 1031060 false 2024-01-09 11:47:00+01 60.2 absolute Europe/Madrid V dd065f6a-4ded-41a8-98d3-ba239f9a2a47 dd065f6a-4ded-41a8-98d3-ba239f9a2a47 60.2 2024-01-09 13:36:52.925381+01 SE_tahal 2024-01-09 13:24:42.488123+01 2024-01-05 23:45:00+01 ce_ercapexp_er3 meter 5deb3780-02e2-4dcf-a6a7-de646991762c 15 minutes 983890 false 2024-01-08 23:45:00+01 0 absolute Europe/Madrid kVArh 31ae7d09-95db-447d-b5bf-358b056ef5bf 31ae7d09-95db-447d-b5bf-358b056ef5bf 0.0 2024-01-09 13:36:52.925381+01"},{"location":"desenvolupadors/2024-02-16-row-level-security/","title":"Row Level Security","text":""},{"location":"desenvolupadors/2024-02-16-row-level-security/#context","title":"Context","text":"
Tenim usuaris externs que necessiten mirar dades de plantes, per\u00f2 nom\u00e9s de les plantes que gestionen. Aix\u00f2 ho podem fer amb Row Level Security a Superset.
El proc\u00e9s \u00e9s el seg\u00fcent:
- Crear un rol
Manteniment_<nom empresa> si no existeix, pel cas de noves plantes. - Afegir una regla Row Level Security Pareu atenci\u00f3 al camp \"clause\", s'afegir\u00e0 com a condici\u00f3 where exclusiva (and) als datasets que seleccionem.
- Donar acc\u00e9s als charts d'un dashboard
- Afegir la o les usu\u00e0ries i assignar-los els rols [Gamma] i
[Manteniment_<nom empresa>]
I ja ho tens. Ho he fet amb energes amb la clause nom_planta in (noms plantes)
"},{"location":"desenvolupadors/2024-02-16-row-level-security/#decisions","title":"Decisions","text":"
Hem decidit a 02/2024 que no val la pena propagar l'empresa de manteniment en el pipe per a cobrir aquest use case, ja que no cal afegir 5m aquestes metadadades per a un tema de visualitzaci\u00f3 i permisos. Ho gestionem en la capa de visualitzaci\u00f3.
Canvis d'empresa de manteniment
Canvis d'empresa de manteniment s'hauran de fer al Superset (i a les metadades de la planta, per coher\u00e8ncia).
"},{"location":"desenvolupadors/2024-02-16-row-level-security/#referencies","title":"Refer\u00e8ncies","text":"
- Issue #101 a gitlab
- Targeta Trello
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/","title":"Macrofase Jardiner 2023","text":"
Document de Roadmap
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#roadmap-overview","title":"Roadmap overview","text":""},{"location":"projecte/2023-06-03-macrofase%20roadmap/#flux-de-dades","title":"Flux de dades","text":"
flowchart LR\n\nrPIs -- push 5' --> dw[db/plants]\nERP -- pull 20' \u00ba\\n meter data --> dw\nSAT -- pull daily\\n irr/expected energy --> dw\nMETEO <-- pull daily\\n meter data + irr/kWh forecast --> dw\nPLANT -- pull 15'\\n devices data --> dw\n\ndw -- dbt views --> prod --> alarms\nprod --> alerts\nprod --> datasets
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-1","title":"Fita 1","text":"
Prioritat 1
Aprox 1-2 fases
Definition of Done (DoD): veure per tema
Cost en mitjos dies de l'equip sencer entre par\u00e8ntesis (N)
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#dashboard-tecnics-de-plantes-critiques","title":"Dashboard t\u00e8cnics de plantes cr\u00edtiques","text":"
Pulir el dashboard de t\u00e8cnics per a Asomada i Llanillos.
DoD: TODO
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#obtencio-de-dades-de-tots-els-equips-i-totes-les-plantes","title":"Obtenci\u00f3 de dades de tots els equips i totes les plantes","text":"
- Definir estrat\u00e8gia d'obtenci\u00f3 de dades (remot/hardware/api's a plataformes...) (1)
- Coordinaci\u00f3 amb altres equips de la cooperativa per veure altres alternatives (6)
- Proves pilot amb alg\u00fan prove\u00efdor (8)
- [-] Analisi de prove\u00efdors/alternatives i selecci\u00f3 de partner per externalitzar l'obtenci\u00f3 de dades dels inversors i sondes (2)
-
[-] Obtenci\u00f3 de dades tots els comptadors per IP.
-
Adaptaci\u00f3 a frequ\u00e8ncia quarthoraria. (3)
-
[-] Suport a la integraci\u00f3 de prova pilot (6)
- Falta saber com es diferencien les plantes
DoD:
- Els equips mencionats en el pipe
- Dset integrat al pipe
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#introduccio-al-jardiner","title":"Introducci\u00f3 al jardiner","text":"
Estat del codi i rep\u00e0s de l'estructura. Documentaci\u00f3 per devs.
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#dt-obt-i-fast-lane","title":"[DT] OBT i fast-lane","text":"
Crear la obt cincminutal incremental i/o el fast-lane cincminutal i la obt di\u00e0ria.
Crear el live per late arriving facts de comptadors i apis.
En funcionament normal s\u00f3n delays de m\u00e0xim un dia. Amb incid\u00e8ncia poden ser 1-2 mesos m\u00e0xim. I en casos excepcionals, molt m\u00e9s (candidat a full-refresh manual o alguna altra cosa)
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#aterracio-de-tot-plegat","title":"Aterraci\u00f3 de tot plegat","text":"
- Intro per devs
- M\u00e9s doc?
- Afegit la prove\u00efdora de dades al pipe
- Decidir com (multisource/sol)
- Refer el pipe (fast+slow)
- Que no tardi 30s
- DBT standards
- Definir el dashboard de t\u00e8cnics
- Volem fer servir redash?
- ~Parlar amb GA per si volen el Dashboard diferent al actual~ El volen tal com s\u00f3n
- Dashboard tecnics Exiom amb Asomada
- Dashboard tecnics Exiom amb Llanillos (source original \u00e9s dset)
- Deute t\u00e8cnic Dev
- CDD
- Private doc github project
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-2-dashboard-overview-plantes","title":"Fita 2: Dashboard Overview plantes","text":"
Prioritat 4
Disseny d'un Dashboard amb un llistat de totes les plantes amb els principals indicadors (mesures en temps real i acumulades i resum alarmes). Pendent fer una proposta per part de GA. Des d'aquest dashboard s'ha de poder accedir al Dashboard de cada planta per obtenir informaci\u00f3 detallada. Esborrany inicial falta versi\u00f3 actualitzada. (6)
Implementaci\u00f3 (9)
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-3-dashboard-planta","title":"Fita 3: Dashboard planta","text":"
Prioritat 5
Definir versi\u00f3 revisada del Dashboard de planta per a totes les plantes: Unifilar din\u00e0mic, dades fixes i reempla\u00e7ar alarmes, millorar la velociatat i treure el dataset de sota, afegir solargis + coses que puguin sortir despr\u00e9s (3)
Implementar versi\u00f3 revisada del Dashboard de planta per a totes les plantes: Unifilar din\u00e0mic, dades fixes i reempla\u00e7ar alarmes, millorar la velocitat i treure el dataset de sota, afegir solargis + coses que puguin sortir despr\u00e9s (3)
Dashboard Ringsted per plantes -- particularitats (4)
Dashboard Gasdtrup -- particularitats (4)
Dashboard de planta t\u00edpica (Alcolea, etc) (0)
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-4-dashboard-tecnics","title":"Fita 4: Dashboard t\u00e8cnics","text":"
Prioritat 6
Definir la versi\u00f3 final amb grafiques de Solar Gis, Alarmes i pot ser alarmes. Pendent fer una revisi\u00f3 de les funcionalitats per part de GA. Incloure un dashboard per cadascuna de les empreses de manteniment. Al dashboard unificat d'alarmes estat actual afegir des de quan est\u00e0 activada l'alarma
Implementar la versi\u00f3 final amb grafiques de Solar Gis, Alarmes i pot ser alarmes. Pendent fer una revisi\u00f3 de les funcionalitats per part de GA. Incloure un dashboard per cadascuna de les empreses de manteniment
Fer el Dashboard per a 8 empreses instal\u00b7ladores. Haur\u00edem de passar a Superset que t\u00e9 una millor gesti\u00f3 d'usuaris?
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-5-dashboard-ineco","title":"Fita 5: Dashboard INECO","text":"
Prioritat 7
Per definir
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-6-noves-queries","title":"Fita 6: Noves queries","text":"
Prioritat 8
- Query: Curves HMCIL + lectures comptador (reclamacions)
- Query: Curves HMCIL + previsi\u00f3 (seguiment)
- Alarma strings per comparaci\u00f3 en plantes grans >1MW
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-7-dashboard-gurb","title":"Fita 7: Dashboard GURB","text":"
Prioritat 9
"},{"location":"projecte/2023-06-03-overview/","title":"Overview del projecte per a usuaris","text":"
Descripci\u00f3 del projecte des del punt de vista de l'usuari
"},{"location":"projecte/2023-06-03-overview/#context-de-naixement","title":"Context de naixement","text":"
- A quin objectiu respon? Guanyar autonomia dins del mercat el\u00e8ctric.
- Context: per qu\u00e8 \u00e9s important? per guanyar en efici\u00e8ncia i optimitzar la producci\u00f3 (problema)
- Intenci\u00f3: on volem arribar? tenir un sistema de monitoritzaci\u00f3 unificat per a totes les plantes de producci\u00f3 (soluci\u00f3)
- Criteris d\u2019\u00e8xit: com mesurar progr\u00e9s i \u00e8xit?
- N\u00famero de plantes que poden gestionar per persona (20-25 plantes/persona)
- Agilitat en detectar incid\u00e8ncies (1.800 \u20ac/dia per p\u00e8rdues de producci\u00f3 seria el cas m\u00e9s extrem per una planta de 5MW. Si triguem una setmana en detectar-ho serien 12.600 \u20ac)
"},{"location":"projecte/2023-06-03-overview/#peces","title":"Peces","text":"
- Usuaris: T\u00e8cnics de planta, Gesti\u00f3 d'Actius, Altres \u00e0mbits de l'ET, IT
- Altres Rols: Prove\u00efdors d'internet
- Interf\u00edcies de contacte:
- redash
- Helpscout/notificacions
- erp
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Sobre Som Energia Jardiner","text":"
Documentaci\u00f3 per eines desenvolupades en el context del projecte somenergia-jardiner de Som Energia.
"},{"location":"#repositoris","title":"Repositoris","text":"
- Jardiner
- Mirror a Github
- Plant reader per la lectura de la API del prove\u00efdor de dades.
- Plantmonitor Legacy project per a les tasques d'ingesta de solargis i meteologica.
"},{"location":"#accessos-directes","title":"Accessos directes","text":"
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/","title":"Com processem les alarmes en DBT?","text":""},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#context-and-problem-statement","title":"Context and Problem Statement","text":"
Alarmes decomptadors venen en batches que poden venir molt tard.
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#decision-drivers","title":"Decision Drivers","text":"
- Necessitem un hist\u00f2ric d'alarmes
- Cal suportar un batch arbitrari (rebre les lectures de les \u00fatlimes 24h o dels \u00faltims 3 mesos)
- Alarma \u00e9s pr\u00f2pia d'un punt en el temps i un aparell o un conjunt (planta)
unsupported ara mateix
- No suportarem out of the box updates de les lectures (el plantmonitor ni ho considera, ara per ara)
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#considered-options","title":"Considered Options","text":"
- Calcular l'alarma de les \u00faltimes N hores (rang definit per l'alarma)
- Calcular per cada row si en aquell moment hi havia alarma o no
pseudocode-ish idea: Energy last 24h is 0 -> alarm
select sum(energy) from select lag(energy, partition by device order by date desc) from (select * from registry where device1 and date > last_date_1 or device2 and date > last_date_2) group
on last_date = select max(date) from processed group by device
- Union amb jinja de 2
select '' as lectura_date, '' as device, '' as alarm
calculated_alarms:
{% foreach row in meters%} UNION opci\u00f3 2 select max(energy) > 0 from registry where device = row['device'] and date > row['date'] - 24h
select from {{ this }} order by date desc limit 1
materialitzada: select max(date) from calculated_alarms group by device
- incremental de dbt + foreach device
{{ config(materialized='incremental') }}
select date, max(energy) > 0 from registry --> caldr\u00e0 un lag o algo
from raw_app_data.events
{% if is_incremental() %} where event_time > (select max(event_time) from {{ this }})
quan entri un device nou hem de comprovar que where event_time > NULL torna totsels resultats
- without lag: left join with itself on row_number()-24 < row_number() < row_number()
source
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: triem l'opci\u00f3 2. per suportar calcular alarmes en el passat
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#positive-consequences","title":"Positive Consequences","text":"
- Suporta la c\u00e0rrega de batches arbitraris
- Mant\u00e9 un hist\u00f2ric d'alarma
"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#negative-consequences","title":"Negative Consequences","text":"
- Requereix materialitzar una taula amb la \u00faltima lectura processada (o tenir un staging on s'inserten les noves)
- Incrementa la complexitat de la query SQL
"},{"location":"adr/2022-10-14-meter_alarm_logic/","title":"Meter alarm logic","text":"
- Status: Unknown
- Deciders: [list everyone involved in the decision]
- Date: [YYYY-MM-DD when the decision was last updated]
Technical Story: [description | ticket/issue URL]
"},{"location":"adr/2022-10-14-meter_alarm_logic/#context-and-problem-statement","title":"Context and Problem Statement","text":"
Les alarmes de comptadors s'han de notificar.
"},{"location":"adr/2022-10-14-meter_alarm_logic/#decision-drivers","title":"Decision Drivers","text":"
coberts: * moxa ens arriben les lectures cada 24h * hi ha comptadors sense lectures durant temps * la notificaci\u00f3 diuen que la volen un cop al dia al llegir del moxa
- tamb\u00e9 volem notificacions de quant s'ha arreglat una planta que estava en alarma
"},{"location":"adr/2022-10-14-meter_alarm_logic/#considered-options","title":"Considered Options","text":"
- fer un post per cada planta un cop cada 24h amb el resum de l'estat de les alarmes [OK|NOK]
- tenir una taula de canvis d'estat i notificar-la
- guardar la taula i fer un diff al dia seguent
"},{"location":"adr/2022-10-14-meter_alarm_logic/#decision-outcome","title":"Decision Outcome","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#negative-consequences","title":"Negative Consequences","text":"
- rebran un e-mail cada dia si hi ha un comptador que no va o reporta 0 --> poden desubstriure's de l'alarma
podem
"},{"location":"adr/2022-10-14-meter_alarm_logic/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#guardar-taula-i-fer-diff","title":"guardar taula i fer diff","text":"
- guardar la taula i fer un diff al dia seguent
ho farem amb un pandas/sql f\u00e0cil per no posar dbt a producci\u00f3 encara
--> ho acabar\u00e0 fent dbt a mig termini
"},{"location":"adr/2022-10-14-meter_alarm_logic/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026
"},{"location":"adr/2022-10-14-spines/","title":"Spines","text":"
- Status: Unknown
- Deciders: [list everyone involved in the decision]
- Date: [YYYY-MM-DD when the decision was last updated]
Technical Story: [description | ticket/issue URL]
"},{"location":"adr/2022-10-14-spines/#context-and-problem-statement","title":"Context and Problem Statement","text":"
[Podem fer un spine amb un left join sense condici\u00f3 contra plant i meter per tenir el carret buit i despr\u00e9s fer un left join amb les lectures
dbt_spine left join plant left join meter on plant.id = meter.plant
"},{"location":"adr/2022-10-14-spines/#decision-drivers","title":"Decision Drivers","text":"
- [driver 1, e.g., a force, facing concern, \u2026]
- [driver 2, e.g., a force, facing concern, \u2026]
- \u2026
"},{"location":"adr/2022-10-14-spines/#considered-options","title":"Considered Options","text":"
- [option 1]
- [option 2]
- [option 3]
- \u2026
"},{"location":"adr/2022-10-14-spines/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].
"},{"location":"adr/2022-10-14-spines/#positive-consequences","title":"Positive Consequences","text":"
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2022-10-14-spines/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2022-10-14-spines/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-14-spines/#option-1","title":"[option 1]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2022-10-14-spines/#option-2","title":"[option 2]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2022-10-14-spines/#option-3","title":"[option 3]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2022-10-14-spines/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026
"},{"location":"adr/2022-10-26-real-time_alarms/","title":"Real time alarms","text":"
- Status: Unknown
- Deciders: [list everyone involved in the decision]
- Date: [YYYY-MM-DD when the decision was last updated]
Technical Story: [description | ticket/issue URL]
"},{"location":"adr/2022-10-26-real-time_alarms/#context-and-problem-statement","title":"Context and Problem Statement","text":"
Necessitem alertes d'incid\u00e8ncies dels diferents dispositius de les plantes. Cada dispositiu t\u00e9 especificitats: granularitat temporal de les lectures, lag natural en al c\u00e0rrega de lectures i casu\u00edstiques concretes que es consideren errors.
Dispositius: - Meter ip Granularitat lectura: 1 hora Cada quan arriben les lectures: 20 min (per\u00f2 ERP sincronitza cada 2 hores) Valors historics updatable: No actualment (a l'erp s\u00ed) Poden apar\u00e8ixer lectures antigues: No actualment (a l'erp s\u00ed)
-
Meter moxa Granularitat lectura: 1 hora Cada quan arriben les lectures: 20 min (per\u00f2 ERP sincronitza cada 24 hores) Batch: 24h Valors historics updatable: No actualment (a l'erp s\u00ed) Poden apar\u00e8ixer lectures antigues: No actualment (a l'erp s\u00ed)
-
Inverter Granularitat lectura: 5 minuts Cada quan arriben les lectures: 5 minuts Valors historics updatable: No Poden apar\u00e8ixer lectures antigues: Rarament
-
String Granularitat lectura: 5 minuts Cada quan arriben les lectures: 5 minuts Valors historics updatable: No Poden apar\u00e8ixer lectures antigues: Rarament
-
Plant
Casu\u00edstica: Meter \u00e9s variable enla cad\u00e8ncia d'enviar lectures (sovint tarda a reconnectar)
- Meter que no conecta Granularitat lectura: 1 hora Cada quan arriben les lectures: Valors historics updatable: No Poden apar\u00e8ixer lectures antigues: No
"},{"location":"adr/2022-10-26-real-time_alarms/#decision-drivers","title":"Decision Drivers","text":"
- Velocitat d'execuci\u00f3 si cal exeutar-ho sovint
- Poder recalcular les alarmes hist\u00f2riques en el futur
- Necessitat de notificar a diversos agents diverses alarmes
- Real-time \u00e9s necessari en algunes alarmes per evitar p\u00e8rdues econ\u00f3miques
- Necessitem un hist\u00f2ric
- Hauriem de tenir un from to de duraci\u00f3 de les alarmes (tot i que conf\u00f3n una mica a projectes)
"},{"location":"adr/2022-10-26-real-time_alarms/#considered-options","title":"Considered Options","text":"
- HI - Calcular realtime a partir de l'hist\u00f2ric individual de cada alarma
- RT - Calcular una taula real-time i una hist\u00f2rica m\u00e9s lenta
- ~~RH - Calcular la real-time i guardar-ho com a hist\u00f2ric~~
- ~~ST - Trobar la manera de fer stream enlloc de batch~~
"},{"location":"adr/2022-10-26-real-time_alarms/#decision-outcome","title":"Decision Outcome","text":"
Hem decidit atacar la opci\u00f3 HI i veure si podem reduir a 5 minuts d'execuci\u00f3 amb incremental. Probablement no ser\u00e0 possible i necessitarem un fast-track (opci\u00f3 RL) per les alertes real-time.24h
Aquest fast track s'executar\u00e0 cincminutalment o podem veure si es podria fer en streaming EL -> T -> N(otificaci\u00f3) Ser\u00e0 exclusiu per alertes i amb poca consolidaci\u00f3 a base de dades i no revisitar\u00e0 el passat. Nom\u00e9s NOW()-ish.
"},{"location":"adr/2022-10-26-real-time_alarms/#positive-consequences","title":"Positive Consequences","text":"
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2022-10-26-real-time_alarms/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2022-10-26-real-time_alarms/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-26-real-time_alarms/#hi-calcular-la-taula-historica","title":"HI - Calcular la taula hist\u00f2rica","text":"
[example | description | pointer to more information | \u2026]
-
- Una \u00fanica l\u00f2gica si no tenim incremental
-
- No hem fet incremental amb DBT mai encara
-
- l'incremental pot reescriure dels dos darrers dies
-
- pots regenerar la taula sempre
-
- fa m\u00e9s complexe les alarmes individuals (over partitions)
DAG:\n(production_inverter) -> (alarm) -> (alrm2)\n\nwith ( Taula production_inverter (registry gapfilled, clean, alarms )) as foo\n\ntime | inverter_id | energy | n_readings | alarm_no_reading | alarm_zero_daylight\n10:00:00 16 34 1 FALSE FALSE\n09:55:00 16 0/NULL 0 TRUE NULL\n09:50:00 16 0 1 FALSE TRUE\n\nselect last row of foo where alarm = TRUE\n
"},{"location":"adr/2022-10-26-real-time_alarms/#rt-calcular-una-taula-real-time-i-una-historica-mes-lenta","title":"RT - Calcular una taula real-time i una hist\u00f2rica m\u00e9s lenta","text":"
[example | description | pointer to more information | \u2026]
-
- Logica m\u00e9s senzilla i r\u00e0pida que podem executar m\u00e9s sovint a airflow
-
- Permet amb un date_trunc() sobre les lectures fer l'agregaci\u00f3 i no cal rolling
-
- Igualment s'ha d'escriure la l\u00f2gica HI per tenir l'hist\u00f2ric correcte
-
- Podries tenir la taula d'estat actual m\u00e9s f\u00e0cilment
-
- La unificada s'hauria de fer unint la taula hist\u00f2rica amb la real-time
-
- requereix molt lag en escriure la hist\u00f2rica, o posar-hi la l\u00f2gica que necessites igualment a la opci\u00f3 HI
-
- Pot generar incongru\u00e8ncies entre el Real-Time (alarmed) que no surtin a l'hist\u00f2ric
-
- A projectes no li agrada tenir una taula estat actual i hist\u00f2ric separat
RT:
real-time, per les alarmes que necessiten notificaci\u00f3 immediata, i per cada tipus d'alarma
with foo as (\nselect\n count(energy),\n max(energy)\nfrom registry\nwhere NOW() - device.alarm_offset < time\ngroup by device\n)\nselect\n now() as time,\n count = 0 as no_reading_alarm,\n max = 0 as no_energy_alarm\nfrom foo\n
notify OK/KO en aquella execuci\u00f3 fent un diff de la taula anterior com ara
hist\u00f2ric: com tenim ara, al final del dia o d'una setmana, re-calcular les alarmes hist\u00f2ricament
"},{"location":"adr/2022-10-26-real-time_alarms/#st-trobar-la-manera-de-fer-stream-enlloc-de-batch","title":"ST- Trobar la manera de fer stream enlloc de batch","text":"
- Dolent perqu\u00e8 no podem regenerar-ho a posteriori
"},{"location":"adr/2022-10-26-real-time_alarms/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/","title":"Pipeline de dades revisited","text":"
- Status: proposed
- Deciders:
- Date:
Technical Story:
Com ser\u00e0 la obt? horaria calculada un cop cada hora? I quan passem a comptadors 15-minutals, segurem amb dbt run 15-minutal? Quina frescor de les dades cal? --> posem-ho a un adr diferent
"},{"location":"adr/2023-06-13-obt_plantes/#context-and-problem-statement","title":"Context and Problem Statement","text":"
Si les dades cincminutals s\u00f3n horaries, la obt de tots els indicadors tamb\u00e9?
Passarem a comptadors 15minutals en algun moment proper.
"},{"location":"adr/2023-06-13-obt_plantes/#decision-drivers","title":"Decision Drivers","text":"
- \u00das real de la obt
- Frescor necess\u00e0ria
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#considered-options","title":"Considered Options","text":"
- [option 1]
- [option 2]
- [option 3]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].
"},{"location":"adr/2023-06-13-obt_plantes/#positive-consequences","title":"Positive Consequences","text":"
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-06-13-obt_plantes/#option-1","title":"[option 1]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#option-2","title":"[option 2]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#option-3","title":"[option 3]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-obt_plantes/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/","title":"Pipeline de dades revisited","text":"
- Status: proposed
- Deciders: Diego, Roger, Pol
- Date: 2023-06-12
Technical Story:
https://trello.com/c/rmCTOJrH/205-decidir-com-farem-lobt-i-el-cincminutal-r%C3%A0pid
Passem a una macrotaula de dades llargues clau-valor per a les dades cincminutals?
Com ser\u00e0 la obt? horaria calculada un cop cada hora? I quan passem a comptadors 15-minutals, segurem amb dbt run 15-minutal? Quina frescor de les dades cal? --> posem la discussi\u00f3 de la obt a un adr diferent
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#context-and-problem-statement","title":"Context and Problem Statement","text":"
Ara que sabem molt m\u00e9s dbt i que incorporarem un prove\u00efdor de dades cincminutals amb dades llargues que deprecar\u00e0 l'esquema antic, ens plantegem si haur\u00edem de basar en una macrotaula de dades llargues per les dades cincminutals.
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#decision-drivers","title":"Decision Drivers","text":"
- \u00das real del cinc-minutal
- Frescor necess\u00e0ria
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#considered-options","title":"Considered Options","text":"
- normalitzar dset i fer un multisource normalitzat com est\u00e0 ara
- desnormalitzar lo que hi ha ara i fer un multisource com ens envia dset
- fer una pseudo-obt llarga a partir de dset i fer le multisource al mart
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: opci\u00f3 3
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-1","title":"[option 1]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-2","title":"[option 2]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-3","title":"[option 3]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026 numbers of links can vary -->
"},{"location":"adr/2023-09-12-redash_vs_superset/","title":"Pipeline de dades revisited","text":"
- Status: proposed
- Deciders: Dades
- Date: 2023-09-12
"},{"location":"adr/2023-09-12-redash_vs_superset/#context-and-problem-statement","title":"Context and Problem Statement","text":"
En el seu moment vam triar redash perqu\u00e8 podia tenir apis com a sources i el dev \u00e9s molt m\u00e9s c\u00f2mode perqu\u00e8 no est\u00e0 tant layeritzat (dataset/query - chart - dashboard).
A la pr\u00e0ctica no hem fet servir mai la lectura directa de les apis perqu\u00e8 les aut\u00e8ntificacions no estaven suportades, ens agradava tenir una c\u00f2pia, t'obligava a treballar en queries de l'sqlite de redash.
A m\u00e9s a m\u00e9s, redash delega la gesti\u00f3 d'usuaris a la db, cosa que implica que no pots limitar que un usuari pugui veure uns valors d'una query i no uns altres, per exemple filtrar per plantes la producci\u00f3, els permisos es fan a nivell de db.
La macrofase de plantmonitor i l'evoluci\u00f3 de l'equip de dades cap a superset amb els altres equips ens porta a replantejar-nos-ho.
Iniciem un spike de superset per a fer els nous dashboards.
"},{"location":"adr/2023-09-12-redash_vs_superset/#decision-drivers","title":"Decision Drivers","text":"
- \u00das real del cinc-minutal
- Frescor necess\u00e0ria
- \u2026
"},{"location":"adr/2023-09-12-redash_vs_superset/#considered-options","title":"Considered Options","text":"
- superset
- redash
- metabase -- en Diego comenta que en la versi\u00f3 free no tens la capa sem\u00e0ntica
"},{"location":"adr/2023-09-12-redash_vs_superset/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: ?
"},{"location":"adr/2023-09-12-redash_vs_superset/#positive-consequences","title":"Positive Consequences","text":"
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-09-12-redash_vs_superset/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/2023-09-12-redash_vs_superset/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#redash","title":"redash","text":"
\ud83d\udc4d
mustache
definir parametres a nivell de default de charts, de chart especific o dashboard
\ud83d\udc4e
no apte per heavy duty (cache regular)
"},{"location":"adr/2023-09-12-redash_vs_superset/#superset","title":"superset","text":"
\ud83d\udc4d
cohesi\u00f3 cooperativa -- tota la cooperativa fa servir superset
\ud83d\udc4e
No pots fixar filtres a nivell de chart al dashboard, cal fer-ne una c\u00f2pia. Implica que si volem canviar com s'ensenya un chart de planta, els haurem de canviar tots un a un.
El jinja \u00e9s funky. No \u00e9s clar com fer servir el \"filter_values\" a dins d'una query. Les queries din\u00e0miques s\u00f3n molt menys directes de fer que al redash.
"},{"location":"adr/template/","title":"Template","text":"
^-- [short title of solved problem and solution]
- Status: [proposed | rejected | accepted | deprecated | \u2026 | superseded by
[ADR-0005](0005-example.md)] - Deciders: [list everyone involved in the decision]
- Date: [YYYY-MM-DD when the decision was last updated]
Technical Story: [description | ticket/issue URL]
"},{"location":"adr/template/#context-and-problem-statement","title":"Context and Problem Statement","text":"
[Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question.]
"},{"location":"adr/template/#decision-drivers","title":"Decision Drivers","text":"
- [driver 1, e.g., a force, facing concern, \u2026]
- [driver 2, e.g., a force, facing concern, \u2026]
- \u2026
"},{"location":"adr/template/#considered-options","title":"Considered Options","text":"
- [option 1]
- [option 2]
- [option 3]
- \u2026
"},{"location":"adr/template/#decision-outcome","title":"Decision Outcome","text":"
Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].
"},{"location":"adr/template/#positive-consequences","title":"Positive Consequences","text":"
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/template/#negative-consequences","title":"Negative Consequences","text":"
- [e.g., compromising quality attribute, follow-up decisions required, \u2026]
- \u2026
"},{"location":"adr/template/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/template/#option-1","title":"[option 1]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/template/#option-2","title":"[option 2]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/template/#option-3","title":"[option 3]","text":"
[example | description | pointer to more information | \u2026]
- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- \u2026
"},{"location":"adr/template/#links","title":"Links","text":"
- [Link type] [Link to ADR]
- \u2026
"},{"location":"deprecated/TODOs/","title":"Views de plantmonitor a migrar a dbt","text":"
~ marca les que segurament no cal fer, al gust
view_availability.sql\n~view_q_hourly_debug.sql~\nview_q_monthly.sql\n~view_cos_phi_hourly_debug.sql~\nview_cos_phi_monthly.sql\nview_expected_power.sql\nview_hd_daily.sql\nview_ht_daily.sql\nview_ht_monthly.sql\nview_pr_hourly.sql\nview_target_energy.sql\n
"},{"location":"deprecated/TODOs/#agregacions","title":"Agregacions","text":"
No cal fer-les perqu\u00e8 son agraupacions que segurament farem servir m\u00e8triques de dbt tipus:
view_inverter_energy_daily.sql\nview_inverter_energy_monthly.sql\nview_meter_export_energy_daily.sql\nview_meter_export_energy_monthly.sql\nview_average_yearly_production_plant.sql\nview_satellite_irradiation.sql\n
A m\u00e9s a m\u00e9s les m\u00e8triques de dbt resolen el problema seg\u00fcent:
sum(a/x) -- horaria
sum(a/x) + sum(b/y) ... mensual wrong
sum((a+b+c)/(x+y+z)) -- mensual correct
"},{"location":"deprecated/TODOs/#views-per-limitar-lacces-a-la-info-dels-tecnics","title":"views per limitar l'acc\u00e9s a la info dels t\u00e8cnics","text":"
No cal fer-ho, limitarem l'acc\u00e9s a la wide table o farem servir els permisos del superset o ja veurem
view_plants_energes.sql\nview_plants_energetica.sql\nview_plants_ercam.sql\nview_plants_exiom.sql\nview_meterregistry_energetica.sql\nview_inverter_energy_daily_exiom.sql\nview_inverter_intensity_energes.sql\nview_inverter_intensity_ercam.sql\nview_inverter_intensity_exiom.sql\nview_inverter_power_energes.sql\nview_inverter_power_ercam.sql\nview_inverter_power_exiom.sql\nview_inverterregistry_energes.sql\nview_inverterregistry_ercam.sql\nview_inverterregistry_exiom.sql\nview_inverter_temperature_exiom.sql\nview_sensorirradiationregistry_energes.sql\nview_sensorirradiationregistry_ercam.sql\n
"},{"location":"deprecated/TODOs/#no-shan-de-fer-obsoletes","title":"no s'han de fer (obsoletes)","text":"
zero_inverter_power_at_daylight.sql\nzero_sonda_irradiation_at_daylight.sql\n
"},{"location":"desenvolupadors/2023-02-08-novu_notifications/","title":"Novu helpers and quickstart","text":""},{"location":"desenvolupadors/2023-02-08-novu_notifications/#swagger-api","title":"Swagger API","text":"
Novu api
"},{"location":"desenvolupadors/2023-02-08-novu_notifications/#overview","title":"overview","text":"
Features in novu get to the api waaaay before the get to the UI, so keep it at hand since most of the changes (adding removing subscribers, topics, etc.) will be done via python, react or postman via api rest calls.
"},{"location":"desenvolupadors/2023-02-08-novu_notifications/#helpers","title":"helpers","text":"
We have a handful of scripts to interact with novu via airflow or command-line scripts/notify_alert.py and scripts/novu_interface.py.
The notify_alert is ran by airflow to periodically check the alarm tables and trigger a notification on change.
novu_interface is more of a command-line util to do novu stuff manually, like adding subscribers or topics.
For example, to add a topic of a plant we'd do
$ python -m scripts.novu_interface --help\n\nUsage: python -m scripts.novu_interface [OPTIONS] URL API_KEY TOPIC_KEY\n TOPIC_NAME\n\nUsage: python -m scripts.novu_interface add-topic [OPTIONS] URL API_KEY\n TOPIC_KEY TOPIC_NAME\n\n\u256d\u2500 Arguments \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 * url TEXT [default: None] [required] \u2502\n\u2502 * api_key TEXT [default: None] [required] \u2502\n\u2502 * topic_key TEXT [default: None] [required] \u2502\n\u2502 * topic_name TEXT [default: None] [required] \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\u256d\u2500 Options \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 --help Show this message and exit. \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
$ python -m scripts.novu_interface add-topic {novu_base_url} {api_key} {topic_key} {topic_name}\n$ python -m scripts.novu_interface add-topic http://moll3.somenergia.lan:3000/v1 {api_key} asomada-topic-1 \"asomada general topic\"\n
and to get the topics
$ python -m scripts.novu_interface get-topic {novu_base_url} {api_key} {topic_key} {topic_name}\n\n2023-01-27 16:39:29,080 [INFO] {\"page\":0,\"totalCount\":1,\"pageSize\":10,\"data\":[\n {\"_id\":\"63d3ef030d600f1df1a3829e\",\"_environmentId\":\"6349479f3043d373c3d5fc07\",\"_organizationId\":\"6349479f3043d373c3d5fc02\",\"key\":\"asomada-topic-1\",\"name\":\"asomada general topic\",\"subscribers\":[]}\n]}\n
Let's add some subcribers to it:
$ python -m scripts.novu_interface add-subscriber http://moll3.somenergia.lan:3000/v1 {api_key} pol_recipient\n$ python -m scripts.novu_interface add-topic-subscriber http://moll3.somenergia.lan:3000/v1 {api_key} asomada-topic-1 pol_recipient\n
"},{"location":"desenvolupadors/2023-06-05-download_solargis_manually/","title":"Manually download a solargis reading","text":"
To download a specific day you just need to call
python -m scripts.solargis <dbapi> <apikey> 2023-05-13 2023-05-13 1 2 3 4\n 5 6 7 9 13 14 15 16 17 22 40\n
See python -m scripts.solargis --help for info on each parameter (essentially time range and plant id list) and the wiki or the pyxis server for credentials.
Note that to download one day you have to put the same from and to date. The from is start day and to date is end day. The script DOES NOT check for duplicates. You'll have to clean up yourself using the query time if you made a mistake.
"},{"location":"desenvolupadors/2023-06-10-overview-devs/","title":"Overview for devs","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#quickstart","title":"quickstart","text":"
To fully deploy the dbt models to the production schema you can do
dbt build -s tag:jardiner,tag:legacy --store-failures --full-refresh --target prod\n
You will need to have the prod profile in your ~/.dbt/profiles.yml file.
"},{"location":"desenvolupadors/2023-06-10-overview-devs/#current-state-of-the-project","title":"Current State of the project","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#pieces","title":"Pieces","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#repos","title":"repos","text":"
Plantmonitor
Fa d'ORM de les plantes, cont\u00e9 la api i \u00e9s codi de les raspberrypis. A m\u00e9s a m\u00e9s tamb\u00e9 t\u00e9 el codi d'ingesta d'apis de tercers.
Warning
La meitat del plantmonitor acabar\u00e0 deprecated pel prove\u00efdor de dades
Plant Reader
Lectura remota de les plantes. Actualment nom\u00e9s Asomada es llegeix remotament i es guarda a plant_lake.
Acabar\u00e0 deprecated pel prove\u00efdor de dades.
Actualment tamb\u00e9 allotja la lectura de la API del prove\u00efdor de dades.
Jardiner
dbt de les plantes. Actualment tot views.
"},{"location":"desenvolupadors/2023-06-10-overview-devs/#data-sources","title":"Data sources","text":"
- rPIs
- plant_reader
- Irradiation satellite data provider (SAT)
- Meteo forecast provider (METEO)
- Plant data provider (PLANT)
- Price provider (Planned) (PRICE)
"},{"location":"desenvolupadors/2023-06-10-overview-devs/#visualization","title":"Visualization","text":"
- redash
- novu notifications -> helpscout/mail
- superset (other \u00e0mbits of the cooperative)
"},{"location":"desenvolupadors/2023-06-10-overview-devs/#project-tructure","title":"Project tructure","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#actual","title":"Actual","text":"
flowchart LR\n\ndw[db/plants]\n\nplants -- pull 5' \\n plant_reader/dags --> lake[db/plant_lake]\nlake -- push 5' \\n airbyte --> dw\n\nplants -- pull 5' \\n plantmonitor/main.py --> rPIs -- push 5' --> dw\nplants -- pull 2h/12h \\n meter \\n ERP's import_tm_data_click.py --> ERP -- pull 20'\u00ba\\n meter data --> dw\nSAT -- pull daily\u00ba\\n irr/expected energy --> dw\nMETEO <-- pull daily\u00ba\\n meter data + irr/kWh forecast --> dw\nplants -- pull 15'\\n plant_reader/dags --> PLANT -- pull 15'\\n devices data --> db/plant_lake\n\ndw -- dbt views --> prod --> alarms\nprod --> alerts\nprod --> datasets
\u00ba: plantmonitor does it. Update rate defined at conf/startup_configuration.py and each task is run as a function.
"},{"location":"desenvolupadors/2023-06-10-overview-devs/#planned-scheme","title":"Planned scheme","text":"
flowchart LR\n\ndw[db/plants]\n\nplants -- pull 2h/12h \\n meter \\n import_tm_data_click.py --> ERP -- pull 20' \u00ba\\n meter data --> dw\nSAT -- pull daily\\n irr/expected energy --> dw\nMETEO <-- pull daily\\n meter data + irr/kWh forecast --> dw\nplants -- pull 15' --> PLANT -- pull 15'\\n devices data --> dw\n\n\ndw -- dbt fast --> fast[dbt_prod/fast_] --> union\ndw -- dbt --> obt --> alarms\nobt --> union\nunion -- notify.py --> alerts\nobt --> datasets
Despr\u00e9s de l'ADR d'estructura de pipeline
See Roadmap
"},{"location":"desenvolupadors/2023-07-10-dbt_cmds_utils/","title":"dbt commands","text":"
- Generar i servir la documentaci\u00f3 per a poder veure el directed graph.
$ dbt docs generate --project-dir dbt_jardiner --target pre\n\n$ dbt docs serve --project-dir dbt_jardiner --target pre --port 8010\n
- Pujar tots els CSVs de seeds
dbt seed --project-dir dbt_jardiner\n
- Podem passar variables per command line (en comptes de definir-les a dbt_project.yml i fer-les accesibles per diferents resources):
dbt run --vars '{\"variable\":\"value\"}'\n
Per fer testing amb un csv determinat:
dbt run --project-dir project_x --target testing --vars '{\"test_sample\":\"nom_del_csv\"}' -m model_x\n
"},{"location":"desenvolupadors/2023-07-13-codi_reciclable/","title":"Reusuable sqls","text":""},{"location":"desenvolupadors/2023-07-13-codi_reciclable/#dbt-sql-for-rolling-window","title":"dbt sql for rolling window","text":"
{%- set plant_alarm_ranges = dbt_utils.get_query_results_as_dict(ref('plant_alarm_ranges')) -%}\n\n{% for plant in plant_alarm_ranges %}\n\n {% if var('plants_code') == 'SomEnergia_Riudarenes_ZE' %}\n {%- set time_to_sum = '1 hour' -%}\n {% elif var('plants_code') == 'SomEnergia_Matallana' %}\n {%- set time_to_sum = '4 hour' -%}\n {% else %}\n {{ ref('accounts') }} accounts\n {% endif %}\n\n SELECT\n *,\n SUM(export_energy_wh) OVER (\n PARTITION BY plant_name\n ORDER BY \"time\"\n RANGE BETWEEN interval {{time_to_sum}} PRECEDING AND CURRENT ROW\n ) as export_energy_wh_sum\n FROM (\n select * from {{ref('meter_registry_cleaned')}} where codename = {{plants_code}}\n ) as meter_registry_cleaned_current_plant\n UNION ALL\n\n{% endfor %}\n
"},{"location":"desenvolupadors/2023-08-28-alta_planta/","title":"Context","text":"
Tenim un pipe de transformaci\u00f3 de dades que acaba en una serie de dashboards de superset que mostra dades de plantes des de el pipe de transformaci\u00f3 de dades arribant del nostre prove\u00efdor de dades.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#protocol-dalta-duna-planta-fotovoltaica","title":"Protocol d'alta d'una planta fotovoltaica","text":"
Per afegir una nova planta a la visualitzaci\u00f3, cal seguir un protocol documentat al Formulari d\u2019inscripci\u00f3 de noves plantes a Superset
El equip de gestio d'actius ha d'omplir aquest formulari amb les dades de la planta i ens ho han de comunicar per a que puguem seguir amb el proc\u00e9s nosaltres.
Obres en curs amb jardiner-admin
Aquesta tasca est\u00e0 en proc\u00e9s de ser portada al jardiner-admin. Per tant, aquest protocol pot canviar en el futur.
Necessites permisos d'administrador
Aquesta tasca \u00e9s per a desenvolupadors que treballen amb el pipe de transformaci\u00f3 de dades. Si no ets un desenvolupador, no cal que facis aquesta tasca.
Aqui sota ens dediquem a documentar t\u00e9cnicament els passos a seguir des de la perspectiva de desenvolupament.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#checklist-general-per-it","title":"Checklist general per IT","text":"
- Tenir acc\u00e9s als seg\u00fcents documents
- Mapeig de senyals
- Dades fixes de planta
- Info api solargis
- Rendiment de planta
- Producci\u00f3 actual de planta
- Omplir dades al excel de mapeigs d\u2019airbyte
- Afegir planta segons l\u2019estructura de plantmonitor
- Dades de Solargis de planta nova s\u00f3n al script a plantmonitor
- Dades apareixen al superset correctament
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#descripcio-de-dades","title":"Descripci\u00f3 de Dades","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-mapeig-de-senyals","title":"Mapeig de senyals de Planta","text":"
Per cada planta, GdA genera fitxers amb informaci\u00f3 de la planta per la seva gesti\u00f3. Aquests fitxers contenen informaci\u00f3 de senyals, aparells, plantes, etc. per\u00f2 sense el context del pipe de Jardiner. Com IT, portem aquesta informaci\u00f3 al fitxer de mapeig de senyals per a que el pipe de Jardiner pugui fer integrar les dades de la planta.
Un exemple d'aquest fitxer pot ser aquest.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-dades-fixes-de-planta","title":"Dades fixes de planta","text":"
Aquest fitxer cont\u00e9 dades fixes de la planta, com ara la pot\u00e8ncia de la planta, la seva ubicaci\u00f3, etc. Es trova a google drive i es transforma amb airbyte aqui
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-info-api-solargis","title":"Info API SolarGIS","text":"
GdA ha de crear una entrada per la planta nova al fitxer excel per aix\u00f2
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-rendiment-de-planta","title":"Rendiment de planta","text":"
T\u00e9 un nom tipus \"C\u00e0lcul Rendiment de Planta <nom planta>\". El podeu buscar al cercador de drive.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-produccio-actual-de-planta","title":"Objectius de producci\u00f3 de planta","text":"
Es troba al fitxer de google sheets aqui i es transforma amb airbyte aqui.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-unificacio-noms-projectes-generacio","title":"Unificaci\u00f3 Noms Projectes Generaci\u00f3","text":"
Cont\u00e9 els llistat de noms unificats en un sol fitxer. Ves a Unificaci\u00f3 Noms Projectes Generaci\u00f3.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-modelos-contadores-inversores-scadas","title":"Modelos contadores_inversores_SCADAS","text":"
Modelos contadores_inversores_SCADAS
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#descripcio-de-tasques","title":"Descripci\u00f3 de tasques","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-mapeig-airbyte","title":"Afegir dades al mapeig d'airbyte","text":"
Tenim un document on consolidem relacions entre UUIDs de senyals, aparells i plantes que inserim amb airbyte
Aquest mapeig es una traducci\u00f3 del fitxer de mapeigs.
Un cop heu acabat de fer el mapeig, cal engegar la tasca de sync de l'airbyte d'uuids per que les dades arribin a la base de dades. Heu de tenir especial cura d'omplir les columnes updated_at i aquelles que posen UUIDs, ja que amb elles se creuen les dades dins del pipe.
update_at s'actualitza automagicament
Al fitxer de google sheets hi ha un script que actualitza la columna updated_at quan hi ha hagut un canvi a nivell de fila. Es molt limitat i heu de supervisar que s'ha actualitzat correctament, ja que de vegades google sheets no ho fa. Podeu mirar a Extensions > Apps Script per veure el codi.
UUID s\u00f3n versi\u00f3 4
Els senyals porten UUID4 (han de ser versio 4) \u00fanics al mapeig de la planta indicada. Es poden generar UUID4 amb https://www.uuidgenerator.net/ O ben v\u00e9 amb un script de python que podeu afegir a ~/.bash_aliases:
#\u00b7function\u00b7to\u00b7genenerate\u00b7random\u00b7uuid4\u00b7using\u00b7python$\nfunction pyuuid4() {\npython -c \"import uuid$\nfor _ in range(${N:-1}):\n print(uuid.uuid4())\" ; }\n
i el podeu executar amb N=<n> pyuuid4 per a generar n UUID4.
$ N=10 pyuuid4\nbf98707e-688c-471c-8700-9ffeaf436531\n8f6bfcc0-48bf-4b7c-895f-0b2efa37dd63\nee5dcfd1-80ba-48fb-800d-5bc8c04d217f\nd32b0a23-c479-4ae9-bb54-43390fedf8e4\neced945e-7a91-4451-bbd5-974a82c9c0d4\nfbacbf3d-23f6-4361-b5ec-b48b390d9c09\nb9c4ea66-672e-4dca-ad08-fddacdfb3627\nae3e242e-9a57-4359-a72b-313a72b48348\n9e11a595-c32c-42e1-8549-a5161c0ea624\n172061ed-96b4-4aa3-b5bd-0c66e850383b\n
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-meteologica-plantmonitor","title":"Afegir planta per
meteologica amb
plantmonitor","text":"
Qu\u00e8 \u00e9s meteologica?
meteologica \u00e9s un servei que ens permet tenir dades meteorol\u00f2giques de les plantes fotovoltaiques. Aquest servei es comunica amb plantmonitor per a obtenir dades de les plantes. Es una API tipus SOAP i teniu la documentaci\u00f3 disponible aqui.
Qu\u00e8 \u00e9s plantmonitor?
Tots els scripts en aquest apartat fan refer\u00e8ncia al repositori de github plantmonitor. \u00c9s un projecte previ a jardiner que ens permet monitoritzar plantes fotovoltaiques quan l'obtenci\u00f3 de dades la gestionem nosaltres.
A plantmonitor tenim un script que ens permet afegir plantes noves. Aquest script s'ha de cridar amb un fitxer yaml que contingui les dades de la planta.
S'ha de crear la planta a la base de dades tant a la raspberrypi com a plantmonitor (no calen els dispositius, que els crea autom\u00e0ticament). Haureu d'afegir plantlocation, plantparameters i moduleparameters. Trobareu un exemple a plantmonitor
Un cop teniu el fitxer yaml, per exemple data/plant-nova.yaml, podeu cridar l'script addPlant.py amb el seg\u00fcent comandament:
PLANTMONITOR_MODULE_SETTINGS='conf.settings.prod' python addPlant.py data/plant-asomada.yaml\n
Haureu de tenir un fitxer .env.prod amb les variables d'entorn necess\u00e0ries per a que l'script funcioni correctament. Mireu a la documentaci\u00f3 de plantmonitor per a m\u00e9s informaci\u00f3.
De moment manualment cal afegir els registres de plantestimatedmonthlyenergy. L'any \u00e9s irrellevant. Si tenim hist\u00f2rics d'energia objectiu, els podeu afegir.
INSERT INTO\n public.plantestimatedmonthlyenergy(plantparameters, \"time\", monthly_target_energy_kwh)\nVALUES\n (33, '2022-01-01 00:00:00+01', 473630),\n (33, '2022-02-01 00:00:00+01', 453370),\n (33, '2022-03-01 00:00:00+01', 592040),\n (33, '2022-04-01 00:00:00+02', 617000),\n (33, '2022-05-01 00:00:00+02', 667210),\n (33, '2022-06-01 00:00:00+02', 648940),\n (33, '2022-07-01 00:00:00+02', 703030),\n (33, '2022-08-01 00:00:00+02', 644500),\n (33, '2022-09-01 00:00:00+02', 574580),\n (33, '2022-10-01 00:00:00+02', 524110),\n (33, '2022-11-01 00:00:00+01', 446490),\n (33, '2022-12-01 00:00:00+01', 442060);\n
-
Executar l\u2019script d\u2019alba i posta per a generar totes les albes i postes dels propers 10 anys
python -m scripts.sun_events -p Alcolea -s 2021-12-13 -e 2021-12-14\n
-
Afegir les dades al diccionari de solargis (Veure com aqui). Quan deixi de ser hard-coded ja no caldr\u00e0 aquest pas.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-solargis","title":"Afegir planta al script de SolarGIS","text":"
S'ha de modificar el codi de plantmonitor per incloure la nova planta. Podeu trobar un exemple de modificacions github
Amb la MR aprovada i fusionada, s'ha de fer git pull des de el servidor de producci\u00f3 i reiniciar el servei de plantmonitor.
ssh someone@somewhere.lan\nsource /home/<someone>/Envs/plantmonitor/bin/activate\ncd /home/<someone>/somenergia/plantmonitor\ngit pull\npython -m scripts.solargis --help\npython -m scripts.solargis [OPTIONS] DBAPI SOLARGIS_API_KEY FROM_DATE TO_DATE [PLANT_IDS]\nsudo supervisorctl restart plantmonitor\n
Aqui, someone \u00e9s el nom de l'usuari que t\u00e9 acc\u00e9s a somewhere.lan, i PLANT_IDS s\u00f3n els ids de les plantes que vau afegir a l'API de SolarGIS amb la PR anterior. Demaneu acc\u00e9s a somewhere.lan a l'equip de dades.
La API de SolarGIS requereix un token d'acc\u00e9s i es crida remotament amb supervisord, amb codi al repositori de plantmonitor.
L'API de SolarGIS \u00e9s SOAP
L'API es del tipus SOAP i no REST. \u00c9s a dir, demana un XML amb les dades de la planta. Podeu trobar informaci\u00f3 sobre com cridar-la a la seva documentaci\u00f3 d'usuari.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#dades-requerides-per-aparells","title":"Dades requerides per aparells","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-comptadors-a-lerp","title":"Afegir Comptadors a l'ERP","text":"
Les lectures a l'ERP s\u00f3n necessaries mentre metelogica faci servir les lectures de comptadors que plantmonitor copia de l'ERP. Tamb\u00e9 s\u00f3n necess\u00e0ries per al generationkwh. Per aix\u00f2 cal agafar les dades de Modelos contadores_inversores_SCADAS i afegir-les a l'erp Infrastructura -> Registradors -> tots els registradors.
- ip lan, port, porta d\u2019enlla\u00e7 i contrasenya
- tel\u00e8fon, si s'escau
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#serveis-de-somenergia-addicionals","title":"Serveis de SomEnergia addicionals","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#nagios","title":"nagios","text":"
Preguntar a Gesti\u00f3 d'Actius si encara fan servir el nagios. Si si, afegir-la al nagios o modificar-la en cas de canvi d\u2019ip p\u00fablica
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#opendata","title":"opendata","text":"
Cal afegir la info de la nova planta a taula literal que es fa servir a somenergia-opendata/som_opendata/queries/plantpower.sql. Un cop aquesta info estigui a plantmonitor, l\u2019opendata podr\u00e0 agafar-la directament per sql i no caldr\u00e0 fer aquest pas.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#generationkwh","title":"generationkwh","text":"
Si la planta est\u00e0 inclosa al generation, \u00e9s important afegir-la a l\u2019erp abans de la mitja nit del dia de posada a producci\u00f3. Si no, cal descartar c\u00e0lculs de drets del generation per a que incloguin els resultats. Si, a m\u00e9s, els drets s\u2019han comen\u00e7at a gastar (15 dies de marge de facturaci\u00f3), cal reperfilar els nous drets amb la producci\u00f3 afegida perqu\u00e8 no superi els drets ja atorgats.
La nova planta es crea fent servir l\u2019script a somenergia-generationkwh/scripts/genkwh_plants.py
Un exemple complert de com afegir una planta amb aquest script el podeu trobar a https://github.com/Som-Energia/somenergia-generationkwh/blob/master/scripts/genkwh_migrate_plantmeter_1_7_0_newplant.sh
Es recomana provar primer a un ERP de testing o local, i, despr\u00e9s aplicar-ho a producci\u00f3.
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#reperfilacio-de-dades","title":"Reperfilaci\u00f3 de dades","text":"
Un exemple d\u2019script de migraci\u00f3 en la que vam haver de reperfilar: https://github.com/Som-Energia/somenergia-generationkwh/blob/master/scripts/genkwh_migrate_plantmeter_1_7_2_fontivsolarfix.sh
"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/","title":"Incremental Dedup","text":"
Exemple de com fer un incremental amb deduplicaci\u00f3 en dbt.
{{\n config(\n materialized = 'incremental',\n unique_key = ['signal_uuid','ts'],\n incremental_strategy = 'merge',\n incremental_predicates = [\n \"DBT_INTERNAL_DEST.queried_at > dateadd(day, -1, current_date)\"\n ]\n )\n}}\n\nwhere\n ts < now() - interval '1 hour' {#- select only freshly ingested rows #}\n\n {% if is_incremental() -%}\n and queried_at > coalesce((select max(queried_at) from {{ this }}), '1900-01-01') and queried_at > now() - interval '2 hour'\n {%- endif %}\n
source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00 energia_comptador 03:00 null 04:00 potencia_inversor 03:00 50 +1d 04:00 potencia_inversor 03:00 50 04:00
Son ara les 15:04. en la materialitzada tindrem
source senyal ts valor potencia_inversor 14:00 67 energia_comptador 14:00 null
o bien, la \u00faltima fila no hi ser\u00e0:
source senyal ts valor potencia_inversor 14:00 67"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/#explicacio","title":"Explicaci\u00f3","text":"
where max(ts) = 14:00:
source senyal ts valor potencia_inversor 14:00 67 energia_comptador 14:00 null
where max(ts) - 24h:
source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00 potencia_inversor 03:00 50 +1d 04:00 potencia_inversor 03:00 50 04:00 potencia_inversor 14:00 67 energia_comptador 14:00 null
unique (signal_uuid, ts):
source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/#prova-practica","title":"prova pr\u00e0ctica","text":"
Abans d'executar l'incremental tenim aquest tres \u00faltims registres a materialized one hour late
group_name queried_at ts signal_code signal_device_type signal_device_uuid signal_frequency signal_id signal_is_virtual signal_last_ts signal_last_value signal_type signal_tz signal_unit signal_uuid signal_uuid_raw signal_value materialized_at SE_vallehermoso 2024-01-09 13:24:42.488123+01 2024-01-05 23:15:00+01 ce_eactexp meter 08bc8d88-4ea2-4ee9-b817-00e1f07debba 15 minutes 983894 false 2024-01-08 23:45:00+01 0 absolute Europe/Madrid kWh 646d1ec2-0ca2-4c22-9739-8161aafb224e 646d1ec2-0ca2-4c22-9739-8161aafb224e 0.0 2024-01-09 13:36:52.925381+01 SE_asomada 2024-01-09 13:24:42.488123+01 2024-01-05 23:00:00+01 s8102 meter c5ed9fab-e73c-40a5-acb3-113e22052fd6 5 minutes 1031060 false 2024-01-09 11:47:00+01 60.2 absolute Europe/Madrid V dd065f6a-4ded-41a8-98d3-ba239f9a2a47 dd065f6a-4ded-41a8-98d3-ba239f9a2a47 60.2 2024-01-09 13:36:52.925381+01 SE_tahal 2024-01-09 13:24:42.488123+01 2024-01-05 23:45:00+01 ce_ercapexp_er3 meter 5deb3780-02e2-4dcf-a6a7-de646991762c 15 minutes 983890 false 2024-01-08 23:45:00+01 0 absolute Europe/Madrid kVArh 31ae7d09-95db-447d-b5bf-358b056ef5bf 31ae7d09-95db-447d-b5bf-358b056ef5bf 0.0 2024-01-09 13:36:52.925381+01"},{"location":"desenvolupadors/2024-02-16-row-level-security/","title":"Row Level Security","text":""},{"location":"desenvolupadors/2024-02-16-row-level-security/#context","title":"Context","text":"
Tenim usuaris externs que necessiten mirar dades de plantes, per\u00f2 nom\u00e9s de les plantes que gestionen. Aix\u00f2 ho podem fer amb Row Level Security a Superset.
El proc\u00e9s \u00e9s el seg\u00fcent:
- Crear un rol
Manteniment_<nom empresa> si no existeix, pel cas de noves plantes. - Afegir una regla Row Level Security Pareu atenci\u00f3 al camp \"clause\", s'afegir\u00e0 com a condici\u00f3 where exclusiva (and) als datasets que seleccionem.
- Donar acc\u00e9s als charts d'un dashboard
- Afegir la o les usu\u00e0ries i assignar-los els rols [Gamma] i
[Manteniment_<nom empresa>]
I ja ho tens. Ho he fet amb energes amb la clause nom_planta in (noms plantes)
"},{"location":"desenvolupadors/2024-02-16-row-level-security/#decisions","title":"Decisions","text":"
Hem decidit a 02/2024 que no val la pena propagar l'empresa de manteniment en el pipe per a cobrir aquest use case, ja que no cal afegir 5m aquestes metadadades per a un tema de visualitzaci\u00f3 i permisos. Ho gestionem en la capa de visualitzaci\u00f3.
Canvis d'empresa de manteniment
Canvis d'empresa de manteniment s'hauran de fer al Superset (i a les metadades de la planta, per coher\u00e8ncia).
"},{"location":"desenvolupadors/2024-02-16-row-level-security/#referencies","title":"Refer\u00e8ncies","text":"
- Issue #101 a gitlab
- Targeta Trello
"},{"location":"desenvolupadors/2024-11-25-fill-missing-signals-data/","title":"Omplir forats de dades de senyals","text":""},{"location":"desenvolupadors/2024-11-25-fill-missing-signals-data/#omplir-forats-de-dades-de-senyals","title":"Omplir forats de dades de senyals","text":"
Actualment, el sistema de monitoratge de senyals de les plantes pot tenir forats de dades. Aquests forats poden ser deguts a problemes de comunicaci\u00f3, errors en la recollida de dades, etc.
Per com funciona el pipe de dades, les dades se descarreguen nom\u00e8s un cop si ha hagut una mesura incorrecte, o si no arriba a temps, s'ha de fer feina manual per tornar a descarregar les dades.
Recullim dos escenaris:
- Dades de comptadors amb granularitat de 15 minuts, actualitzades diariament
- Dades de senyals sense categoritzar, amb granularitat hor\u00e0ria i actualitzades cada 5 minuts.
Segons el escenari, el procediment a seguir \u00e9s diferent.
Dades no son a temps real
Les dades de senyals no s\u00f3n a temps real, i es disponibilitzen a l'api fins a 15 minuts despr\u00e8s de que han set generades. Per aix\u00f2 que no parlem de servir dades en temps real, perqu\u00e9 les dades disponibles arriben amb un retard.
"},{"location":"desenvolupadors/2024-11-25-fill-missing-signals-data/#dades-de-comptadors","title":"Dades de comptadors","text":"
Actualment les dades de comptadors es llegeixen di\u00e0riament amb un DAG d'Airflow (aqui). Aquest DAG llegeix executa un script que es pot trobar a aqu\u00ed.
De manera resumida, el script fa el seg\u00fcent:
- Descarrega les dades de la API de DSET al moment de l'execuci\u00f3.
- Llegeix les dades de comptadors de la base de dades de SomEnergia. Agafem totes les senyals amb la seva \u00faltima data de disponible.
- Comparem les dates disponibles amb les dates de la API de DSET.
- Si les dates de la API de DSET s\u00f3n m\u00e9s recents, les descarreguem i les guardem a la base de dades de SomEnergia.
- Si les dates de la API de DSET s\u00f3n iguals m\u00e9s antigues, no fem res
- Si les dates no hi son a la nostra base de dades, les descarreguem i les guardem a la base de dades de SomEnergia.
- Aix\u00f3 t\u00e9 un caveat que afecta a forats: si hi ha un forat per\u00f2 tenim dades m\u00e9s recents, no les podrem detectar per que ja tenim una mesura m\u00e9s recent.
- Per aix\u00f2, si hi ha un forat, s'ha de fer manualment.
"},{"location":"desenvolupadors/2024-11-25-fill-missing-signals-data/#esborrar-i-tornar-a-llegir-les-dades","title":"Esborrar i tornar a llegir les dades","text":"
Per omplir forats de dades de comptadors, s'ha de seguir el seg\u00fcent procediment:
-
Anar a la base de dades de SomEnergia i buscar la senyal amb el forat de dades. Per exemple: volem omplir dades de la planta Fontivsolar i Valteina entre el 2024-06-10 i el 2024-08-01.
select * from lake.dset_meters_readings dmr\nwhere group_name in\n(\n 'SE_fontivsolar',\n 'SE_valteina',\n)\nand ts between '2024-06-10' and '2024-08-01'\n
-
Esborrem aquestes dades per garantir que el script detecti els forats despr\u00e8s. Per aix\u00f2 nom\u00e8s cal canviar la query anterior per una query de delete.
delete from lake.dset_meters_readings dmr -- nou\nwhere group_name in\n(\n 'SE_fontivsolar',\n 'SE_valteina',\n)\nand ts between '2024-06-10' and '2024-08-01'\n
Tingues cura amb el delete
El delete esborra les dades de la base de dades. Assegura't de que les dades que esborres s\u00f3n les correctes.
Lectura de api es fa a nivel de senyal, no de planta
Les dades de comptadors es llegeixen a nivell de senyal, no de planta. Per aix\u00f2, s'ha de buscar les senyals amb forats, no les plantes. Aqui esborrem filtrant per planta o group_name per\u00f2 podriem fer-ho per signal_name tamb\u00e9.
-
Executar el script de lectura de comptadors. Aquest script es pot trobar a aqu\u00ed.
python scripts/read_dset_meters.py\n--db-url <URI_DB> \\\n--api-base-url https://api.dset-energy.com \\\n--api-key <DSET_API_KEY> \\\n--schema lake \\\n--sig-detail \\\n--apply-k-value \\\n--return-null-values \\\n--date-from 2024-06-10Z \\\n--date-to 2024-08-01Z \\\n--max-workers 4\n
Si tens dubtes revisa la documentaci\u00f3 del script amb python scripts/read_dset_meters.py --help.
-
Comprovar que les dades s'han omplert correctament. Per aix\u00f2, torna a executar la query de l'inici per veure si les dades s'han omplert correctament. Deuria ser aix\u00ed, i hauries de veure les dades que has omplert.
select * from lake.dset_meters_readings dmr\nwhere group_name in\n(\n 'SE_fontivsolar',\n 'SE_valteina',\n)\nand ts between '2024-06-10' and '2024-08-01'\n
-
Si les dades s'han omplert correctament, ja haur\u00e0s acabat. Si no, revisa els logs del script per veure si hi ha hagut algun error.
"},{"location":"desenvolupadors/2024-11-25-fill-missing-signals-data/#executar-dbt-full-refresh","title":"Executar
dbt --full-refresh","text":"
Amb els forats omplerts, s'han de refer i propagar calculs fets amb dbt al passat amb les dades noves. Com no fem aquest calcul cada vegada al ser molt intensiu, nom\u00e9s el fem per els \u00faltims dos dies de dades. Aquesta l\u00f3gica es troba aqu\u00ed
Per aix\u00f2, s'ha de fer un dbt --full-refresh per que dbt torni a calcular les dades amb les dades noves del passat.
dbt build -s tag:jardiner --store-failures --full-refresh --target prod\n
Aquesta tasca \u00e9s molt intensiva i pot trigar molta estona. Per aix\u00f2, s'ha de fer amb cura i en un moment de baixa c\u00e0rrega.
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/","title":"Macrofase Jardiner 2023","text":"
Document de Roadmap
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#roadmap-overview","title":"Roadmap overview","text":""},{"location":"projecte/2023-06-03-macrofase%20roadmap/#flux-de-dades","title":"Flux de dades","text":"
flowchart LR\n\nrPIs -- push 5' --> dw[db/plants]\nERP -- pull 20' \u00ba\\n meter data --> dw\nSAT -- pull daily\\n irr/expected energy --> dw\nMETEO <-- pull daily\\n meter data + irr/kWh forecast --> dw\nPLANT -- pull 15'\\n devices data --> dw\n\ndw -- dbt views --> prod --> alarms\nprod --> alerts\nprod --> datasets
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-1","title":"Fita 1","text":"
Prioritat 1
Aprox 1-2 fases
Definition of Done (DoD): veure per tema
Cost en mitjos dies de l'equip sencer entre par\u00e8ntesis (N)
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#dashboard-tecnics-de-plantes-critiques","title":"Dashboard t\u00e8cnics de plantes cr\u00edtiques","text":"
Pulir el dashboard de t\u00e8cnics per a Asomada i Llanillos.
DoD: TODO
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#obtencio-de-dades-de-tots-els-equips-i-totes-les-plantes","title":"Obtenci\u00f3 de dades de tots els equips i totes les plantes","text":"
- Definir estrat\u00e8gia d'obtenci\u00f3 de dades (remot/hardware/api's a plataformes...) (1)
- Coordinaci\u00f3 amb altres equips de la cooperativa per veure altres alternatives (6)
- Proves pilot amb alg\u00fan prove\u00efdor (8)
- [-] Analisi de prove\u00efdors/alternatives i selecci\u00f3 de partner per externalitzar l'obtenci\u00f3 de dades dels inversors i sondes (2)
-
[-] Obtenci\u00f3 de dades tots els comptadors per IP.
-
Adaptaci\u00f3 a frequ\u00e8ncia quarthoraria. (3)
-
[-] Suport a la integraci\u00f3 de prova pilot (6)
- Falta saber com es diferencien les plantes
DoD:
- Els equips mencionats en el pipe
- Dset integrat al pipe
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#introduccio-al-jardiner","title":"Introducci\u00f3 al jardiner","text":"
Estat del codi i rep\u00e0s de l'estructura. Documentaci\u00f3 per devs.
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#dt-obt-i-fast-lane","title":"[DT] OBT i fast-lane","text":"
Crear la obt cincminutal incremental i/o el fast-lane cincminutal i la obt di\u00e0ria.
Crear el live per late arriving facts de comptadors i apis.
En funcionament normal s\u00f3n delays de m\u00e0xim un dia. Amb incid\u00e8ncia poden ser 1-2 mesos m\u00e0xim. I en casos excepcionals, molt m\u00e9s (candidat a full-refresh manual o alguna altra cosa)
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#aterracio-de-tot-plegat","title":"Aterraci\u00f3 de tot plegat","text":"
- Intro per devs
- M\u00e9s doc?
- Afegit la prove\u00efdora de dades al pipe
- Decidir com (multisource/sol)
- Refer el pipe (fast+slow)
- Que no tardi 30s
- DBT standards
- Definir el dashboard de t\u00e8cnics
- Volem fer servir redash?
- ~Parlar amb GA per si volen el Dashboard diferent al actual~ El volen tal com s\u00f3n
- Dashboard tecnics Exiom amb Asomada
- Dashboard tecnics Exiom amb Llanillos (source original \u00e9s dset)
- Deute t\u00e8cnic Dev
- CDD
- Private doc github project
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-2-dashboard-overview-plantes","title":"Fita 2: Dashboard Overview plantes","text":"
Prioritat 4
Disseny d'un Dashboard amb un llistat de totes les plantes amb els principals indicadors (mesures en temps real i acumulades i resum alarmes). Pendent fer una proposta per part de GA. Des d'aquest dashboard s'ha de poder accedir al Dashboard de cada planta per obtenir informaci\u00f3 detallada. Esborrany inicial falta versi\u00f3 actualitzada. (6)
Implementaci\u00f3 (9)
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-3-dashboard-planta","title":"Fita 3: Dashboard planta","text":"
Prioritat 5
Definir versi\u00f3 revisada del Dashboard de planta per a totes les plantes: Unifilar din\u00e0mic, dades fixes i reempla\u00e7ar alarmes, millorar la velociatat i treure el dataset de sota, afegir solargis + coses que puguin sortir despr\u00e9s (3)
Implementar versi\u00f3 revisada del Dashboard de planta per a totes les plantes: Unifilar din\u00e0mic, dades fixes i reempla\u00e7ar alarmes, millorar la velocitat i treure el dataset de sota, afegir solargis + coses que puguin sortir despr\u00e9s (3)
Dashboard Ringsted per plantes -- particularitats (4)
Dashboard Gasdtrup -- particularitats (4)
Dashboard de planta t\u00edpica (Alcolea, etc) (0)
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-4-dashboard-tecnics","title":"Fita 4: Dashboard t\u00e8cnics","text":"
Prioritat 6
Definir la versi\u00f3 final amb grafiques de Solar Gis, Alarmes i pot ser alarmes. Pendent fer una revisi\u00f3 de les funcionalitats per part de GA. Incloure un dashboard per cadascuna de les empreses de manteniment. Al dashboard unificat d'alarmes estat actual afegir des de quan est\u00e0 activada l'alarma
Implementar la versi\u00f3 final amb grafiques de Solar Gis, Alarmes i pot ser alarmes. Pendent fer una revisi\u00f3 de les funcionalitats per part de GA. Incloure un dashboard per cadascuna de les empreses de manteniment
Fer el Dashboard per a 8 empreses instal\u00b7ladores. Haur\u00edem de passar a Superset que t\u00e9 una millor gesti\u00f3 d'usuaris?
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-5-dashboard-ineco","title":"Fita 5: Dashboard INECO","text":"
Prioritat 7
Per definir
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-6-noves-queries","title":"Fita 6: Noves queries","text":"
Prioritat 8
- Query: Curves HMCIL + lectures comptador (reclamacions)
- Query: Curves HMCIL + previsi\u00f3 (seguiment)
- Alarma strings per comparaci\u00f3 en plantes grans >1MW
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-7-dashboard-gurb","title":"Fita 7: Dashboard GURB","text":"
Prioritat 9
"},{"location":"projecte/2023-06-03-overview/","title":"Overview del projecte per a usuaris","text":"
Descripci\u00f3 del projecte des del punt de vista de l'usuari
"},{"location":"projecte/2023-06-03-overview/#context-de-naixement","title":"Context de naixement","text":"
- A quin objectiu respon? Guanyar autonomia dins del mercat el\u00e8ctric.
- Context: per qu\u00e8 \u00e9s important? per guanyar en efici\u00e8ncia i optimitzar la producci\u00f3 (problema)
- Intenci\u00f3: on volem arribar? tenir un sistema de monitoritzaci\u00f3 unificat per a totes les plantes de producci\u00f3 (soluci\u00f3)
- Criteris d\u2019\u00e8xit: com mesurar progr\u00e9s i \u00e8xit?
- N\u00famero de plantes que poden gestionar per persona (20-25 plantes/persona)
- Agilitat en detectar incid\u00e8ncies (1.800 \u20ac/dia per p\u00e8rdues de producci\u00f3 seria el cas m\u00e9s extrem per una planta de 5MW. Si triguem una setmana en detectar-ho serien 12.600 \u20ac)
"},{"location":"projecte/2023-06-03-overview/#peces","title":"Peces","text":"
- Usuaris: T\u00e8cnics de planta, Gesti\u00f3 d'Actius, Altres \u00e0mbits de l'ET, IT
- Altres Rols: Prove\u00efdors d'internet
- Interf\u00edcies de contacte:
- redash
- Helpscout/notificacions
- erp
"}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 9ed08e1..74b57bc 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ