Adaptations to industry demand #317

hazemakhalek · 2024-05-28T12:04:26Z

Closes #315

Changes proposed in this Pull Request

The PR further enhances the carrier aggregation functions by sourcing the needed data from the helpers.py file.
The PR also redistributes the fuels to the right carriers, with bio now reflecting biofuels in general rather than only biomass.

Checklist

I tested my contribution locally and it seems to work fine.
Code and workflow changes are sufficiently documented.
Newly introduced dependencies are added to envs/environment.yaml and envs/environment.docs.yaml.
Changes in configuration options are added in all of config.default.yaml, config.tutorial.yaml, and test/config.test1.yaml.

finozzifa · 2024-05-29T08:30:57Z

scripts/helpers.py

@@ -620,26 +620,22 @@ def get_conv_factors(sector):
            "Gas Coke": 0.007326,
            "Refinery gas": 0.01375,
            "Coal Tar": 0.007778,
+            "Paraffin waxes": 0.01117,
+            "Ethane": 0.01289,
+            "Oil shale": 0.00247,
        }
    return fuels_conv_toTWh


 def aggregate_fuels(sector):


Hi @hazemakhalek,

thanks a lot for this great PR.

Problem I see

I noticed that not all the fuels available in get_conv_factors() are present in aggregate_fuels(). Moreover there are also some fuels available in aggregate_fuels() which are not present in get_conv_factors() .

Why I think it is a problem

My understanding of the code is that what appears in get_conv_factors() should be also "mapped" in aggregate_fuels().

In build_base_industry_totals.py we have in fact the dictionary fuel_dict being built as follows

# Fetch the fuel categories from the helpers script ( gas_fuels, oil_fuels, biomass_fuels, coal_fuels, heat, electricity, ) = aggregate_fuels("industry") # Create fuel dictionary to use for mapping all fuels to the pypsa representative fuels fuel_dict = { element: var_name for var_name, element_list in [ ("gas", gas_fuels), ("oil", oil_fuels), ("biomass", biomass_fuels), ("heat", heat), ("coal", coal_fuels), ("electricity", electricity), ] for element in element_list }

Such dictionary is then used as to populate the "Carrier" column. Namely:

df["carrier"] = df["Commodity"].map(fuel_dict)

Of course, it could be that my understanding of the code is incorrect. In this case I will drop this review.

What I propose to enrich

The fuels available in get_conv_factors(), but not present in aggregate_fuels() are:

'Additives and Oxygenates', 'Refinery Gas', 'Coal Tar', 'Brown Coal Briquettes', 'Gas coke', 'Aviation gasoline', 'Gasoline-type jet fuel', 'Peat Products', 'Oil shale', 'Bio jet kerosene', 'Paraffin waxes'

The fuels available in aggregate_fuels(), but not present in get_conv_factors() are

'Natural gas (including LNG)', 'Coke Oven Gas', 'Natural Gas (including LNG)', 'Black Liquor', 'Biogases', 'Gasworks Gas', 'Blast Furnace Gas'

Hey @finozzifa ,

Thanks for the quick review! You're interpretation is correct.

I tested it quickly now and the first part of your suggestion is correct and need to be added. As of now, these fuels are calculated but not added to the aggregate value of the representative fuel. (e.g. Aviation gasoline is not added to Oil)

The second part however is intended, the fuels you show here are given in the dataset in energy units already (Terajoules)
These are handled on the line;

df.loc[index_energy_TJ, "Quantity_TWh"] = df.loc[index_energy_TJ].apply( lambda x: x["Quantity"] / 3600, axis=1)

The fuels conversion factors are only used to convert from mass or volume to energy.

Hi @hazemakhalek ,

thanks for your answer and for your explanation :)

In order words, we should include all entries from get_conv_factors() to aggregate_fuels(), whereas it is fine if aggregate_fuels() misses some entries available in get_conv_factors().

Did I get it right?

speak soon,

Fabrizio

Answer from PyPSA-Earth-Sec meeting on May 30th 2024

All fuels should be present in aggregate_fuels(), but not all fuels should be in get_conv_factors().

Thanks @hazemakhalek

Great discussion!
To add, some factors in the list may be typos or refuses; for simplicity, the naming may be revised to be normalized and maybe avoid duplicated entries that may introduce noise.

What do you think?

Hey Davide, I agree with you in general.
The typos in the data could be handled in a more consistent way to correct the trailing spaces, typos of the source.
If you handle this in a specific way for the power model let me know so we can make it coherent.

If not, I will start an issue which will be good for as an early task for some new comers.

for more information, see https://pre-commit.ci

davide-f

Great:D
Added my 5 cents :) maybe good to discuss tomorrow?!?!

scripts/build_base_energy_totals.py

davide-f · 2024-06-05T22:43:44Z

scripts/build_base_energy_totals.py

@@ -113,12 +132,16 @@ def calc_sector(sector):
            sectors_dfs[sector] = df_sector.copy()

            if sector == "consumption by households":
+                if snakemake.config["sector"]["coal"]["shift_to_elec"]:


It seems to be that here as well operations may be similar and may be aggregated.
For example, if we define a dictionary like:

d_config = { "consumption by households": ["Electricity"], "services": ["oil", "biomass", ...] ... }

And we also standardize the oil_fuels/biomass_fuels etc in an automated way, the script can be significantly improved.
Probably worth to keep in mind this and add an issue about it?
Revising this script can be quite heavy and I don't think it is a priority

I think an issue here will be useful. Again, I think this is an interesting task for early users especially with software background

Added to issue: #358

davide-f · 2024-06-05T22:45:55Z

scripts/helpers.py

@@ -620,26 +620,22 @@ def get_conv_factors(sector):
            "Gas Coke": 0.007326,
            "Refinery gas": 0.01375,
            "Coal Tar": 0.007778,
+            "Paraffin waxes": 0.01117,
+            "Ethane": 0.01289,
+            "Oil shale": 0.00247,
        }
    return fuels_conv_toTWh


 def aggregate_fuels(sector):


Great discussion!
To add, some factors in the list may be typos or refuses; for simplicity, the naming may be revised to be normalized and maybe avoid duplicated entries that may introduce noise.

What do you think?

scripts/helpers.py

for more information, see https://pre-commit.ci

davide-f · 2024-07-18T08:52:30Z

scripts/build_base_energy_totals.py

@@ -243,7 +288,64 @@ def calc_sector(sector):
                    ].Quantity_TWh.sum(),
                    4,
                )
+            elif sector == "other energy":


These part may be improved (now or in the future) to be more compact code-wise.

davide-f

Great @hazemakhalek :D
I've added a recommendation and then for me the PR is ready to fly

scripts/build_base_industry_totals.py

for more information, see https://pre-commit.ci

hazemakhalek · 2024-07-19T21:44:03Z

@davide-f Done, please merge the PR

davide-f · 2024-07-19T22:57:53Z

Great :D done :)

adapt carrier aggregation method

045b0a8

finozzifa reviewed May 29, 2024

View reviewed changes

hazemakhalek and others added 2 commits May 31, 2024 14:35

Merge branch 'main' into adaptations_to_industry_demand

0600f44

[pre-commit.ci] auto fixes from pre-commit.com hooks

31a3354

for more information, see https://pre-commit.ci

davide-f reviewed Jun 5, 2024

View reviewed changes

hazemakhalek and others added 3 commits June 18, 2024 12:48

enhance the industry scripts and adapt the fuel aggregation

e870f90

Merge branch 'main' into adaptations_to_industry_demand

d410171

[pre-commit.ci] auto fixes from pre-commit.com hooks

f504501

for more information, see https://pre-commit.ci

hazemakhalek marked this pull request as ready for review June 24, 2024 15:03

hazemakhalek and others added 15 commits June 24, 2024 17:08

Update build_base_industry_totals.py

3944831

Update build_base_energy_totals.py

2a965ad

minor bug in build_base_energy_totals

e9cd429

[pre-commit.ci] auto fixes from pre-commit.com hooks

a690b66

for more information, see https://pre-commit.ci

Update build_base_industry_totals.py

c812db1

adapt build_base_energy and build_base_industry to params

3233fa7

add missing param in snakefile

3219a68

[pre-commit.ci] auto fixes from pre-commit.com hooks

eaacba4

for more information, see https://pre-commit.ci

fix param in snakefile

ab04ea8

Merge branch 'main' into adaptations_to_industry_demand

cfb9d7b

[pre-commit.ci] auto fixes from pre-commit.com hooks

2a7972e

for more information, see https://pre-commit.ci

reset config.pypsa-earth.yaml

92b3c19

[pre-commit.ci] auto fixes from pre-commit.com hooks

68e8888

for more information, see https://pre-commit.ci

Update Snakefile

698bda7

handle exception to avoid reference before assignment error

da82c5e

davide-f reviewed Jul 18, 2024

View reviewed changes

Update config.test_myopic.yaml to include new parameter

b0cee2a

hazemakhalek requested a review from davide-f July 18, 2024 10:52

hazemakhalek mentioned this pull request Jul 18, 2024

Improve build_base_energy_totals.py #358

Open

davide-f reviewed Jul 18, 2024

View reviewed changes

scripts/build_base_industry_totals.py Outdated Show resolved Hide resolved

Update build_base_industry_totals.py

e847f45

hazemakhalek and others added 3 commits July 19, 2024 16:35

Update Snakefile

230e173

[pre-commit.ci] auto fixes from pre-commit.com hooks

d73648f

for more information, see https://pre-commit.ci

Update build_base_industry_totals.py

5f1cac9

davide-f merged commit b7da641 into main Jul 19, 2024
4 of 5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adaptations to industry demand #317

Adaptations to industry demand #317

hazemakhalek commented May 28, 2024

finozzifa May 29, 2024 •

edited

Loading

hazemakhalek May 29, 2024

finozzifa May 29, 2024

finozzifa May 30, 2024 •

edited

Loading

davide-f Jun 5, 2024

hazemakhalek Jun 18, 2024

davide-f left a comment

davide-f Jun 5, 2024

hazemakhalek Jun 18, 2024

hazemakhalek Jul 18, 2024

davide-f Jun 5, 2024

davide-f Jul 18, 2024

davide-f left a comment

hazemakhalek commented Jul 19, 2024

davide-f commented Jul 19, 2024

Adaptations to industry demand #317

Adaptations to industry demand #317

Conversation

hazemakhalek commented May 28, 2024

Closes #315

Changes proposed in this Pull Request

Checklist

finozzifa May 29, 2024 • edited Loading

Choose a reason for hiding this comment

Problem I see

Why I think it is a problem

What I propose to enrich

Choose a reason for hiding this comment

Choose a reason for hiding this comment

finozzifa May 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davide-f left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davide-f left a comment

Choose a reason for hiding this comment

hazemakhalek commented Jul 19, 2024

davide-f commented Jul 19, 2024

finozzifa May 29, 2024 •

edited

Loading

finozzifa May 30, 2024 •

edited

Loading