How to solve this error java.lang.OutOfMemoryError: Java heap space

[ODOU]

Hi all,
I am getting this error java.lang.OutOfMemoryError: Java heap space while running scenario.commit(comment) command. I suspect the model I am building is a bit large (Regional with 15 nodes [countries]). The exported scenario file in csv is about 50MB. Is there anyway to expand the allocated memory on one hand and on the other hand how to make the model solve faster while launching scenario.solve() ?

I will highly appreciate the support of the community. Regards,

[glatterf42]

Hi Odou,
an error like yours usually means on of two things happened: either you didn't assign enough heap space to or there was too much memory consumed on the program side. This might be due to bad programming (i.e. a memory leak) or due to incorrect use (e.g. trying to store lots and lots of objects in heap memory without ever releasing them). From what I've read, 90% of the time the issue is one the code side. However, since message_ix and ixmp are in extensive use, I'm not convinced that there is a memory leak issue in a standard function like committing to the database. So you might want to try to increase your heap space, but also: "In [509]" from your screenshot sounds like a lot of input lines. If you do create and commit a lot of scenarios in the notebook(or code file) you get the error from, it might be worth investigating if you could move some of these scenarios to a new notebook. This would allow you to close the connection to the database in one notebook and open a new one in the new notebook, which might release stress from your heap space.
A lot of this answer is paraphrased from stackoverflow, so I'd recommend reading that answer, too. They further recommend profiling your program and you might be able to do so following the links provided there. However, you could try my suggestions first. Instructions on how to increase heap space can also be found in the answer I linked here.

Please let us know if this solves your issue :)

[khaeru]

Underlying the message_ix package is another one called ixmp that handles data storage. By default, when using current versions of message_ix and ixmp, you are using something called the ixmp JDBCBackend, which is built in Java.

The ixmp.JDBCBackend documentation gives some examples of how to pass arguments for starting the Java Virtual Machine (JVM); the one particular example is about increasing the Java heap space (-Xmg4G means to allocate 4 GB of memory for the JVM; this is fully explained in the further linked Java documentation).

Practically, you can cause this to always happen by modifying your ixmp configuration file (usually somewhere like $HOME/.local/share/ixmp/config.json). If you see:

{
  "platform": {
    "default": "local",
    "local": {
      "class": "jdbc",
      "driver": "hsqldb",
      "path": "/home/username/.local/share/ixmp/localdb/default"
    }
  }
}

…then edit to add the jvmargs, for example like:

{
  "platform": {
    "default": "local",
    "local": {
      "class": "jdbc",
      "driver": "hsqldb",
      "jvmargs": "-Xmx10G",
      "path": "/home/username/.local/share/ixmp/localdb/default"
    }
  }
}

This is equivalent to passing jvmargs="-Xmx10G" every time you call ixmp.Platform(name="local", …).

Of course, the correct choice of heap size depends on your model needs, system available memory, and other factors; there is no one specific size.

[khaeru]

to use HiGHS what would be requirement and how long it can take to implement it ?

To be clear when I say “we have talked about” I mean “we have not actually tried (or succeeded).” In trying, we might figure out what kind of adaptations would be needed (if any), and then one could guess how long those might take.

As for trying, it's a one-line change:

message_ix/message_ix/model/MESSAGE/auxiliary_settings.gms

Line 23 in 139c832

option LP = CPLEX ;

If you do try, please share your experience in a fresh discussion thread, rather than this one about Java heap space.

P.S. I am not sure if the limit on the LP problem size for the GAMS free/trial license is for CPLEX in particular, or for any solver. In the latter case, this switch would not solve the problem.

[brendandanaher]

Thank you @khaeru. Modifying the config.json file worked very well for me at first. However, when running my code, over time it eventually gets slower and slower every time I re-initialize the scenario as seen in the following code block.

import message_ix
import ixmp as ix
from ixmp import Scenario
from message_ix import make_df
#from functions.format_demand import format_demand
import pandas as pd
from pathlib import Path
from message_ix.reporting import Reporter
from update_units import unit_correction
import numpy as np


# Stand-alone model
modelName = 'MESSAGEix_CA_Hydrogen'      # A MESSAGE model name
scenarioName = 'Carbon Price'       # A MESSAGE model scenario name
ssp = 'SSP1'

mp = ix.Platform(name="p1")

# Creating a new MESSAGE stand-alone model
msg = message_ix.Scenario(mp, modelName, scenarioName,
                            version='new', annotation='')

I've gotten to the point where it takes 20 minutes to run this simple segment of code (this took 30 seconds originally) and I have "jvmargs" set to -Xmx12G in my config.json file. If anyone has a solution to this issue that would be much appreciated. I have attempted clearing memory and temporary files etc with no success. Not sure what else I could try.

config.json

[glatterf42]

Moved this to #825 since it's a new topic :)

[meng25meng]

I have the same problem, about the solver, have you solved this problem

[SongminYu]

Hi everyone who encounters this error message, I met the same problem and now it is "solved". Maybe this could also help so I leave my steps here.

• First, I don't know the reason but I don't find the config file. I am using ixmp3.8 with Macbook. But I managed to edit it in PyCharm (Help > Edit Custom VM Options....). However, after doing this, I still see the problem.
• From the other discussion (Loading Scenarios from Platform gets very slow over time #825), I saw recommendations by @khaeru to use several platforms/databases. I tried but it seems I can only use "local" or "default", which I think are the same database?
• In the end, I found the database folder user/.local/share/ixmp/localdb and removed everything in it (3GB+). This "solved" the problem.