Plotting Errors

[prajwalvinod]

Hi @LucasAlegre,

After running a bunch of simulations for my environment, I try to plot some of the results that I get from the csv file but I get the following error:

python plots.py -f Outputs\dqn_test_conn0_ep1
Traceback (most recent call last):
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\groupby.py", line 1490, in array_func
    result = self.grouper._cython_operation(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\ops.py", line 959, in _cython_operation
    return cy_op.cython_operation(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\ops.py", line 657, in cython_operation
    return self._cython_op_ndim_compat(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\ops.py", line 497, in _cython_op_ndim_compat
    return self._call_cython_op(
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\ops.py", line 541, in _call_cython_op
    func = self._get_cython_function(self.kind, self.how, values.dtype, is_numeric)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\ops.py", line 173, in _get_cython_function
    raise NotImplementedError(
NotImplementedError: function is not implemented for this dtype: [how->mean,dtype->object]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Anaconda3\Lib\site-packages\pandas\core\nanops.py", line 1692, in _ensure_numeric
    x = float(x)
        ^^^^^^^^
ValueError: could not convert string to float: "{'r': 0.0, 'l': 20000, 't': 9204.664227}"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Anaconda3\Lib\site-packages\pandas\core\nanops.py", line 1696, in _ensure_numeric
    x = complex(x)
        ^^^^^^^^^^
ValueError: complex() arg is a malformed string

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\prajw\PythonFiles\Thesis\RL_for_Freeway\Hegyi_Model\plots.py", line 90, in <module>
    plot_df(main_df, xaxis=args.xaxis, yaxis=args.yaxis, label=next(labels), color=next(colors), ma=args.ma)
  File "C:\Users\prajw\PythonFiles\Thesis\RL_for_Freeway\Hegyi_Model\plots.py", line 45, in plot_df
    mean = df.groupby(xaxis).mean()[yaxis]
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\groupby.py", line 1855, in mean
    result = self._cython_agg_general(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\groupby.py", line 1507, in _cython_agg_general
    new_mgr = data.grouped_reduce(array_func)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\internals\managers.py", line 1503, in grouped_reduce
    applied = sb.apply(func)
              ^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\internals\blocks.py", line 329, in apply
    result = func(self.values, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\groupby.py", line 1503, in array_func
    result = self._agg_py_fallback(values, ndim=data.ndim, alt=alt)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\groupby.py", line 1457, in _agg_py_fallback
    res_values = self.grouper.agg_series(ser, alt, preserve_dtype=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\ops.py", line 994, in agg_series
    result = self._aggregate_series_pure_python(obj, func)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\ops.py", line 1015, in _aggregate_series_pure_python
    res = func(group)
          ^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\groupby\groupby.py", line 1857, in <lambda>
    alt=lambda x: Series(x).mean(numeric_only=numeric_only),
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\generic.py", line 11556, in mean
    return NDFrame.mean(self, axis, skipna, numeric_only, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\generic.py", line 11201, in mean
    return self._stat_function(
           ^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\generic.py", line 11158, in _stat_function
    return self._reduce(
           ^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\series.py", line 4670, in _reduce
    return op(delegate, skipna=skipna, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\nanops.py", line 96, in _f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\nanops.py", line 158, in f
    result = alt(values, axis=axis, skipna=skipna, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\nanops.py", line 421, in new_func
    result = func(values, axis=axis, skipna=skipna, mask=mask, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\nanops.py", line 727, in nanmean
    the_sum = _ensure_numeric(values.sum(axis, dtype=dtype_sum))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\nanops.py", line 1699, in _ensure_numeric
    raise TypeError(f"Could not convert {x} to numeric") from err
TypeError: Could not convert {'r': 0.0, 'l': 20000, 't': 9204.664227} to numeric

Any idea as to why there is such a complex error ?

Thanks

P.S. The python file is as given in the library just a changed name. The output files I'm using are given in the zipped folder.
Outputs.zip

[LucasAlegre]

Are you using Windows? I fear this may be due to Windows saving CSV files differently than Linux.

[prajwalvinod]

Hey @LucasAlegre,

Yes I am running Windows 11. Is there a fix or a workaround to this ? And additionally is there anything wrong with the data in the saved files ?

Thanks.

[prajwalvinod]

Hey @LucasAlegre,

I solved the above error by changing the "-sep" argument to ";" instead of ",". But now I get a new error as follows:

python plots.py -f Outputs\dqn_test_10\dqn_test_10_conn0_ep4.csv
Traceback (most recent call last):
  File "D:\Anaconda3\Lib\site-packages\pandas\core\indexes\base.py", line 3653, in get_loc
    return self._engine.get_loc(casted_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas\_libs\index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'system_total_waiting_time'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\prajw\PythonFiles\Thesis\RL_for_Freeway\Hegyi_Model\plots.py", line 90, in <module>
    plot_df(main_df, xaxis=args.xaxis, yaxis=args.yaxis, label=next(labels), color=next(colors), ma=args.ma)
  File "C:\Users\prajw\PythonFiles\Thesis\RL_for_Freeway\Hegyi_Model\plots.py", line 43, in plot_df
    df[yaxis] = pd.to_numeric(df[yaxis], errors="coerce")  # convert NaN string to NaN value
                              ~~^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\frame.py", line 3761, in __getitem__
    indexer = self.columns.get_loc(key)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Anaconda3\Lib\site-packages\pandas\core\indexes\base.py", line 3655, in get_loc
    raise KeyError(key) from err
KeyError: 'system_total_waiting_time'

I checked some other issue posts but I could not get a proper solution for the above error.
How do I get around this error ?

Thanks.

[prajwalvinod]

Hey @LucasAlegre,

Any idea as to why the above is happening ?
Additionally, is there any provision/way to plot rewards ?

Regards

Prajwal

[TerzaSlayer2000]

Hey @prajwalvinod
Did you solve the above plotting error?
KeyError: 'system_total_waiting_time'