Logging

[hannbus]

Regarding the logging there are some things to discuss. Most important is the question whether or not there should be a log file and what should go in there. @JonasSchaub , you suggested to tie the logger to the cider class. So I instantiated a logger called CIDER and now there is 'CIDER' in all the logging messages instead of 'root' but I am not sure if that is what you meant.

[JonasSchaub]

That is exactly what I meant, yes! Although it is not very important here, in a multi-class application, it always has to be indicated, in which class the logged message originated.

And to the log file: We anyway always have an output folder when we run CIDER, right? So it would be no problem to place a log file there as well. Would it be much hassle to log in a log file as well as in the console (the Jupyter notebook)? In Java, one log message can go to both automatically. If you otherwise have to basically copy the code that produces the log message for both destinations, I wouldn't like it.

[hannbus]

Thanks for the reply. The logging works now in the notebook and in the log file.

[JonasSchaub]

Great! Are exceptions and errors also logged in the log file now automatically?

[JonasSchaub]

Not raising the error is definitely not what we try to do here, correct.
With the instantiation I am basically thinking of something like this:

keyErrorObj = KeyError("No SDFiles found in the given directory %s!" % (data_dir))
logger.error(str(keyErrorObj), exc_info=True) 
raise keyErrorObj

I quickly tested an analogous simpler example and it seems to work this way.

[hannbus]

In principle it works like you wrote, but the logger does not get the traceback for the log file or the notebook output. Instead it always writes 'NoneType: None'. This should be because the logger gets the traceback from the raised error with 'exc_info=True' and now the error is only raised after the logger writes to the log file (or rather in the notebook output).

[JonasSchaub]

Ok, then maybe the best solution here is to simply raise the error but define an uncaught exception handler once in the file (if it is defined in CIDER.py it will also work in the notebook, I hope) that writes any uncaught error to the log file. How to do that is described in the top answer here: https://stackoverflow.com/questions/6234405/logging-uncaught-exceptions-in-python/16993115#16993115
Hope that helps and works!

[hannbus]

I don't really get how you mean this. Should I raise an exception as well as an error and this exeption is then caught and logged by the handler?

[JonasSchaub]

Are exception and error two different things in Python? I used them synonymously here, sorry.
If you set an uncaught exception/error handler, every exception/error should be handled (i.e. written to a log file) by it, I hope.

[hannbus]

I am sorry, but I still don't get how I should do this. Maybe I just don't understand the concept of errors, exceptions and their handeling not well enought.
As far as I understand is:
If I use 'raise ValueError/KeyError/...' the execution of the cell will be immediately stopped causing error, afterwards there is no way to do the logging with the traceback included.
For raising a exception I will need to use a try-except block and an uncaught exception is an exception that has not been handled by within the try-except block and therefore causes the execution of the cell to fail with an error. (So to delibrately generate a uncaught exception might not be best practise, would it?) If I generate an uncaught exception and let it be handled by a function (the exception handler), I don't think I can get the different error messages I wrote for the 'raise Error' statements but I would get one message to log and the cell would execute without any error.
Maybe you can tell me where and what I got wrong and how you thought about it. Hopefully I can understand it then and make the logging work in a nice way. Sorry again for the cumbersome process of this issue.

[JonasSchaub]

No worries, Hannah, you are understanding this properly. You are just struggling with the concept of an "uncaught exception handler" and that is relatable. Plus, I know all these things in Java but have no detailed idea of how they translate into Python and that probably adds to the overall confusion.

Short answer is (based on my experience in Java): When an error/exception is thrown, the program does not immediately stop its execution. For example, it first retraces the stacktrace to see whether there is a try-except block on some level around the code that threw the error. If there is not, mechanisms like the uncaught exception handler get activated that can e.g. log the exception to a file before it finally gets thrown and the program shuts down. When an exception is thrown, the program basically starts a controlled shut-down. The situation is very different when e.g. the program runs out of memory (out of memory error, one of the worst errors you can produce). Then, the shutdown is not as controlled and e.g. the uncaught exception handler might not be called. This would be a "proper crash".
What I would like to achieve here (if possible) is that in situations where you deliberately throw/raise an error to end the program because something (the user did) was wrong, the program shuts down (yes) but the reason for the shutdown (the exception/error) is still logged to the log file to "complete the record", so to speak. And this can be achieved the way you did it (raising the error, logging it, raising it again) but it would be more elegant to configure the uncaught exception handler to log any uncaught exceptions to your log file and in the methods to simply raise the exceptions (shutting down the program but also relying on the uncaught exception handler to automatically log it before this). If this is even possible in Python.

Ok, the "short" answer got away from me there, sorry. I would be happy to have a short Zoom call to talk about this further, if you would like it. I am relatively free today and tomorrow (got only two meetings today at 11 am and at 2 pm and tomorrow at 9 am).

[hannbus]

I guess a short meeting would be quite helpful. I have time today and tomorrow as well. Maybe you can choose the time and send me the link per mail.

[JonasSchaub]

I just did that 👍