Sections 1 and 2

.gitignore

@@ -1,3 +1,11 @@
+# our project-specific files
+data/out*
+.DS_Store
+*/.DS_Store
+
+
+## standard files...
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

1.Baby_steps.md

@@ -0,0 +1,118 @@
+## Take Baby Steps
+
+**Objectives**
+* Learn how to run your code from the terminal, both interactively & in batch
+* Learn the scheduler job info command 'bjobs' & its options
+
+In this section, we're going to assume
+It is important for you to be able to learn some basics of using the terminal, as one 
+can run batch jobs most effectively & flexibly from the terminal.
+
+### Run Interactively from the Terminal
+
+When starting out using a compute cluster, or often called high-performance computing (HPC)
+or high-throughput computing (HTC) systems, running applications interactively most
+effectively emulates how we work on our local laptop and desktop systems. Through GUI
+interfaces, we can 
+
+LSF, our scheduler, follows most resource manager conventions
+
+1. Open your remote desktop program (NoMachine) and connect to the HSBGrid compute cluster.
+
+2. After the session starts, open the Gnome terminal program (Applications > System Tools > Terminal).
+
+3. By default, most applications are not 'loaded' into our session (in the terminal). Since
+Spyder is bundled with Python, we'll 'load' in Spyder through the appropriate `module load`:
+
+```{bash}
+module load python/3.8.5
+```
+
+4. With Python, Spyder, and all the data science environment tools in our $PATH, we can
+now run Spyder. To run interactive sessions from a terminal, we can follow the posted 
+[Interactive Job instructions](https://www.hbs.edu/research-computing-services/resources/compute-cluster/running-jobs/running-a-program-submitting-a-job.aspx), 
+(Working with Custom Submission Scripts > Interactive) using the MATLAB example, to run Spyder:
+
+```{bash}
+bsub -q short_int -Is -W 60 -R "rusage[mem=4000]" -M 4000 -hl spyder
+```
+
+If successful, after a few moments Spyder should open. Much has happened here:
+
+This is all well and good, but we've only told Spyder to open, and not necessarily to open
+the code file. To do that, include the path to your code file by dragging the file icon
+into the terminal window after you've typed in the spyder job submission command **but not
+yet pressed Enter**. Your final result should look like:
+
+```{bash}
+bsub -q short_int -Is -W 60 -R "rusage[mem=4000]" -M 4000 -hl spyder /path/to/code/file.py
+```
+
+5. Although we've loaded the code file interactively, our goal is to run it. So let's make
+those adjustments. Instead of using Spyder to view and run our code, let's simply run
+it using the Python interpreter by substituting in the python command:
+
+```{bash}
+bsub -q short_int -Is -W 60 -R "rusage[mem=4000]" -M 4000 -hl python /path/to/code/file.py
+```
+
+If all goes well, you should see in your terminal window the code running, albeit slowly.
+That's fine. Let's press Ctrl-C to abort the code.
+
+### Run Your Python Code in Batch
+
+This is actually going to be much easier than it sounds, but there are a number of 
+'friction points' that will appear. Those will become apparent.
+
+1. In your terminal session, switch to running your code in batch by switching to a batch
+queue (`long` or `short`) and removing the 'interactive' flag (`Is`). The `short` queue is
+most appropriate, as this code will likely take minutes to run (~8.5 min on my 2020 iMac).
+
+```{bash}
+bsub -q short -W 60 -R "rusage[mem=4000]" -M 4000 -hl python /path/to/code/file.py
+```
+
+Since we're running now in batch, our scheduler will respond with our job ID, and 
+we cross our fingers that it is off an running. We have every reason to expect it to
+run OK, as it ran OK when doing it interactively.
+
+### Get the Job Info with the `bsub` Command
+
+Well, as you may have noticed, you're not quite sure what is going on. But we do have a
+few options:
+
+* Look at the directories for any file changes
+* Ask the scheduler if it's still running
+* Look at the job run details
+
+Since we know we're writing out the running sum every 10 lines, our output file should
+be present and growning, which we can see:
+
+```{bash}
+ls -al data/
+
+XXX
+```
+
+Yup, it's there. But we can't see what's going on. Let's park that for a moment.
+
+Our next option is asking the scheduler for our job status and info, which is easy to get:
+
+```{bash}
+bjobs
+
+XXX
+```
+
+We can get a detailed report on our job by using the `-l` (long) option:
+
+```{bash}
+bjobs -l JOBID
+
+XXX
+```
+
+Indeed, we can see our job is running, and it's pretty active, as the IDLE_FACTOR is
+greater than 0.5, indicating 50% efficiency. Aside, we can see our MAX MEM is XXX; 
+and so we can downgrade our RAM ask, in order to give RAM for others who might need it.
+