From cfdb88fd6b11ca12b681eca18e84e009faf0171c Mon Sep 17 00:00:00 2001 From: Eric Koch Date: Mon, 30 Sep 2024 11:38:41 -0400 Subject: [PATCH] Add examples to the tutorial on how to pass a process pool for parallelization --- examples/tutorial.ipynb | 114 ++++++++++++++++++++++++++++------------ 1 file changed, 79 insertions(+), 35 deletions(-) diff --git a/examples/tutorial.ipynb b/examples/tutorial.ipynb index 65f6e423..47309dfb 100644 --- a/examples/tutorial.ipynb +++ b/examples/tutorial.ipynb @@ -19,7 +19,7 @@ }, { "cell_type": "code", - "execution_count": 120, + "execution_count": 1, "metadata": {}, "outputs": [], "source": [ @@ -53,7 +53,7 @@ }, { "cell_type": "code", - "execution_count": 121, + "execution_count": 2, "metadata": {}, "outputs": [], "source": [ @@ -67,14 +67,14 @@ }, { "cell_type": "code", - "execution_count": 122, + "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "/home/eric/Dropbox/code_development/filaments/build/lib.linux-x86_64-3.6/fil_finder/filfinder2D.py:138: UserWarning: No beam width given. Using 0 pixels.\n", + "/Users/ekoch/Library/CloudStorage/Dropbox/code_development/FilFinder/fil_finder/filfinder2D.py:150: UserWarning: No beam width given. Using 0 pixels.\n", " warnings.warn(\"No beam width given. Using 0 pixels.\")\n" ] } @@ -96,7 +96,7 @@ }, { "cell_type": "code", - "execution_count": 123, + "execution_count": 4, "metadata": {}, "outputs": [], "source": [ @@ -106,7 +106,10 @@ { "cell_type": "markdown", "metadata": { - "collapsed": true + "collapsed": true, + "jupyter": { + "outputs_hidden": true + } }, "source": [ "If [spectral-cube](https://spectral-cube.readthedocs.io/en/latest/) is installed, the `Projection` or `Slice` classes can also be passed to `FilFinder2D`:" @@ -114,17 +117,9 @@ }, { "cell_type": "code", - "execution_count": 124, + "execution_count": 5, "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING: A 'NAXIS1' keyword already exists in this header. Inserting duplicate keyword. [astropy.io.fits.header]\n" - ] - } - ], + "outputs": [], "source": [ "from spectral_cube import Projection\n", "\n", @@ -142,12 +137,14 @@ "\n", "Note that numerical inputs must be given as `~astropy.units.Quantity` object with the appropriate unit.\n", "\n", - "**Distance** -- To facilitate conversions to physical units, a distance can be given to `FilFinder2D`:" + "### Distance \n", + "\n", + "To facilitate conversions to physical units, a distance can be given to `FilFinder2D`:" ] }, { "cell_type": "code", - "execution_count": 125, + "execution_count": 6, "metadata": {}, "outputs": [], "source": [ @@ -158,25 +155,27 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Angular Scale** -- If no header information is given, the pixel-to-angular conversion can be given:" + "### Angular Scale\n", + "\n", + "If no header information is given, the pixel-to-angular conversion can be given:" ] }, { "cell_type": "code", - "execution_count": 126, + "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "/home/eric/Dropbox/code_development/filaments/build/lib.linux-x86_64-3.6/fil_finder/base_conversions.py:55: UserWarning: Cannot find 'BMAJ' in the header. Try installing the `radio_beam` package for loading header information.\n", + "/Users/ekoch/Library/CloudStorage/Dropbox/code_development/FilFinder/fil_finder/base_conversions.py:55: UserWarning: Cannot find 'BMAJ' in the header. Try installing the `radio_beam` package for loading header information.\n", " warn(\"Cannot find 'BMAJ' in the header. Try installing\"\n", - "/home/eric/Dropbox/code_development/filaments/build/lib.linux-x86_64-3.6/fil_finder/base_conversions.py:63: UserWarning: Cannot find 'BMIN' in the header. Assuming circular beam.\n", + "/Users/ekoch/Library/CloudStorage/Dropbox/code_development/FilFinder/fil_finder/base_conversions.py:63: UserWarning: Cannot find 'BMIN' in the header. Assuming circular beam.\n", " warn(\"Cannot find 'BMIN' in the header. Assuming circular beam.\")\n", - "/home/eric/Dropbox/code_development/filaments/build/lib.linux-x86_64-3.6/fil_finder/base_conversions.py:69: UserWarning: Cannot find 'BPA' in the header. Assuming PA of 0.\n", + "/Users/ekoch/Library/CloudStorage/Dropbox/code_development/FilFinder/fil_finder/base_conversions.py:69: UserWarning: Cannot find 'BPA' in the header. Assuming PA of 0.\n", " warn(\"Cannot find 'BPA' in the header. Assuming PA of 0.\")\n", - "/home/eric/Dropbox/code_development/filaments/build/lib.linux-x86_64-3.6/fil_finder/filfinder2D.py:138: UserWarning: No beam width given. Using 0 pixels.\n", + "/Users/ekoch/Library/CloudStorage/Dropbox/code_development/FilFinder/fil_finder/filfinder2D.py:150: UserWarning: No beam width given. Using 0 pixels.\n", " warnings.warn(\"No beam width given. Using 0 pixels.\")\n" ] } @@ -189,12 +188,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Beamwidth** -- If the major axis of the beam is contained in the header, it will be automatically read in. If that information is not in the header, the beam size can be passed separately:" + "### Beam width \n", + "\n", + "If the major axis of the beam is contained in the header, it will be automatically read in. If that information is not in the header, the beam size can be passed separately:" ] }, { "cell_type": "code", - "execution_count": 127, + "execution_count": 8, "metadata": {}, "outputs": [], "source": [ @@ -205,12 +206,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Custom Filament Masks** -- If you have a pre-computed filament mask, the mask array can be passed:" + "### Custom Filament Masks\n", + "\n", + "If you have a pre-computed filament mask, the mask array can be passed:" ] }, { "cell_type": "code", - "execution_count": 128, + "execution_count": 9, "metadata": {}, "outputs": [], "source": [ @@ -226,12 +229,14 @@ "source": [ "The custom mask must have the same shape as the inputed image.\n", "\n", - "**Save Name** -- A prefix for saved plots and table can be given:" + "### Save Name\n", + "\n", + "A prefix for saved plots and table can be given:" ] }, { "cell_type": "code", - "execution_count": 129, + "execution_count": 10, "metadata": {}, "outputs": [], "source": [ @@ -247,7 +252,7 @@ }, { "cell_type": "code", - "execution_count": 130, + "execution_count": 11, "metadata": {}, "outputs": [], "source": [ @@ -261,6 +266,42 @@ "The beamwidth is $24''$ and is defined in the header." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Parallel Processing options\n", + "\n", + "`FilFinder2D` now allows operations over invidividual filaments to be parallelized with the `pool` and `nthreads` kwargs.\n", + "\n", + "The default is `None` which will trigger creating a `oncurrent.futures.ProcessPoolExecutor` with `nthread` processes (default is 1).\n", + "\n", + "Alternatively, you can use [`mpi4py.futures.MPIPoolExecutor`](https://mpi4py.readthedocs.io/en/stable/mpi4py.futures.html#mpi4py.futures.MPIPoolExecutor) as the `pool` when working in an MPI environment. By default the `max_workers` will be set by the [`MPI4PY_FUTURES_MAX_WORKERS`](https://mpi4py.readthedocs.io/en/stable/mpi4py.futures.html#envvar-MPI4PY_FUTURES_MAX_WORKERS) environment variable.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [], + "source": [ + "from concurrent.futures import ProcessPoolExecutor\n", + "\n", + "mypool = ProcessPoolExecutor(max_workers=2)\n", + "fil = FilFinder2D(hdu, save_name=\"FilFinder_Output\", pool=mypool)\n", + "\n", + "# is equivalent to\n", + "\n", + "fil = FilFinder2D(hdu, save_name=\"FilFinder_Output\", pool=None, nthreads=2)\n", + "\n", + "# Using mpi4py:\n", + "# from mpi4py.futures import MPIPoolExecutor\n", + "\n", + "# mypool = MPIPoolExecutor(max_workers=None)\n", + "# fil = FilFinder2D(hdu, save_name=\"FilFinder_Output\", pool=mypool)\n", + "\n" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -272,7 +313,10 @@ { "cell_type": "markdown", "metadata": { - "collapsed": true + "collapsed": true, + "jupyter": { + "outputs_hidden": true + } }, "source": [ "Prior to creating the mask, it can be helpful to first *flatten* the image of bright compact sources. `FilFinder2D` uses an arctan transform, where the data are first normalized by some percentile value of the data:" @@ -2169,7 +2213,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -2183,9 +2227,9 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.4" + "version": "3.11.9" } }, "nbformat": 4, - "nbformat_minor": 2 + "nbformat_minor": 4 }