Skip to content

Commit

Permalink
Add examples to the tutorial on how to pass a process pool for parall…
Browse files Browse the repository at this point in the history 
…elization
  • Loading branch information
@e-koch
e-koch committed Sep 30, 2024
1 parent a0f9d69 commit cfdb88f
Showing 1 changed file with 79 additions and 35 deletions.
114 changes: 79 additions & 35 deletions examples/tutorial.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
},
{
"cell_type": "code",
"execution_count": 120,
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -53,7 +53,7 @@
},
{
"cell_type": "code",
"execution_count": 121,
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -67,14 +67,14 @@
},
{
"cell_type": "code",
"execution_count": 122,
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/eric/Dropbox/code_development/filaments/build/lib.linux-x86_64-3.6/fil_finder/filfinder2D.py:138: UserWarning: No beam width given. Using 0 pixels.\n",
"/Users/ekoch/Library/CloudStorage/Dropbox/code_development/FilFinder/fil_finder/filfinder2D.py:150: UserWarning: No beam width given. Using 0 pixels.\n",
" warnings.warn(\"No beam width given. Using 0 pixels.\")\n"
]
}
Expand All @@ -96,7 +96,7 @@
},
{
"cell_type": "code",
"execution_count": 123,
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -106,25 +106,20 @@
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
"collapsed": true,
"jupyter": {
"outputs_hidden": true
}
},
"source": [
"If [spectral-cube](https://spectral-cube.readthedocs.io/en/latest/) is installed, the `Projection` or `Slice` classes can also be passed to `FilFinder2D`:"
]
},
{
"cell_type": "code",
"execution_count": 124,
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: A 'NAXIS1' keyword already exists in this header. Inserting duplicate keyword. [astropy.io.fits.header]\n"
]
}
],
"outputs": [],
"source": [
"from spectral_cube import Projection\n",
"\n",
Expand All @@ -142,12 +137,14 @@
"\n",
"Note that numerical inputs must be given as `~astropy.units.Quantity` object with the appropriate unit.\n",
"\n",
"**Distance** -- To facilitate conversions to physical units, a distance can be given to `FilFinder2D`:"
"### Distance \n",
"\n",
"To facilitate conversions to physical units, a distance can be given to `FilFinder2D`:"
]
},
{
"cell_type": "code",
"execution_count": 125,
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -158,25 +155,27 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"**Angular Scale** -- If no header information is given, the pixel-to-angular conversion can be given:"
"### Angular Scale\n",
"\n",
"If no header information is given, the pixel-to-angular conversion can be given:"
]
},
{
"cell_type": "code",
"execution_count": 126,
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/eric/Dropbox/code_development/filaments/build/lib.linux-x86_64-3.6/fil_finder/base_conversions.py:55: UserWarning: Cannot find 'BMAJ' in the header. Try installing the `radio_beam` package for loading header information.\n",
"/Users/ekoch/Library/CloudStorage/Dropbox/code_development/FilFinder/fil_finder/base_conversions.py:55: UserWarning: Cannot find 'BMAJ' in the header. Try installing the `radio_beam` package for loading header information.\n",
" warn(\"Cannot find 'BMAJ' in the header. Try installing\"\n",
"/home/eric/Dropbox/code_development/filaments/build/lib.linux-x86_64-3.6/fil_finder/base_conversions.py:63: UserWarning: Cannot find 'BMIN' in the header. Assuming circular beam.\n",
"/Users/ekoch/Library/CloudStorage/Dropbox/code_development/FilFinder/fil_finder/base_conversions.py:63: UserWarning: Cannot find 'BMIN' in the header. Assuming circular beam.\n",
" warn(\"Cannot find 'BMIN' in the header. Assuming circular beam.\")\n",
"/home/eric/Dropbox/code_development/filaments/build/lib.linux-x86_64-3.6/fil_finder/base_conversions.py:69: UserWarning: Cannot find 'BPA' in the header. Assuming PA of 0.\n",
"/Users/ekoch/Library/CloudStorage/Dropbox/code_development/FilFinder/fil_finder/base_conversions.py:69: UserWarning: Cannot find 'BPA' in the header. Assuming PA of 0.\n",
" warn(\"Cannot find 'BPA' in the header. Assuming PA of 0.\")\n",
"/home/eric/Dropbox/code_development/filaments/build/lib.linux-x86_64-3.6/fil_finder/filfinder2D.py:138: UserWarning: No beam width given. Using 0 pixels.\n",
"/Users/ekoch/Library/CloudStorage/Dropbox/code_development/FilFinder/fil_finder/filfinder2D.py:150: UserWarning: No beam width given. Using 0 pixels.\n",
" warnings.warn(\"No beam width given. Using 0 pixels.\")\n"
]
}
Expand All @@ -189,12 +188,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"**Beamwidth** -- If the major axis of the beam is contained in the header, it will be automatically read in. If that information is not in the header, the beam size can be passed separately:"
"### Beam width \n",
"\n",
"If the major axis of the beam is contained in the header, it will be automatically read in. If that information is not in the header, the beam size can be passed separately:"
]
},
{
"cell_type": "code",
"execution_count": 127,
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -205,12 +206,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"**Custom Filament Masks** -- If you have a pre-computed filament mask, the mask array can be passed:"
"### Custom Filament Masks\n",
"\n",
"If you have a pre-computed filament mask, the mask array can be passed:"
]
},
{
"cell_type": "code",
"execution_count": 128,
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -226,12 +229,14 @@
"source": [
"The custom mask must have the same shape as the inputed image.\n",
"\n",
"**Save Name** -- A prefix for saved plots and table can be given:"
"### Save Name\n",
"\n",
"A prefix for saved plots and table can be given:"
]
},
{
"cell_type": "code",
"execution_count": 129,
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -247,7 +252,7 @@
},
{
"cell_type": "code",
"execution_count": 130,
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -261,6 +266,42 @@
"The beamwidth is $24''$ and is defined in the header."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Parallel Processing options\n",
"\n",
"`FilFinder2D` now allows operations over invidividual filaments to be parallelized with the `pool` and `nthreads` kwargs.\n",
"\n",
"The default is `None` which will trigger creating a `oncurrent.futures.ProcessPoolExecutor` with `nthread` processes (default is 1).\n",
"\n",
"Alternatively, you can use [`mpi4py.futures.MPIPoolExecutor`](https://mpi4py.readthedocs.io/en/stable/mpi4py.futures.html#mpi4py.futures.MPIPoolExecutor) as the `pool` when working in an MPI environment. By default the `max_workers` will be set by the [`MPI4PY_FUTURES_MAX_WORKERS`](https://mpi4py.readthedocs.io/en/stable/mpi4py.futures.html#envvar-MPI4PY_FUTURES_MAX_WORKERS) environment variable.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"from concurrent.futures import ProcessPoolExecutor\n",
"\n",
"mypool = ProcessPoolExecutor(max_workers=2)\n",
"fil = FilFinder2D(hdu, save_name=\"FilFinder_Output\", pool=mypool)\n",
"\n",
"# is equivalent to\n",
"\n",
"fil = FilFinder2D(hdu, save_name=\"FilFinder_Output\", pool=None, nthreads=2)\n",
"\n",
"# Using mpi4py:\n",
"# from mpi4py.futures import MPIPoolExecutor\n",
"\n",
"# mypool = MPIPoolExecutor(max_workers=None)\n",
"# fil = FilFinder2D(hdu, save_name=\"FilFinder_Output\", pool=mypool)\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand All @@ -272,7 +313,10 @@
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
"collapsed": true,
"jupyter": {
"outputs_hidden": true
}
},
"source": [
"Prior to creating the mask, it can be helpful to first *flatten* the image of bright compact sources. `FilFinder2D` uses an arctan transform, where the data are first normalized by some percentile value of the data:"
Expand Down Expand Up @@ -2169,7 +2213,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
Expand All @@ -2183,9 +2227,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
"nbformat_minor": 4
}

0 comments on commit cfdb88f

Please sign in to comment.