Connect to a PSRDada ringbuffer, optionally downsample and compress, and write out the data in FITS format.
This program is part of the data handling pipeline for the AA-ALERT project. See dadatrigger for an introduction and dataflow schema.
$ dadafits -k <hexadecimal key> -l <logfile> -t <template_directory> -d <output_directory> -S <synthesized beam table> -s <synthesize these beams>Command line arguments:
- -k Set the (hexadecimal) key to connect to the ringbuffer.
- -l Absolute path to a logfile (to be overwritten)
- -t Template directory (defaults to the directory templates in the current working directory)
- -d Output directory
- -S Synthesized beam table
- -s Selection of synthesized beams
These modes are for archiving data; the program can be run as part of the real-time pipeline.
The program implements different modes:
- mode 0: Stokes I + IAB (incoherent beams, so only one "tied-array" beam)
- mode 2: Stokes I + TAB (multiple tied-array beams)
In these modes data is also:
- integrated over time to reduce sample rate to 1250 samples per 1.024 seconds
- summed over frequencies to reduce total number of frequencies to 768
- compressed to 1 bit
For details see this section below.
The data input rate is set per science case. Supported cases:
- case 3: 12500 samples per second, 9 beams
- case 4: 12500 samples per second, 12 beams
These modes are for analysing event data, and are not optimized for real-time use.
The program implements different modes:
- mode 1: Stokes IQUV + IAB
- mode 3: Stokes IQUV + TAB
The data input rate is set per science case. Supported cases:
- case 3: 12500 samples per second, 9 beams
- case 4: 12500 samples per second, 12 beams
Metadata is read from the PSRDada header block, and copied to the FITS header. Note that much of the metadata available in the header block is ignored, due to code constraints and optimizations.
For values that should be present see the table below.
| header key | type | units | description | notes |
|---|---|---|---|---|
| MIN_FREQUENCY | double | Mhz | Center of lowest frequency band of observation | |
| BW | double | Mhz | Total bandwidth of observation | |
| PADDED_SIZE | int | bytes | Length of the fastest dimension of the data array | |
| SCIENCE_CASE | int | 1 | Mode of operation of ARTS, determines data rate | Must be 3 or 4 |
| SCIENCE_MODE | int | 1 | Mode of operation of ARTS, determines data layout | Either 0,1,2, or 3 |
| RA_HMS | string | HH:MM:SS.ssss | Right ascension | maps to RA |
| DEC_HMS | string | +HH:MM:SS.ssss | Declination | maps to DEC |
| SCANLEN | float | seconds | Requested observation length | maps to SCANLEN |
| FREQ | float | degrees | Center frequency | maps to OBSFREQ |
| SOURCE | string | text | Source name | maps to SRC_NAME |
| UTC_START | char | YYYY-MM-DDTHH:MM:SS | Human readable timestamp of the start of the observation. | The program will silently modify the separators to conform to FITS standard. However, whitespace characters as in '2018-04-18 14:40:10' will not work |
| MJD_START | double | days since epoch | Modified Julian Date | maps to STT_IMJD and STT_SMJD |
| LST_START | double | degrees | Local siderial time | maps to STT_LST |
| AZ_START | float | degrees | Azimuth angle of telescope | set per row in the SUBINT binary table as TEL_AZ, assumed constant over the run |
| ZA_START | float | degrees | Zenith angle of telescope | set per row in the SUBINT binary table as TEL_ZEN, assumed constant over the run |
| PARSET | string | Parameter file for the observation | maps to PARSET |
For modes 0 and 2 (ie Stokes I data), a ringbuffer page is interpreted as an array of Stokes I: [NTABS, NCHANNELS, padded_size] Array padding along the fastest dimension is implemented to facilitate memory copies.
For modes 1 and 3 (ie Stokes IQUV), a ringbuffer page is an interleaved array: [tab, channel_offset, sequence_number, packet] Where:
- tab ranges from 0 to 0 (IAB) or 9 (science case 3 TAB) or 11 (science case 4 TAB)
- channel_offset ranges from 0 upto 383 (NCHANNELS/4 - 1)
- sequence_number ranges from 0 upto 24
- packet is a direct copy of a UDP datapacket coming from the network, making up 8000 bytes
The packet itself is an array: [time, channel, polarization] where:
- time runs from 0 to 499, to get actual time, sequence_number * 500 should be added
- channel runs from 0 to 4, to get actual channel, channel_offset * 4 should be added
- polarization stands for the 4 Stokes components, IQUV.
Output files are created in the directory specified on the commandline. A template is used for the FITS file and is selected based on science case and mode. Templates are searched for in the template directory in the current working directory; or its location can be specified as a command line argument.
Data is stored one beam per file.
For TAB the filename is tabX.fits, where X indicates the TAB number. A=0, B=1, etc.
For synthesized beams the filename is synXX.fits, where XX is the synthesized beam number
To connect to the PSRDada ring buffer, we depend on PSRDada code. Ensure PSRDada is compiled with shared libraries enabled and libpsrdada.so can be found through LD_LIBRARY_PATH.
Building is done using CMake:
mkdir build && cd build
cmake ../
make && make installCompression to one bit is done for each batch of 1.024 seconds, and each frequency channel, independently. Downsampling is done by simple summation.
First, we calculate the average and standard deviation. Then, each sample is encoded as 0 (equal to, or below average) or 1 (above average).
The average and standard deviation are stored in the FITS file as offset and scale, where:
offset = avg - std
scale = 2.0 * std
Combined, the downsampling and compression achieve a reduction in data size of a factor ~140 compared to the filterbank output format (See also dadafilterbank). A factor 20 is achieved from the reduction in time and frequency resolution, another factor 7 by 1-bit compression.
The reduced resolution of the data imply that transient events can only be detected if they are very bright. However, for periodic signals many pulses can be summed and the pulse profile extracted. An example of a pulsar signal in 1-bit downsampled data is shown here:
The tied-array beams can be combined to form synthesized beams; providing more accurate localisation. A synthesized beam is a linear combination (simple sum) of tied-array beams.
The synthesized beam table lists per synthesized beam the constituent tied-array beams. The following rules apply:
- allow comments: everything following a '#' until the next newline is a comment, lines starting with '#' are ignored
- any and all white space is ignored and is only relevant for separating the numbers
- completely empty lines are ignored
The indexing scheme is as follows:
- tied-array beams must be numbered from 0 (central beam) to NTABS - 1
- a higher tied-array beam index implies a position right or westward of the center
- the middle synthesised beam index denotes the central position (e.g. 35 if there are 71 synthesized beams in total)
- a lower synthesised beam index implies a position right or westward of the center
- a higher synthesised beam index implies a position left or eastward of the center
An example synthesised beam table for 12 TABs and 71 SBs is included: sbtable-sc4-12tabs-71-sbs.txt
The observation parameterset is a string of key-value pairs. Keys and values are separated by =,
white space around the separator is ignored. Key-value pairs are separated by a newline character.
The parameterset is first compressed with bz2 compression, then encoded with hexencoding.
The encoded parset is then stored to the FITS header in the PARSET field.
For offline processing of IQUV data, the data are first read from disk into a PSRDada ringbuffer. dadafitsthen
deinterleaves these data and writes them to disk in FITS format.
Writing either 12 tied-array beams or one synthesised beam to disk takes roughly 13 seconds
per page of 1.024 seconds.
Jisk Attema, Netherlands eScience Center
Leon Oostrum, ASTRON / UvA
Gijs Molenaar, Pythonic.nl