Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GMRT SETI is having evolving issues concerning getting started with rawspec #52

Closed
texadactyl opened this issue Jan 30, 2022 · 39 comments
Closed

GMRT SETI is having evolving issues concerning getting started with rawspec #52

texadactyl opened this issue Jan 30, 2022 · 39 comments
Assignees
@texadactyl

Comments

@texadactyl
Copy link
Contributor

texadactyl commented Jan 30, 2022 •
edited
Loading

Starting with rawspec v3.1.1, GMRT is seeing crashes. This issue will be updated as more and more information becomes available.

The current (2022-01-30) header information:

rawhdr: All fields of the first header .....
         BACKEND ='GUPPI   '
         ACC_LEN =                    1
         AZ      =             433.0963
         BANDNUM =                    0
         BANKNAM ='GWBH8   '
         BANKNUM =                    0
         BASE_BW =                200.0
         BINDHOST='eth4    '
         BLOCSIZE=            134217728
         BMAJ    = 0.009263915095687008
         BMIN    = 0.009263915095687008
         CAL_DCYC=                  0.5
         CAL_FREQ=                  0.0
         CAL_MODE='OFF     '
         CHAN_BW =          0.048828125
         CHAN_DM =                  0.0
         DAQCTRL ='start   '
         DAQPULSE='Tue Sep 22 00:24:27 2020'
         DAQSTATE='record  '
         DATADIR ='/data/dir'
         DATAHOST='blr2-1-10-0.gb.nrao.edu'
         DATAPORT=                60000
         DEC     =              25.6911
         DEC_STR ='+25:41:27.9600'
         DIRECTIO=                    0
         DISKSTAT='waiting '
         DROPAVG =                    0
         DROPBLK =                    0
         DROPTOT =                    0
         DS_FREQ =                    1
         DS_TIME =                    1
         FD_POLN ='LIN     '
         FFTLEN  =                 2048
         FRONTEND='ArrayN  '
         LST     =              83464.0
         NBIN    =                    0
         NBITS   =                    8
         NDROP   =                    0
         NETBUFST='1/24    '
         NETSTAT ='receiving'
         NPKT    =                16384
         NPOL    =                    1
         NRCVR   =                    1
         OBSBW   =                100.0
         OBSERVER='Raghav Girgaonkar'
         OBSFREQ =                600.0
         OBS_MODE='RAW     '
         OBSNCHAN=                 2048
         OFFSET0 =                  0.0
         OFFSET1 =                  0.0
         OFFSET2 =                  0.0
         OFFSET3 =                  0.0
         ONLY_I  =                    0
         OVERLAP =                    0
         PARFILE ='/opt/dibas/etc/config/example.par'
         PFB_OVER=                    0
         PKTFMT  ='1SFA    '
         PKTIDX  =                    0
         PKTSIZE =            134217728
         PKTSTOP =                    0
         POL_TYPE='AABBAA  '
         PROJID  ='GMRT_GWB_TEST'
         RA      =              70.4408
         RA_STR  ='04:41:45.7920'
         SCALE0  =                  1.0
         SCALE1  =                  1.0
         SCALE2  =                  1.0
         SCALE3  =                  1.0
         SCAN    =                   10
         SCANLEN =                 23.0
         SCANREM =                  0.0
         SRC_NAME='B0329+54'
         STT_IMJD=                59114
         STT_OFFS=                    0
         STT_SMJD=                15868
         STTVALID=                    1
         TBIN    =            2.048e-05
         TELESCOP='GMRT    '
         TFOLD   =                    0
         TRK_MODE='TRACK   '
         ZA      =              69.1473
         END
rawhdr: Critical rawspec fields .....
        OBSNCHAN = 2048
        NPOL = 1
        NBITS = 8
        BLOCSIZE = 134217728
        PKTIDX = 0
        DIRECTIO = 0
        BEAM_ID is not present but not required
        NBEAM is not present but not required
        NANTS is not present but not required
        TBIN = 2.048e-05
        OBSFREQ = 600.0
        OBSBW = 100.0
rawhdr: No errors found.

cc: @david-macmahon @Akshay-e @RaghavGirgaonkar
@texadactyl
Copy link
Contributor Author

Is this related to issue #10 ?

@texadactyl
Copy link
Contributor Author

Since DIRECTIO = 0, the first data block should start immediately after the 80-byte END record.
@Akshay-e @RaghavGirgaonkar : Is this the case?

@Akshay-Eranhalodi
Copy link

Yes, the data writing starts after the END keyword in the header. There are 82 lines, so 6560 bytes of header and then data.

@texadactyl
Copy link
Contributor Author

texadactyl commented Jan 30, 2022 •
edited
Loading

@Akshay-e reports the following.

rawspec is able to produce the .fil files, adding the --debug option gives:

writing output files in SIGPROC Filterbank format
working stem: /Data/raghav/PASV_Data/PASV_to_GUPPI/NEW_TEST_guppi
opening file: /Data/raghav/PASV_Data/PASV_to_GUPPI/NEW_TEST_guppi.0000.raw
rawspec-main: open_output_file_per_antenna_and_write_header - successful
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
dump_file_thread_func: write for nants=0
opening file: /Data/raghav/PASV_Data/PASV_to_GUPPI/NEW_TEST_guppi.0001.raw [No such file or directory]
dump_file_thread_func: write for nants=0
output product 0: 1088 spectra

While trying to create .h5 files:

rawspec 3.1.1+dirty using librawspec 3.1.1+dirty and cuFFT  unknown/old
writing output files in FBH5 format
working stem: /Data/raghav/PASV_Data/PASV_to_GUPPI/NEW_TEST_guppi
opening file: /Data/raghav/PASV_Data/PASV_to_GUPPI/NEW_TEST_guppi.0000.raw
2022-01-31_00:32:05 fbh5_set_str_attr: CLASS = FILTERBANK
2022-01-31_00:32:05 fbh5_set_str_attr: VERSION = 2.0
2022-01-31_00:32:05 fbh5_set_str_attr: VERSION_RAWSPEC = 3.1.1+dirty
2022-01-31_00:32:05 fbh5_set_str_attr: VERSION_LIBRAWSPEC = 3.1.1+dirty
2022-01-31_00:32:05 fbh5_set_str_attr: VERSION_CUFFT =  unknown/old
2022-01-31_00:32:05 fbh5_set_str_attr: VERSION_HDF5 = 1.12.0
2022-01-31_00:32:05 fbh5_set_str_attr: BITSHUFFLE = ENABLED
2022-01-31_00:32:05 fbh5_open: Chunk dimensions = (1, 1, 2097152)
2022-01-31_00:32:05 fbh5_set_dataset_int_attr: machine_id = 20
2022-01-31_00:32:05 fbh5_set_dataset_int_attr: telescope_id = 7
2022-01-31_00:32:05 fbh5_set_dataset_int_attr: data_type = 1
2022-01-31_00:32:05 fbh5_set_dataset_int_attr: nchans = 2097152
2022-01-31_00:32:05 fbh5_set_dataset_int_attr: nfpc = 1024
2022-01-31_00:32:05 fbh5_set_dataset_int_attr: nbeams = 1
2022-01-31_00:32:05 fbh5_set_dataset_int_attr: ibeam = -1
2022-01-31_00:32:05 fbh5_set_dataset_int_attr: nbits = 32
2022-01-31_00:32:05 fbh5_set_dataset_int_attr: nifs = 1
2022-01-31_00:32:05 fbh5_set_dataset_double_attr: src_raj = 4.696053
2022-01-31_00:32:05 fbh5_set_dataset_double_attr: src_dej = 25.691100
2022-01-31_00:32:05 fbh5_set_dataset_double_attr: az_start = 0.000000
2022-01-31_00:32:05 fbh5_set_dataset_double_attr: za_start = 0.000000
2022-01-31_00:32:05 fbh5_set_dataset_double_attr: fch1 = 550.000000
2022-01-31_00:32:05 fbh5_set_dataset_double_attr: foff = 0.000048
2022-01-31_00:32:05 fbh5_set_dataset_double_attr: tstart = 59114.183657
2022-01-31_00:32:05 fbh5_set_dataset_double_attr: tsamp = 0.020972
2022-01-31_00:32:05 fbh5_set_str_attr: source_name = B0329+54
2022-01-31_00:32:05 fbh5_set_str_attr: rawdatafile = NEW_TEST_guppi.0000.raw
2022-01-31_00:32:05 fbh5_open: Dataset metadata stored; done.
2022-01-31_00:32:05 fbh5_show_context(fbh5_open): active = 1
2022-01-31_00:32:05 fbh5_show_context(fbh5_open): elem_size = 4
2022-01-31_00:32:05 fbh5_show_context(fbh5_open): tint_size = 8388608
2022-01-31_00:32:05 fbh5_show_context(fbh5_open): offset_dims = (0, 0, 0)
2022-01-31_00:32:05 fbh5_show_context(fbh5_open): filesz_dims = (1, 1, 2097152)
2022-01-31_00:32:05 fbh5_show_context(fbh5_open): byte_count = 0
2022-01-31_00:32:05 fbh5_show_context(fbh5_open): dump_count = 0
open_output_file: fbh5_open(/Data/raghav/PASV_Data/PASV_to_GUPPI/NEW_TEST_guppi.rawspec.0000.h5) successful
rawspec-main: open_output_file_per_antenna_and_write_header - successful
dump_file_thread_func: write for nants=0
2022-01-31_00:32:05 fbh5_show_context(fbh5_write): active = 1
2022-01-31_00:32:05 fbh5_show_context(fbh5_write): elem_size = 4
2022-01-31_00:32:05 fbh5_show_context(fbh5_write): tint_size = 8388608
2022-01-31_00:32:05 fbh5_show_context(fbh5_write): offset_dims = (0, 0, 0)
2022-01-31_00:32:05 fbh5_show_context(fbh5_write): filesz_dims = (1, 1, 2097152)
2022-01-31_00:32:05 fbh5_show_context(fbh5_write): byte_count = 0
2022-01-31_00:32:05 fbh5_show_context(fbh5_write): dump_count = 0
2022-01-31_00:32:05 fbh5_write: dump 1, offset=(0, 0, 0), selection=(32, 1, 2097152), filesize=(32, 1, 2097152)
Segmentation fault (core dumped)

@texadactyl
Copy link
Contributor Author

@Akshay-e Please show the 2 command lines used

@Akshay-Eranhalodi
Copy link

rawspec -j -z -f 1024 -t 1 NEW_TEST_guppi # for h5 file
rawspec -z -f 1024 -t 1 NEW_TEST_guppi # for .fil file

@texadactyl
Copy link
Contributor Author

Odd: You used -f 1024 yet the FFTLEN in the header = 2048.

@david-macmahon
Copy link
Contributor

FFTLEN is a vestigal header keyword that rawspec ignores.

@david-macmahon
Copy link
Contributor

Is this single dish data??? For a telescope array like GMRT I would have expected a NANTS header with value greater than 1, but there is no NANTS header so it is treated as if NANTS=1.

@Akshay-Eranhalodi
Copy link

@david-macmahon This is only a trial observation data set that we are using recorded with a single dish for the pulsar B0329+54.

@Akshay-Eranhalodi
Copy link

Odd: You used -f 1024 yet the FFTLEN in the header = 2048.

Our guppi raw file has 2048 channels. So using -f 1024 while running rawspec means we are creating 1024 fine channels per 2048 coarse channel right ?

@david-macmahon
Copy link
Contributor

Does this file exist at the Berkeley data center?

@Akshay-Eranhalodi
Copy link

No, not right now. If it's needed I can try adding this to blpc1 tomorrow.

@david-macmahon
Copy link
Contributor

Even just the first header and data block would be useful (and easier/quicker to copy). If you know the header contains 6560 bytes and the data block contains 134217728 bytes you can make a raw file oneblock.0000.raw containing just the first header+data block using this command:

dd if= PASV_Data/PASV_to_GUPPI/NEW_TEST_guppi.0000.raw bs=$((6560+134217728)) count=1 of=oneblock.0000.raw

@texadactyl
Copy link
Contributor Author

@Akshay-e @RaghavGirgaonkar
Please run gdb on the rawspec failure case (h5) and post the resultant stack trace here:

gdb --args 'rawspec' '-j' '-z' '-f' '1024' '-t' '1' 'NEW_TEST_guppi'

@RaghavGirgaonkar
Copy link

RaghavGirgaonkar commented Jan 31, 2022 •
edited
Loading

This is the output after running the gdb command mentioned above:

(gdb) run
Starting program: /Data/eakshay/rawspec/rawspec -j -z -f 1024 -t 1 NEW_TEST_guppi
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
rawspec 3.1.1+dirty using librawspec 3.1.1+dirty and cuFFT  unknown/old
writing output files in FBH5 format
working stem: NEW_TEST_guppi
opening file: NEW_TEST_guppi.0000.raw
[New Thread 0x7fffe828c700 (LWP 31824)]
[New Thread 0x7fffe7a8b700 (LWP 31825)]
2022-01-31_10:56:47 fbh5_set_str_attr: CLASS = FILTERBANK
2022-01-31_10:56:47 fbh5_set_str_attr: VERSION = 2.0
2022-01-31_10:56:47 fbh5_set_str_attr: VERSION_RAWSPEC = 3.1.1+dirty
2022-01-31_10:56:47 fbh5_set_str_attr: VERSION_LIBRAWSPEC = 3.1.1+dirty
2022-01-31_10:56:47 fbh5_set_str_attr: VERSION_CUFFT =  unknown/old
2022-01-31_10:56:47 fbh5_set_str_attr: VERSION_HDF5 = 1.12.0
2022-01-31_10:56:47 fbh5_set_str_attr: BITSHUFFLE = ENABLED
2022-01-31_10:56:47 fbh5_open: Chunk dimensions = (1, 1, 2097152)
2022-01-31_10:56:47 fbh5_set_dataset_int_attr: machine_id = 20
2022-01-31_10:56:47 fbh5_set_dataset_int_attr: telescope_id = 7
2022-01-31_10:56:47 fbh5_set_dataset_int_attr: data_type = 1
2022-01-31_10:56:47 fbh5_set_dataset_int_attr: nchans = 2097152
2022-01-31_10:56:47 fbh5_set_dataset_int_attr: nfpc = 1024
2022-01-31_10:56:47 fbh5_set_dataset_int_attr: nbeams = 1
2022-01-31_10:56:47 fbh5_set_dataset_int_attr: ibeam = -1
2022-01-31_10:56:47 fbh5_set_dataset_int_attr: nbits = 32
2022-01-31_10:56:47 fbh5_set_dataset_int_attr: nifs = 1
2022-01-31_10:56:47 fbh5_set_dataset_double_attr: src_raj = 4.696053
2022-01-31_10:56:47 fbh5_set_dataset_double_attr: src_dej = 25.691100
2022-01-31_10:56:47 fbh5_set_dataset_double_attr: az_start = 0.000000
2022-01-31_10:56:47 fbh5_set_dataset_double_attr: za_start = 0.000000
2022-01-31_10:56:47 fbh5_set_dataset_double_attr: fch1 = 550.000000
2022-01-31_10:56:47 fbh5_set_dataset_double_attr: foff = 0.000048
2022-01-31_10:56:47 fbh5_set_dataset_double_attr: tstart = 59114.183657
2022-01-31_10:56:47 fbh5_set_dataset_double_attr: tsamp = 0.020972
2022-01-31_10:56:47 fbh5_set_str_attr: source_name = B0329+54
2022-01-31_10:56:47 fbh5_set_str_attr: rawdatafile = NEW_TEST_guppi.0000.raw
2022-01-31_10:56:47 fbh5_open: Dataset metadata stored; done.
2022-01-31_10:56:47 fbh5_show_context(fbh5_open): active = 1
2022-01-31_10:56:47 fbh5_show_context(fbh5_open): elem_size = 4
2022-01-31_10:56:47 fbh5_show_context(fbh5_open): tint_size = 8388608
2022-01-31_10:56:47 fbh5_show_context(fbh5_open): offset_dims = (0, 0, 0)
2022-01-31_10:56:47 fbh5_show_context(fbh5_open): filesz_dims = (1, 1, 2097152)
2022-01-31_10:56:47 fbh5_show_context(fbh5_open): byte_count = 0
2022-01-31_10:56:47 fbh5_show_context(fbh5_open): dump_count = 0
open_output_file: fbh5_open(NEW_TEST_guppi.rawspec.0000.h5) successful
rawspec-main: open_output_file_per_antenna_and_write_header - successful
[New Thread 0x7fffe728a700 (LWP 31829)]
[New Thread 0x7fffe6a89700 (LWP 31831)]
dump_file_thread_func: write for nants=0
2022-01-31_10:56:47 fbh5_show_context(fbh5_write): active = 1
2022-01-31_10:56:47 fbh5_show_context(fbh5_write): elem_size = 4
2022-01-31_10:56:47 fbh5_show_context(fbh5_write): tint_size = 8388608
2022-01-31_10:56:47 fbh5_show_context(fbh5_write): offset_dims = (0, 0, 0)
2022-01-31_10:56:47 fbh5_show_context(fbh5_write): filesz_dims = (1, 1, 2097152)
2022-01-31_10:56:47 fbh5_show_context(fbh5_write): byte_count = 0
2022-01-31_10:56:47 fbh5_show_context(fbh5_write): dump_count = 0
2022-01-31_10:56:47 fbh5_write: dump 1, offset=(0, 0, 0), selection=(32, 1, 2097152), filesize=(32, 1, 2097152)

Program received signal SIGBUS, Bus error.
[Switching to Thread 0x7fffe6a89700 (LWP 31831)]
0x00007fffe98c90f1 in _int_free () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-323.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 zlib-1.2.7-17.el7.x86_64

@RaghavGirgaonkar
Copy link

RaghavGirgaonkar commented Jan 31, 2022 •
edited
Loading

The above command creates an H5 file, but the command gdb --args 'rawspec' '-j' '-z' '-f' '1' '-t' '1' 'NEW_TEST_guppi' gives the following output:

(gdb) run
Starting program: /Data/eakshay/rawspec/rawspec -j -z -f 1 -t 1 NEW_TEST_guppi
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
rawspec 3.1.1+dirty using librawspec 3.1.1+dirty and cuFFT  unknown/old
writing output files in FBH5 format
working stem: NEW_TEST_guppi
opening file: NEW_TEST_guppi.0000.raw
[New Thread 0x7fffe828c700 (LWP 42052)]
[New Thread 0x7fffe7a8b700 (LWP 42053)]
2022-01-31_11:01:30 fbh5_set_str_attr: CLASS = FILTERBANK
2022-01-31_11:01:30 fbh5_set_str_attr: VERSION = 2.0
2022-01-31_11:01:30 fbh5_set_str_attr: VERSION_RAWSPEC = 3.1.1+dirty
2022-01-31_11:01:30 fbh5_set_str_attr: VERSION_LIBRAWSPEC = 3.1.1+dirty
2022-01-31_11:01:30 fbh5_set_str_attr: VERSION_CUFFT =  unknown/old
2022-01-31_11:01:30 fbh5_set_str_attr: VERSION_HDF5 = 1.12.0
2022-01-31_11:01:30 fbh5_set_str_attr: BITSHUFFLE = ENABLED
2022-01-31_11:01:30 fbh5_open: Chunk dimensions = (1, 1, 2048)
2022-01-31_11:01:30 fbh5_set_dataset_int_attr: machine_id = 20
2022-01-31_11:01:30 fbh5_set_dataset_int_attr: telescope_id = 7
2022-01-31_11:01:30 fbh5_set_dataset_int_attr: data_type = 1
2022-01-31_11:01:30 fbh5_set_dataset_int_attr: nchans = 2048
2022-01-31_11:01:30 fbh5_set_dataset_int_attr: nfpc = 1
2022-01-31_11:01:30 fbh5_set_dataset_int_attr: nbeams = 1
2022-01-31_11:01:30 fbh5_set_dataset_int_attr: ibeam = -1
2022-01-31_11:01:30 fbh5_set_dataset_int_attr: nbits = 32
2022-01-31_11:01:30 fbh5_set_dataset_int_attr: nifs = 1
2022-01-31_11:01:30 fbh5_set_dataset_double_attr: src_raj = 4.696053
2022-01-31_11:01:30 fbh5_set_dataset_double_attr: src_dej = 25.691100
2022-01-31_11:01:30 fbh5_set_dataset_double_attr: az_start = 0.000000
2022-01-31_11:01:30 fbh5_set_dataset_double_attr: za_start = 0.000000
2022-01-31_11:01:30 fbh5_set_dataset_double_attr: fch1 = 550.024414
2022-01-31_11:01:30 fbh5_set_dataset_double_attr: foff = 0.048828
2022-01-31_11:01:30 fbh5_set_dataset_double_attr: tstart = 59114.183657
2022-01-31_11:01:30 fbh5_set_dataset_double_attr: tsamp = 0.000020
2022-01-31_11:01:30 fbh5_set_str_attr: source_name = B0329+54
2022-01-31_11:01:30 fbh5_set_str_attr: rawdatafile = NEW_TEST_guppi.0000.raw
2022-01-31_11:01:30 fbh5_open: Dataset metadata stored; done.
2022-01-31_11:01:30 fbh5_show_context(fbh5_open): active = 1
2022-01-31_11:01:30 fbh5_show_context(fbh5_open): elem_size = 4
2022-01-31_11:01:30 fbh5_show_context(fbh5_open): tint_size = 8192
2022-01-31_11:01:30 fbh5_show_context(fbh5_open): offset_dims = (0, 0, 0)
2022-01-31_11:01:30 fbh5_show_context(fbh5_open): filesz_dims = (1, 1, 2048)
2022-01-31_11:01:30 fbh5_show_context(fbh5_open): byte_count = 0
2022-01-31_11:01:30 fbh5_show_context(fbh5_open): dump_count = 0
open_output_file: fbh5_open(NEW_TEST_guppi.rawspec.0000.h5) successful
rawspec-main: open_output_file_per_antenna_and_write_header - successful
[New Thread 0x7fffe728a700 (LWP 42058)]
[New Thread 0x7fffe6a89700 (LWP 42061)]
dump_file_thread_func: write for nants=0
2022-01-31_11:01:32 fbh5_show_context(fbh5_write): active = 1
2022-01-31_11:01:32 fbh5_show_context(fbh5_write): elem_size = 4
2022-01-31_11:01:32 fbh5_show_context(fbh5_write): tint_size = 8192
2022-01-31_11:01:32 fbh5_show_context(fbh5_write): offset_dims = (0, 0, 0)
2022-01-31_11:01:32 fbh5_show_context(fbh5_write): filesz_dims = (1, 1, 2048)
2022-01-31_11:01:32 fbh5_show_context(fbh5_write): byte_count = 0
2022-01-31_11:01:32 fbh5_show_context(fbh5_write): dump_count = 0
2022-01-31_11:01:32 fbh5_write: dump 1, offset=(0, 0, 0), selection=(32768, 1, 2048), filesize=(32768, 1, 2048)

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe6a89700 (LWP 42061)]
0x00007fffe98c8f6f in _int_free () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-323.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 zlib-1.2.7-17.el7.x86_64

@RaghavGirgaonkar
Copy link

RaghavGirgaonkar commented Jan 31, 2022 •
edited
Loading

@david-macmahon @texadactyl

I have uploaded two files in blpc1 one with padding in the header and one without. Each file has one block and one header. The paths for the files are:
/datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_withpadding.0000.raw
/datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_NOpadding.0000.raw

One should have file size = 6560+134217728 = 134224288 and the other should have 6560+96+134217728 = 134224384

@texadactyl
Copy link
Contributor Author

rawspec is reporting 0 spectra in both files.

@RaghavGirgaonkar
Copy link

@texadactyl What is the exact rawspec command you are using?

@texadactyl
Copy link
Contributor Author

WITHPAD='/datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_withpadding.0000.raw'
NOPAD='/datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_NOpadding.0000.raw'

rawspec -j -z -f 1024 -t 1 -d . $NOPAD
rawspec -j -z -f 1024 -t 1 -d . $WITHPAD

@texadactyl
Copy link
Contributor Author

I can see that both files are 129 M

@RaghavGirgaonkar
Copy link

If I use the same command but without the -j option I am getting the following:

rawspec -f 1024 -t 1 oneblock_withpadding
working stem: oneblock_withpadding
opening file: oneblock_withpadding.0000.raw
opening file: oneblock_withpadding.0001.raw [No such file or directory]
output product 0: 32 spectra

@RaghavGirgaonkar
Copy link

RaghavGirgaonkar commented Jan 31, 2022 •
edited
Loading

@texadactyl The file sizes may seem to be 129 M but on doing ls -ltr, these are the real sizes in bytes

-rwxrwxrwx 1 raghav root 134224384 Jan 31 00:20 oneblock_withpadding.0000.raw
-rwxrwxrwx 1 raghav root 134224288 Jan 31 00:21 oneblock_NOpadding.0000.raw

@texadactyl
Copy link
Contributor Author

rawspec 3.1.1+dirty using librawspec 3.1.1+dirty and cuFFT 10.2.1.245
writing output files in SIGPROC Filterbank format
working stem: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_withpadding.0000.raw
opening file: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_withpadding.0000.raw.0000.raw [No such file or directory]
output product 0: 0 spectra

===================== without pad
rawspec 3.1.1+dirty using librawspec 3.1.1+dirty and cuFFT 10.2.1.245
writing output files in SIGPROC Filterbank format
working stem: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_NOpadding.0000.raw
opening file: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_NOpadding.0000.raw.0000.raw [No such file or directory]
output product 0: 0 spectra

@RaghavGirgaonkar
Copy link

@texadactyl In your case, it's taking the filename as the stem I think

@texadactyl
Copy link
Contributor Author

My comment about file size was about both of them containing data, not accuracy

@texadactyl
Copy link
Contributor Author

texadactyl commented Jan 31, 2022 •
edited
Loading

I need my morning coffee!
Got 32 spectra in SIGPROC mode.

@RaghavGirgaonkar
Copy link

I think the command you should run is:

WITHPAD='/datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_withpadding
NOPAD='/datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_NOpadding

rawspec -j -z -f 1024 -t 1 -d . $NOPAD
rawspec -j -z -f 1024 -t 1 -d . $WITHPAD

@texadactyl
Copy link
Contributor Author

Ok, in fbh5 mode, I can process the first block without any errors

===================== with pad
rawspec 3.1.1+dirty using librawspec 3.1.1+dirty and cuFFT 10.2.1.245
writing output files in FBH5 format
working stem: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_withpadding
opening file: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_withpadding.0000.raw
2022-01-31_06:28:43 fbh5_set_str_attr: CLASS = FILTERBANK
2022-01-31_06:28:43 fbh5_set_str_attr: VERSION = 2.0
2022-01-31_06:28:43 fbh5_set_str_attr: VERSION_RAWSPEC = 3.1.1
2022-01-31_06:28:43 fbh5_set_str_attr: VERSION_LIBRAWSPEC = 3.1.1+dirty
2022-01-31_06:28:43 fbh5_set_str_attr: VERSION_CUFFT = 10.2.1.245
2022-01-31_06:28:43 fbh5_set_str_attr: VERSION_HDF5 = 1.8.16
2022-01-31_06:28:43 fbh5_set_str_attr: BITSHUFFLE = ENABLED
2022-01-31_06:28:43 fbh5_open: Chunk dimensions = (1, 1, 2097152)
2022-01-31_06:28:43 fbh5_set_dataset_int_attr: machine_id = 20
2022-01-31_06:28:43 fbh5_set_dataset_int_attr: telescope_id = 7
2022-01-31_06:28:43 fbh5_set_dataset_int_attr: data_type = 1
2022-01-31_06:28:43 fbh5_set_dataset_int_attr: nchans = 2097152
2022-01-31_06:28:43 fbh5_set_dataset_int_attr: nfpc = 1024
2022-01-31_06:28:43 fbh5_set_dataset_int_attr: nbeams = 1
2022-01-31_06:28:43 fbh5_set_dataset_int_attr: ibeam = -1
2022-01-31_06:28:43 fbh5_set_dataset_int_attr: nbits = 32
2022-01-31_06:28:43 fbh5_set_dataset_int_attr: nifs = 1
2022-01-31_06:28:43 fbh5_set_dataset_double_attr: src_raj = 4.696053
2022-01-31_06:28:43 fbh5_set_dataset_double_attr: src_dej = 25.691100
2022-01-31_06:28:43 fbh5_set_dataset_double_attr: az_start = 0.000000
2022-01-31_06:28:43 fbh5_set_dataset_double_attr: za_start = 0.000000
2022-01-31_06:28:43 fbh5_set_dataset_double_attr: fch1 = 550.000000
2022-01-31_06:28:43 fbh5_set_dataset_double_attr: foff = 0.000048
2022-01-31_06:28:43 fbh5_set_dataset_double_attr: tstart = 59114.183657
2022-01-31_06:28:43 fbh5_set_dataset_double_attr: tsamp = 0.020972
2022-01-31_06:28:43 fbh5_set_str_attr: source_name = B0329+54
2022-01-31_06:28:43 fbh5_set_str_attr: rawdatafile = oneblock_withpadding.0000.raw
2022-01-31_06:28:43 fbh5_open: Dataset metadata stored; done.
2022-01-31_06:28:43 fbh5_show_context(fbh5_open): active = 1
2022-01-31_06:28:43 fbh5_show_context(fbh5_open): elem_size = 4
2022-01-31_06:28:43 fbh5_show_context(fbh5_open): tint_size = 8388608
2022-01-31_06:28:43 fbh5_show_context(fbh5_open): offset_dims = (0, 0, 0)
2022-01-31_06:28:43 fbh5_show_context(fbh5_open): filesz_dims = (1, 1, 2097152)
2022-01-31_06:28:43 fbh5_show_context(fbh5_open): byte_count = 0
2022-01-31_06:28:43 fbh5_show_context(fbh5_open): dump_count = 0
open_output_file: fbh5_open(./oneblock_withpadding.rawspec.0000.h5) successful
rawspec-main: open_output_file_per_antenna_and_write_header - successful
opening file: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_withpadding.0001.raw [No such file or directory]
dump_file_thread_func: write for nants=0
2022-01-31_06:28:43 fbh5_show_context(fbh5_write): active = 1
2022-01-31_06:28:43 fbh5_show_context(fbh5_write): elem_size = 4
2022-01-31_06:28:43 fbh5_show_context(fbh5_write): tint_size = 8388608
2022-01-31_06:28:43 fbh5_show_context(fbh5_write): offset_dims = (0, 0, 0)
2022-01-31_06:28:43 fbh5_show_context(fbh5_write): filesz_dims = (1, 1, 2097152)
2022-01-31_06:28:43 fbh5_show_context(fbh5_write): byte_count = 0
2022-01-31_06:28:43 fbh5_show_context(fbh5_write): dump_count = 0
2022-01-31_06:28:43 fbh5_write: dump 1, offset=(0, 0, 0), selection=(32, 1, 2097152), filesize=(32, 1, 2097152)
2022-01-31_06:28:44 fbh5_write: dump 1 E.T. = 1.114 s
2022-01-31_06:28:44 fbh5_set_ds_label: label = time, dims_index = 0
2022-01-31_06:28:44 fbh5_set_ds_label: label = feed_id, dims_index = 1
2022-01-31_06:28:44 fbh5_set_ds_label: label = frequency, dims_index = 2
2022-01-31_06:28:46 fbh5_close: Context closed.
2022-01-31_06:28:46 fbh5_close: 1 dumps processed.
2022-01-31_06:28:46 fbh5_close: 32 time integrations processed.
2022-01-31_06:28:46 fbh5_close: Compressed 268.44 MiB --> 235.06 MiB
output product 0: 32 spectra

===================== without pad
rawspec 3.1.1+dirty using librawspec 3.1.1+dirty and cuFFT 10.2.1.245
writing output files in FBH5 format
working stem: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_NOpadding
opening file: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_NOpadding.0000.raw
2022-01-31_06:28:47 fbh5_set_str_attr: CLASS = FILTERBANK
2022-01-31_06:28:47 fbh5_set_str_attr: VERSION = 2.0
2022-01-31_06:28:47 fbh5_set_str_attr: VERSION_RAWSPEC = 3.1.1
2022-01-31_06:28:47 fbh5_set_str_attr: VERSION_LIBRAWSPEC = 3.1.1+dirty
2022-01-31_06:28:47 fbh5_set_str_attr: VERSION_CUFFT = 10.2.1.245
2022-01-31_06:28:47 fbh5_set_str_attr: VERSION_HDF5 = 1.8.16
2022-01-31_06:28:47 fbh5_set_str_attr: BITSHUFFLE = ENABLED
2022-01-31_06:28:47 fbh5_open: Chunk dimensions = (1, 1, 2097152)
2022-01-31_06:28:47 fbh5_set_dataset_int_attr: machine_id = 20
2022-01-31_06:28:47 fbh5_set_dataset_int_attr: telescope_id = 7
2022-01-31_06:28:47 fbh5_set_dataset_int_attr: data_type = 1
2022-01-31_06:28:47 fbh5_set_dataset_int_attr: nchans = 2097152
2022-01-31_06:28:47 fbh5_set_dataset_int_attr: nfpc = 1024
2022-01-31_06:28:47 fbh5_set_dataset_int_attr: nbeams = 1
2022-01-31_06:28:47 fbh5_set_dataset_int_attr: ibeam = -1
2022-01-31_06:28:47 fbh5_set_dataset_int_attr: nbits = 32
2022-01-31_06:28:47 fbh5_set_dataset_int_attr: nifs = 1
2022-01-31_06:28:47 fbh5_set_dataset_double_attr: src_raj = 4.696053
2022-01-31_06:28:47 fbh5_set_dataset_double_attr: src_dej = 25.691100
2022-01-31_06:28:47 fbh5_set_dataset_double_attr: az_start = 0.000000
2022-01-31_06:28:47 fbh5_set_dataset_double_attr: za_start = 0.000000
2022-01-31_06:28:47 fbh5_set_dataset_double_attr: fch1 = 550.000000
2022-01-31_06:28:47 fbh5_set_dataset_double_attr: foff = 0.000048
2022-01-31_06:28:47 fbh5_set_dataset_double_attr: tstart = 59114.183657
2022-01-31_06:28:47 fbh5_set_dataset_double_attr: tsamp = 0.020972
2022-01-31_06:28:47 fbh5_set_str_attr: source_name = B0329+54
2022-01-31_06:28:47 fbh5_set_str_attr: rawdatafile = oneblock_NOpadding.0000.raw
2022-01-31_06:28:47 fbh5_open: Dataset metadata stored; done.
2022-01-31_06:28:47 fbh5_show_context(fbh5_open): active = 1
2022-01-31_06:28:47 fbh5_show_context(fbh5_open): elem_size = 4
2022-01-31_06:28:47 fbh5_show_context(fbh5_open): tint_size = 8388608
2022-01-31_06:28:47 fbh5_show_context(fbh5_open): offset_dims = (0, 0, 0)
2022-01-31_06:28:47 fbh5_show_context(fbh5_open): filesz_dims = (1, 1, 2097152)
2022-01-31_06:28:47 fbh5_show_context(fbh5_open): byte_count = 0
2022-01-31_06:28:47 fbh5_show_context(fbh5_open): dump_count = 0
open_output_file: fbh5_open(./oneblock_NOpadding.rawspec.0000.h5) successful
rawspec-main: open_output_file_per_antenna_and_write_header - successful
opening file: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/oneblock_NOpadding.0001.raw [No such file or directory]
dump_file_thread_func: write for nants=0
2022-01-31_06:28:47 fbh5_show_context(fbh5_write): active = 1
2022-01-31_06:28:47 fbh5_show_context(fbh5_write): elem_size = 4
2022-01-31_06:28:47 fbh5_show_context(fbh5_write): tint_size = 8388608
2022-01-31_06:28:47 fbh5_show_context(fbh5_write): offset_dims = (0, 0, 0)
2022-01-31_06:28:47 fbh5_show_context(fbh5_write): filesz_dims = (1, 1, 2097152)
2022-01-31_06:28:47 fbh5_show_context(fbh5_write): byte_count = 0
2022-01-31_06:28:47 fbh5_show_context(fbh5_write): dump_count = 0
2022-01-31_06:28:47 fbh5_write: dump 1, offset=(0, 0, 0), selection=(32, 1, 2097152), filesize=(32, 1, 2097152)
2022-01-31_06:28:47 fbh5_write: dump 1 E.T. = 1.090 s
2022-01-31_06:28:47 fbh5_set_ds_label: label = time, dims_index = 0
2022-01-31_06:28:47 fbh5_set_ds_label: label = feed_id, dims_index = 1
2022-01-31_06:28:47 fbh5_set_ds_label: label = frequency, dims_index = 2
2022-01-31_06:28:50 fbh5_close: Context closed.
2022-01-31_06:28:50 fbh5_close: 1 dumps processed.
2022-01-31_06:28:50 fbh5_close: 32 time integrations processed.
2022-01-31_06:28:50 fbh5_close: Compressed 268.44 MiB --> 235.06 MiB
output product 0: 32 spectra

@texadactyl
Copy link
Contributor Author

texadactyl commented Jan 31, 2022 •
edited
Loading

So, is the issue in transition to block 2?

Answer: Not in the sample at

blpc1:  /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/TEST_padding_v2_guppi.0000.raw

@texadactyl
Copy link
Contributor Author

Ran turbo_seti on produced .h5 file.

turbo_seti version 2.0.18
blimpy version 2.0.35
h5py version 2.10.0

find_doppler    INFO     {'DIMENSION_LABELS': array([b'time', b'feed_id', b'frequency'], dtype=object), 'az_start': 0.0, 'data_type': 1, 'fch1': 550.0, 'foff': 4.76837158203125e-05, 'ibeam': -1, 'machine_id': 20, 'nbeams': 1, 'nbits': 32, 'nchans': 2097152, 'nfpc': 1024, 'nifs': 1, 'rawdatafile': 'oneblock_withpadding.0000.raw', 'source_name': 'B0329+54', 'src_dej': <Angle 25.6911 deg>, 'src_raj': <Angle 4.69605333 hourangle>, 'telescope_id': 7, 'tsamp': 0.02097152, 'tstart': 59114.183657407404, 'za_start': 0.0}
find_doppler    INFO     File: ./oneblock_withpadding.rawspec.0000.h5
 drift rates (min, max): (0.000000, 10.000000)
 SNR: 25.000000

Starting ET search using ./oneblock_withpadding.rawspec.0000.h5
find_doppler    INFO     Parameters: datafile=./oneblock_withpadding.rawspec.0000.h5, max_drift=10.0, min_drift=0.0, snr=25.0, out_dir=./, coarse_chans=, flagging=False, n_coarse_chan=None, kernels=None, gpu_backend=True, precision=2, append_output=False, log_level_int=20, obs_info={'pulsar': 0, 'pulsar_found': 0, 'pulsar_dm': 0.0, 'pulsar_snr': 0.0, 'pulsar_stats': array([0., 0., 0., 0., 0., 0.]), 'RFI_level': 0.0, 'Mean_SEFD': 0.0, 'psrflux_Sens': 0.0, 'SEFDs_val': [0.0], 'SEFDs_freq': [0.0], 'SEFDs_freq_up': [0.0]}
find_doppler.1  INFO     Top hit found! SNR 33.185186, Drift Rate 0.000000, index 62403
find_doppler.1  INFO     Top hit found! SNR 1280.141072, Drift Rate 0.000000, index 1048064
Search time:  0.24 min

@texadactyl
Copy link
Contributor Author

No pad case was the same:

turbo_seti version 2.0.18
blimpy version 2.0.35
h5py version 2.10.0

find_doppler    INFO     {'DIMENSION_LABELS': array([b'time', b'feed_id', b'frequency'], dtype=object), 'az_start': 0.0, 'data_type': 1, 'fch1': 550.0, 'foff': 4.76837158203125e-05, 'ibeam': -1, 'machine_id': 20, 'nbeams': 1, 'nbits': 32, 'nchans': 2097152, 'nfpc': 1024, 'nifs': 1, 'rawdatafile': 'oneblock_NOpadding.0000.raw', 'source_name': 'B0329+54', 'src_dej': <Angle 25.6911 deg>, 'src_raj': <Angle 4.69605333 hourangle>, 'telescope_id': 7, 'tsamp': 0.02097152, 'tstart': 59114.183657407404, 'za_start': 0.0}
find_doppler    INFO     File: ./oneblock_NOpadding.rawspec.0000.h5
 drift rates (min, max): (0.000000, 10.000000)
 SNR: 25.000000

Starting ET search using ./oneblock_NOpadding.rawspec.0000.h5
find_doppler    INFO     Parameters: datafile=./oneblock_NOpadding.rawspec.0000.h5, max_drift=10.0, min_drift=0.0, snr=25.0, out_dir=./, coarse_chans=, flagging=False, n_coarse_chan=None, kernels=None, gpu_backend=True, precision=2, append_output=False, log_level_int=20, obs_info={'pulsar': 0, 'pulsar_found': 0, 'pulsar_dm': 0.0, 'pulsar_snr': 0.0, 'pulsar_stats': array([0., 0., 0., 0., 0., 0.]), 'RFI_level': 0.0, 'Mean_SEFD': 0.0, 'psrflux_Sens': 0.0, 'SEFDs_val': [0.0], 'SEFDs_freq': [0.0], 'SEFDs_freq_up': [0.0]}
find_doppler.1  INFO     Top hit found! SNR 33.185186, Drift Rate 0.000000, index 62403
find_doppler.1  INFO     Top hit found! SNR 1280.141072, Drift Rate 0.000000, index 1048064
Search time:  0.16 min

@RaghavGirgaonkar
Copy link

@texadactyl Can you point me to the installation of rawspec you are using on blpc1 and how to source it? The one we have by default doesn't have the fbh5 option. I have a larger GUPPI Raw file that I have on blpc1 which has more than 3 blocks. I could point to you and also run it on that and check if it works.

@texadactyl
Copy link
Contributor Author

texadactyl commented Jan 31, 2022 •
edited
Loading

Install rawspec under your $HOME

You need to adjust PATH and LD_LIBRARY_PATH.
E.g. in my .profile:

source /opt/conda/init.sh # <-----------important to transition from Python 3.5.something to 3.7.6

export PATH="$HOME/rawspec:$HOME/bin:$HOME/.local/bin:$PATH"
export CUDA_PATH=/usr/local/cuda
export LD_LIBRARY_PATH=$HOME/rawspec 

export HDF5_USE_FILE_LOCKING='FALSE'```

@texadactyl
Copy link
Contributor Author

When you run turboSETI, if you see any stupid lock error messages, this will probably fix it:
export HDF5_USE_FILE_LOCKING='FALSE'

@RaghavGirgaonkar
Copy link

@texadactyl The larger file is at /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/TEST_padding_v2_guppi.0000.raw

This has probably around 34 blocks

@texadactyl
Copy link
Contributor Author

writing output files in FBH5 format
working stem: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/TEST_padding_v2_guppi
opening file: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/TEST_padding_v2_guppi.0000.raw
opening file: /datax/scratch/AMITY_INDIA/raghav/GSB_iFFT_pipeline/PASV_to_GUPPI/TEST_padding_v2_guppi.0001.raw [No such file or directory]
output product 0: 352 spectra

h5diag: Begin
h5diag: VERSION attribute: 2.0
h5diag: header: {'DIMENSION_LABELS': array([b'time', b'feed_id', b'frequency'], dtype=object), 'az_start': 0.0, 'data_type': 1, 'fch1': 550.0, 'foff': 1.1920928955078125e-05, 'ibeam': -1, 'machine_id': 20, 'nbeams': 1, 'nbits': 32, 'nchans': 2097152, 'nfpc': 1024, 'nifs': 1, 'rawdatafile': 'TEST_padding_v2_guppi.0000.raw', 'source_name': 'B0740-28', 'src_dej': <Angle 25.6911 deg>, 'src_raj': <Angle 4.69605333 hourangle>, 'telescope_id': 7, 'tsamp': 0.08388608, 'tstart': 59114.183657407404, 'za_start': 0.0}
h5diag: data shape: (352, 1, 2097152)
Rawspec version: 3.1.1
Librawspec version: 3.1.1+dirty
cuFFT version: 10.2.1.245
HDF5 library version: 1.8.16
Bitshuffle: ENABLED
h5diag: No errors detected

@texadactyl
Copy link
Contributor Author

Closing as previous failures do not seem to be a rawspec software problem.

@texadactyl texadactyl self-assigned this Feb 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@texadactyl texadactyl

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@david-macmahon @texadactyl @RaghavGirgaonkar @Akshay-Eranhalodi