Figure out what is causing intermittent periods of time when data is unable to be inserted

[ryantanaka]

First observed about about a month ago around 10:25-10:45am.

Observed again 2/12/18 around the same time period for about the same duration.

Scrape-util will be started via the crontab, however the program appears to be doing nothing. State timestamps are not updated so this is fine, the program recovers after about 10:45 ~ 10:50am and acts accordingly. Confirmed with @eileenpeppard that she sees data coming in.

Reason for this is currently unknown. Scrape-util output to stdout and stderr is tossed out because it's running as a cronjob. Errors do not appear in the appropriate errors folder so I'm currently unaware of why this happens but will investigate further. It could be a timeout happening at this time, but why?

[ryantanaka]

@ryantanaka note:
2/23/18 still seeing inconsistent upload times.

Looking at the times output files for uhm_frog.

-rw-r--r-- 1 root     staff  1.7M Feb 23 08:56 export-1519412204.csv   
-rw-r--r-- 1 root     staff  462K Feb 23 09:06 export-1519412786.csv    <-- 10 min diff
-rw-r--r-- 1 root     staff  471K Feb 23 09:16 export-1519413386.csv    <-- 10 min diff
-rw-r--r-- 1 root     staff  479K Feb 23 09:26 export-1519413983.csv    <-- 10 min diff
-rw-r--r-- 1 root     staff  484K Feb 23 09:31 export-1519414304.csv    <-- 5 min diff (ok)
-rw-r--r-- 1 root     staff  525K Feb 23 10:16 export-1519416993.csv    <-- ~45 min diff (why??)
-rw-r--r-- 1 root     staff  530K Feb 23 10:26 export-1519417582.csv    <-- 10 min diff

Looking at the syslog. Cron job gets started on schedule.

ryan@csbcd01:/var/log$ sudo cat syslog | grep /usr/local/bin/scrape-util | tail -n 40
Feb 23 07:40:01 csbcd01 CRON[1258]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 07:45:01 csbcd01 CRON[1500]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 07:50:01 csbcd01 CRON[1753]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 07:55:01 csbcd01 CRON[1995]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:00:01 csbcd01 CRON[2255]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:05:01 csbcd01 CRON[2505]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:10:01 csbcd01 CRON[2752]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:15:01 csbcd01 CRON[2996]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:20:01 csbcd01 CRON[3246]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:25:01 csbcd01 CRON[3486]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:30:01 csbcd01 CRON[3737]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:35:01 csbcd01 CRON[3978]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:40:01 csbcd01 CRON[4227]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:45:01 csbcd01 CRON[4470]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:50:01 csbcd01 CRON[4715]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 08:55:01 csbcd01 CRON[4955]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:00:01 csbcd01 CRON[5210]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:05:01 csbcd01 CRON[5450]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:10:01 csbcd01 CRON[5697]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:15:01 csbcd01 CRON[5937]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:20:01 csbcd01 CRON[6189]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:25:01 csbcd01 CRON[6429]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:30:01 csbcd01 CRON[6674]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:35:01 csbcd01 CRON[6917]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:40:01 csbcd01 CRON[7163]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:45:01 csbcd01 CRON[7403]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:50:01 csbcd01 CRON[7645]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 09:55:01 csbcd01 CRON[7885]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 10:00:01 csbcd01 CRON[8139]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 10:05:01 csbcd01 CRON[8380]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 10:10:01 csbcd01 CRON[8704]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 10:15:01 csbcd01 CRON[8948]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 10:20:01 csbcd01 CRON[9201]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 10:25:01 csbcd01 CRON[9443]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 10:30:01 csbcd01 CRON[9687]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 10:35:01 csbcd01 CRON[10002]: (root) CMD (   /usr/local/bin/scrape-util)
Feb 23 10:40:01 csbcd01 CRON[10421]: (root) CMD (   /usr/local/bin/scrape-util)