Forecasted Grant coming in as Posted status

[ClaireValdivia]

Why is this issue important?

Forecasted Grant coming in as posted

Current State

I'm seeing a grant (opp number: HHS-2024-ACF-ACYF-CY-0058) show up in Grant Finder with the status "Posted," but in grants.gov, it is Forecasted

another example - HHS-2024-ACF-ACYF-YO-0047
odd behavior with this one is that it doesn't seem like description data is coming through to grant finder

Expected State

status should reflect what is in grants.gov

Implementation Plan

The following functions need to be added...

Relevant Code Snippets

No response

[TylerHendrickson]

@ClaireValdivia @as1729 I investigated this one after the call this morning.

The issue appears to be a relic of the grants scraper, which was known to pull in grants with an invalid status. The scraper has been disabled on production for 2 weeks (since 9/29/23), per #1999, so it shouldn't be an issue going forward, but clearly there are a few records that could be cleaned up.

A few options for moving forward:

1. Presumably, adding support for forecasted grants in grants-ingest would resolve the issue by virtue of updating these records to the appropriate status.
2. We could simply delete these records from the database. The query for identifying records that were not sourced by grants-ingest (and therefore identifies grants that came in through the scraper) is pretty straightforward:

   => select count(*) from grants where revision_id is null;
    count 
   -------
      961
   (1 row)
   => select count(grant_id), opportunity_status from grants 
   -> where revision_id is null
   -> group by opportunity_status;
    count | opportunity_status 
   -------+--------------------
      676 | forecasted
      285 | posted
   (2 rows)

   So these scraper-only records could be cleaned up with a simple delete from grants where revision_id is null;

Note: the above query samples show that there are some 285 grants with a "posted" status in this data set. I didn't check every last one of them, but for the ones I did, I got the "There is no record found for your search." page on grants.gov when looking up the grants by grant_id.

Just for the sake of having a record, here are all of the grant_id values for grants that are listed as "posted" in our DB but are lacking a revision_id (meaning they came from the scraper):

337986, 335255, 0, 334981, 343193, 338763, 339593, 345196, 340193, 346243, 341560, 343838, 341299, 341525, 342573, 342749, 343444, 343873, 344227, 344228, 344599, 345158, 345686, 345698, 345701, 345936, 345198, 345193, 345687, 346869, 347427, 347507, 347341, 347570, 347320, 347940, 339981, 342468, 343565, 343302, 343840, 343489, 339092, 345148, 346297, 344974, 337973, 335003, 338254, 339591, 338954, 338171, 338174, 338106, 338129, 338579, 338210, 338494, 337188, 338309, 335024, 339017, 340153, 338575, 337189, 338625, 338445, 339007, 341204, 338612, 333333, 339412, 339556, 339555, 339165, 338634, 338949, 339424, 338857, 339045, 339001, 344189, 338791, 339407, 338851, 339002, 339062, 338847, 338923, 339087, 344027, 339303, 339089, 339090, 339091, 345013, 337438, 339287, 339881, 339458, 339658, 340236, 339503, 340176, 340617, 346129, 339764, 339791, 340189, 339599, 339972, 340196, 340496, 340976, 340514, 340321, 340410, 340649, 340337, 340557, 340645, 341223, 341012, 341524, 341559, 341561, 346385, 344400, 341107, 341619, 337905, 341123, 341169, 341396, 342729, 342104, 342643, 342654, 343307, 342632, 340529, 342813, 342918, 348736, 342748, 343192, 346388, 343554, 343922, 344804, 343423, 344031, 344217, 344100, 343959, 344222, 344682, 344855, 344093, 344135, 344915, 344912, 344490, 344551, 344552, 344554, 344813, 344719, 345831, 345042, 345082, 344819, 344821, 344860, 344831, 344779, 344780, 345029, 344717, 345048, 345056, 345054, 345036, 345165, 345192, 345194, 345197, 345190, 345189, 345202, 345201, 345199, 345204, 345207, 343854, 345216, 345217, 345375, 345245, 345934, 345911, 345544, 345700, 345704, 345628, 345406, 345520, 341838, 341292, 345989, 345195, 345303, 345999, 345456, 346484, 345684, 345681, 345689, 345691, 345683, 345733, 345603, 347063, 348857, 346664, 347028, 346758, 342909, 347110, 348105, 348116, 346831, 344792, 347140, 348893, 350171, 347155, 347148, 346886, 346939, 348821, 347204, 347206, 347202, 347372, 342986, 348901, 347361, 347393, 347383, 347637, 347571, 349197, 347759, 347938, 347774, 348666, 348651, 348259, 348033, 348032, 348574, 348650, 348707, 348239, 348610, 348611, 349336, 348652, 348671, 348427, 348741, 348495, 349004, 348747, 349337, 349041, 342961, 349172, 349099, 349171, 349269, 350028, 350131

My suggestion for next steps would be to move forward with Option 2.

[TylerHendrickson]

Completed.