How should we handle deleted child elections in the API

[GeoWill]

Issue

Deleted elections appear in the list of all elections for an organisation, but not in the children of the group elections.
Take http://elections.democracyclub.org.uk/api/organisations/local-authority/ASH/1974-04-01/elections/ as an example.

The children of local.ashfield.2023-05-04 only include 'public' elections, i.e. with latest ModerationStatus set to Approved.

However there are four elections:

• local.ashfield.central-new-cross.2023-05-04
• local.ashfield.jacksdale.2023-05-04
• local.ashfield.st-marys.2023-05-04
• local.ashfield.summit.2023-05-04

That are listed with local.ashfield.2023-05-04 as their group_id. But they also have "deleted": True (because most recent ModerationStatus == Deleted).

I bring it up because the same endpoint but for http://127.0.0.1:8000/api/organisations/parl/parl-hoc/1832-06-07/elections/?format=json is very slow, and I'm trying to speed it up without breaking anything.

It sort of feels to me like this endpoint should do one of the following:

• show all elections, and all children regardless of moderation status.
• only show elections that are Approved
• have a parameter to show intention.

For reference, at the moment I don't think deleted elections appear as children of undeleted election groups, because the deleted flag is either/or rather than 'include deleted'. To put it another way /api/elections/local.ashfield.2023-05-04/ shows the election with a list of undeleted children, but /api/elections/local.ashfield.2023-05-04/?deleted=1 is not found. On the other hand /api/elections/local.city-of-london.2021-05-06/?deleted=1 shows a list of deleted children, because the group and all the children have current_status=Deleted.

Hopefully this all makes some sense, but maybe it doesn't matter?
I don't know how much this endpoint gets used, but someone tried to use it for parl/parl-hoc/1832-06-07 and it fell over which was why I started poking. I think I err to either show everything, or show only where current_status=Approved. This is the main question I'd like to resolve, but I think maybe it spills out wider, hence making it a Discussion.

Refs

Draft PR which cuts down queries and replicates current behaviour
Original PR
Original Issue that lead to PR

[chris48s]

I don't think I have really strong views on where you go next, but I can somewhat explain my original rationale for why this is the way it is:

If you think about /api/elections as a public API endpoint that might be consumed by things other than DC's own applications to "give me a list of elections", including deleted (or suggested, or rejected) elections by default mostly doesn't make sense. It is likely to lead to elections that aren't real elections popping up in user-facing scenarios, and because deleted elections are uncommon its the sort of thing people are unlikely to notice. I could see "ah yes - you need to filter by moderation_status=approved" being a really common answer to support queries if we did that.

Suggested and rejected elections are just never exposed via the API at all. Deleted elections are a bit of a special case, because they did exist as an "approved" election for some period of time before being deleted. When we implemented soft deletes in #418 we implemented this deleted flag 436df3e mainly because YNR was importing data from EE every hour (assume it still is, probably). That param gives YNR a query that allows it to find out about an soft-deleted elections, and delete any elections that were soft-deleted from EE after YNR has already imported them DemocracyClub/yournextrepresentative#672

On one hand, perhaps this was not the optimal solution.
On the other hand, you're still using it ~5 years later 😄

[GeoWill]

Yeah. That makes sense. So removing the deleted elections from the api/<org>/elections endpoint would probably be the right thing to do.