the ETL should be made transactional

[fgregg]

At various places, we delete objects and then replace them, but it can happen that the process of replacing them is not successful. We should make the deleting and the replacing happen within one transaction.

the import_contributions and import_expenditures are good examples.

openness-project-nmid/camp_fin/management/commands/import_transactions.py

Lines 146 to 214 in 5bf35c0

	def import_contributions(self, f, quarters, year, batch_size):
	reader = csv.DictReader(f)
	batch = []
	n_deleted = 0
	n_imported = 0
	for _, records in self._records_by_filing(reader, quarters):
	for i, record in enumerate(records):
	if i == 0:
	try:
	filing = self._get_filing(record)
	except ValueError:
	break
	# The contributions files are organized by the year
	# of the transaction date, not the date of the
	# filing, so transactions from the same filing can
	# appear in multiple contribution files.
	#
	# We need to make sure we just clear out the
	# contributions in a file that were purportedly made
	# in a given year.
	n_loans_deleted, _ = models.Loan.objects.filter(
	filing=filing, received_date__year=year
	).delete()
	n_events_deleted, _ = models.SpecialEvent.objects.filter(
	filing=filing, event_date__year=year
	).delete()
	n_transactions_deleted, _ = (
	models.Transaction.objects.filter(
	filing=filing, received_date__year=year
	)
	.exclude(transaction_type__description="Monetary Expenditure")
	.delete()
	)
	n_deleted += (
	n_loans_deleted + n_events_deleted + n_transactions_deleted
	)
	contributor = self.make_contributor(record)
	if (
	record["Contribution Type"] in {"Loans Received", "Special Event"}
	or "Contribution" in record["Contribution Type"]
	):
	contribution = self.make_contribution(record, contributor, filing)
	batch.append(contribution)
	else:
	self.stderr.write(
	f"Could not determine contribution type from record: {record['Contribution Type']}"
	)
	if len(batch) % batch_size == 0:
	self._save_batch(batch)
	n_imported += batch_size
	batch = []
	if len(batch) > 0:
	self._save_batch(batch)
	n_imported += len(batch)
	self.stdout.write(
	self.style.NOTICE(
	f"Deleted {n_deleted} records, created {n_imported} records"
	)
	)

we could do the transaction in a fine-grained way, or we could have the entire run of the script be in a transaction the way that pupa works.

[fgregg]

making the whole think atomic could be quite good, and looks pretty easy? https://github.com/opencivicdata/pupa/blob/2f7847cb87ed467f7afeec3f51cc704471b679c1/pupa/cli/commands/update.py#L259