Accumulated Technical Debt

[GracefulLemming]

The Big Picture

Over the past 3 years we have ballooned from a small reading environment with limited language tools to a massive hub for collections, translation and editing, and curricular materials. We have made a number of major changes to our backend infrastructure to support this scaling. Since development work tracks closely to grant goals, we have a large amount of technical debt including stale code, lacking documentation, and ad-hoc solutions that need to be fixed before we can realize our goal of giving this code directly to communities.

Use this thread to...

• Identify stale code
• Identify insufficient or inconsistent types and contracts
• Explore abstractions and other refactoring
• Consider further modularization of functionality
• Note missing or out of date documentation
• Note any inconsistent code practices
• Any other clean-up related work!

Future Work

Once we clear our current grant goals, we will pivot to a large code clear-out using items in this thread as a guide.

[CharlieMcVicker]

I think we should exclusively use query_as! and not query! in database_sql.rs

[GracefulLemming]

There are a number of leftover artifacts from attempts to make DAILP more linguist-focused. Off of memory, this includes some semantics such as AnnotatedForms over something simpler like Words and some timeline tools for comparative and historical morphology.

Most items simply need to be renamed or removed.

Some features we may want to consider making into a Cherokee-first version. For example, while a language timeline linking Cherokee to something like proto-iroquoian is not of interest to us, people are interested in knowing which words are old words. As of right now though, this feels like more trouble than its worth.

[GracefulLemming]

@CharlieMcVicker and I recently resurfaced this, with specific discussion on both the "linguist/learner" distinction and the naming schemes within our writing system options

[GracefulLemming]

Sometimes a build can succeed in GitHub Actions but fail in AWS Amplify. To make debugging in these scenarios easier, Terraform + Amplify should log config.resources.aws-amplify-app.[app name].production-branch.status to somewhere like an S3 bucket that dev team can access.

[CharlieMcVicker]

We should make all of our migrations reversible! This will require writing down migration steps for all existing migrations.

See: Reverting Migrations section of https://lib.rs/crates/sqlx-cli

[LN7-cyber]

Document our stack with an image overlay and include brief descriptions of all parts of the stack.

[LN7-cyber]

AWS bucket documentation overview. Defining items, what they house and access to them.

[LN7-cyber]

User permissions, once set, include an overview in the documentation. Include a summary of how these were community identified to maintain oversight on culturally sensitive materials and workflows for particular collections, such as CWKW.

[LN7-cyber]

For community-based designs of reading and writing environments, include documentation of the process and potentially the materials we use to collect that feedback. Link out to the scholarship our team has produced related to this (Snead and Cushman, MLJ 2022; Cushman and Trevino, ASJ, on decolonial translation process 2021).

[LN7-cyber]

Documentation regarding known issues for internal and external purposes.

[GracefulLemming]

We have a few conceptual inconsistencies creating needless complexity:

1. We half commit to the idea that everything is a document

• In the editing environment, we can only edit documents that fit our interlinearization
• Dictionaries & sim. resources are treated as quasi-documents– they are imported as documents with IDs and slugs but are treated as distinct from non-resource documents

2. We have inconsistent design patterns across our stack. Some examples:

• We should prefer join tables over lists on tables
• We should ensure there are strict, useful, and predictable contracts between frontend and backend (notably: a document "path" should be able to straightforwardly create a valid url location, not be used for making the TOC of a collection...)

[CharlieMcVicker]

We should setup persistent logging (ie. to S3) for our GraphQL lambda function

[CharlieMcVicker]

We may someday want to get off Wordpress entirely. While going through the backlog, Naomi and I came across this idea:

Route wordpress single page contents query through our GraphQL server

Eventually I’d like us to route all requests to Wordpress through our own GraphQL server so that we can gradually migrate away from Wordpress later on, or at least give the front-end just one schema to worry about.

• Shelby

I'm not sure how viable this is now, but thinking long term about transitioning away from Wordpress belongs on this thread.

[LN7-cyber]

Maybe, but let's get a good sense of how many grassroots organizations/community groups/or tribes rely on WordPress before we do this.

[CharlieMcVicker]

Legacy browser support may be important for many community members. Moving to NextJS (away from vite) may be a way to get IE11 support. See this ticket for more info:

https://dailp.atlassian.net/browse/DP-95?atlOrigin=eyJpIjoiMjFlMDkyMjZlN2U0NDkyMGFkZTQ2ZTZmYmU0ZDI1OWQiLCJwIjoiaiJ9

[CharlieMcVicker]

How can we cultivate a better design language for users?

Here is a card that talks about some ideas: https://dailp.atlassian.net/browse/DP-131?atlOrigin=eyJpIjoiOWNmNTc4MWM2ZjdkNGQwYjk2ODJhODkwODUwNGZhYWYiLCJwIjoiaiJ9

[GracefulLemming]

Our project structure (specifically in the frontend) is bloated and inconsistent, making it difficult for devs to learn what various parts of the code do. Here are a few steps we could take:

1. Break down our monorepo. With our current size and scope, we should discuss if it makes sense to keep all of our code in one repo or break it down into a few separate repos (one for terraform, one for website, etc.). This will also make sharing parts of functionality easier in the future.
2. Reorganize our frontend directories. One example I have been looking at: https://scrimba.com/articles/react-project-structure/#ideal-folder-structure

[GracefulLemming]

We need testing infrastructure. Before we had features hidden behind auth and group guards, we could get away with ad-hoc testing. Now, for our sanity, we need a way to systematically test code.

[GracefulLemming]

We are overengineering a lot of things, esp. on the frontend. How can we trim these back to keep our devs sane?

As far as @CharlieMcVicker and I can tell now, the primary offender is the word pane

[LN7-cyber]

Yes, concern expressed from some community reviewers that the design of the word pane is overly done and complex. Granted we are showing the deepest IGT in our community feedback sessions. How can we layer the front end design of word pane to manage the information load on community members and trim our engineering on front end?

[GracefulLemming]

@LN7-cyber @CharlieMcVicker we should think about whether we want to do a repo-wide audit/review sometime in the late winter into spring. Were finding a lot as we go but I know a lot of issues are still slipping through

[GracefulLemming]

#393 (comment)

[GracefulLemming]

Add improved error handling across the stack, especially for errors in the writing environment that break user flows

[LN7-cyber]

Absolutely! We need to know when, where, and how errors are appearing in code. Good call.

[GracefulLemming]

We need to reconcile the difference between data ingested from sheets and data input from users.

[GracefulLemming]

phonemic spelling and normalized source are no longer used (except for some anomalous cases)