WEBDEV-6081 Refactoring to better separate data source concerns from …

…UI (#330)

* Create basic state management setup

* Facet state refactoring to minimize repetition

* Move facet refactor to different branch

* First pass extracting CB data source into a reactive controller

* Better abstraction for data source

* Setup for reactive controller lifecycle

* Top-level exports for data source

* Remove obsolete state management stub

* Build out some more robust interfaces for state management

* Facet state adjustments

* Facet state adjustments

* Migrate tile mapping function into new data source

* Add methods for tile mapping & check/unchecking to interface

* Migrate tile mapping/checking methods into data source

* Keep track of data source size

* Adjust some cb methods to delegate to data source

* Include data source size in interface

* Minor adjustments to how cb retrieves cell models

* Reorganize some properties

* Request host updates from reactive controller

* Fix some tests

* Add script for quicker test running

* Streamline models

* Conform collection browser to search state model

* Convert derived sortParam property into getter

* Fix some breaking tests related to sortParam

* Fix a model property naming issue

* Connect up data source w/ cb query change handlers

* Model updates for cb host

* Migrate several fetch methods over to data source

* Fix some convenience methods in cb

* Try migrating some page fetching routines

* Adjust search state model to accommodate page fetch

* Migrate page fetching & related props into data source

* Clean up cb a bit and delegate all remaining page fetches

* Excise collection-name-cache in favor of data source map

* Use basic map for collection titles

(no need for specialized get behavior since everywhere
it is used has a null-guard anyway)

* Data source should not fetch facets when suppressed

* Remove filterMap from search state interface (it's part of the datasource)

* Cleanup to avoid passing data source around

* Cleaning up some lingering details in cb

* Migrate prefix filters over to data source

* Improve data source documentation + general cleanliness

* Remove a bit of dead code in cb

* Fetch controller

* Cleaner data types for page target params

* Export TileModel interface

* Use updated search service models

* Ensure that profile pages are allowed to perform no-query searches

* Add profile target to query keys

* Ensure query change updates occur on profile target changes

* Use hits/aggregations from page_elements branch if available

* Rely on search service to handle page_element unpacking

* Emit event for any query state changes

* Add debug logging & fix result count bug

* Fix broken test

* Properly reset endOfDataReached upon query change

* Fix a bug where checking tile in manage view doesn't work

* Ensure result views are refreshed when checking/unchecking all in manage mode

* Fix several unit tests

* Fix bug where scroller tile count not being set correctly on short result sets

* Unit Tests Updates (#336)

* Use hits/aggregations from page_elements branch if available

* Rely on search service to handle page_element unpacking

* Sort Bar Test - Ensure Resize Observer fires ONLY when needed
- this breaks View when dynamically toggling sort bar on an already loaded collection browser

* TESTS - Fix some, comment some out that need rework with the refactor - WEBDEV-6801

* Unskip a few tests that were fixed in the refactor branch

* Uncomment a few more tests that have been fixed

* Further test fixes

* Fix last two remaining unit test failures

---------

Co-authored-by: Laton Vermette <[email protected]>

* Pull some helper functions out into utils

* Promote TileModel from a bare interface to a class, to better organize conversion from search results

* Move reset method up for clarity

* Move RequestKind into models file

* Upgrade search service version & fix tests that use tile model

* Fixes some display issues related to updated model structure & adding web capture links

* Refactor data source installation & fix broken web archive images

* Ensure data source only modifies host when it is attached

* Unskip fixed tests

* Do not rely on current host state for fetch invalidation, since it may be attached to a different data source

* Apply loading states on host connect

* Fix boolean flip bug on facet fetch

* Apply correct default sorts for profile tabs

* Ensure facets & date picker maintain the correct state when data source installed

* Debug logging for page scroll issue

* Add some missing unit tests

* Additional unit tests & minor refactor for letter filter updates

* Additional test fixes

* Unit test for empty results case

* Fix issues with letter filter bar & extra fetches

* Clean up some debug logging

* Upgrade search-service dependency

* Upgrades search-service off alpha tag

* Remove unused local-cache dependency

* Small fix for web archive results beyond first page + tests

---------

Co-authored-by: Isa Herico Velasco <[email protected]>

index.ts

@@ -1,6 +1,9 @@
 export { CollectionBrowser } from './src/collection-browser';
+export { CollectionBrowserDataSource } from './src/data-source/collection-browser-data-source';
+export { CollectionBrowserDataSourceInterface } from './src/data-source/collection-browser-data-source-interface';
+export { CollectionBrowserQueryState } from './src/data-source/collection-browser-query-state';
 export { SortFilterBar } from './src/sort-filter-bar/sort-filter-bar';
-export { CollectionDisplayMode, SortField } from './src/models';
+export { CollectionDisplayMode, SortField, TileModel } from './src/models';
 export { CollectionBrowserLoadingTile } from './src/tiles/collection-browser-loading-tile';
 export { CollectionTile } from './src/tiles/grid/collection-tile';
 export { AccountTile } from './src/tiles/grid/account-tile';

package.json

@@ -14,6 +14,7 @@
     "format": "eslint --ext .ts,.html . --fix --ignore-path .gitignore && prettier \"**/*.ts\" --write --ignore-path .gitignore",
     "circular": "madge --circular --extensions ts .",
     "test": "tsc && yarn run lint && yarn run circular && wtr --coverage",
+    "test:fast": "tsc && wtr --coverage",
     "test:watch": "tsc && concurrently -k -r \"tsc --watch --preserveWatchOutput\" \"wtr --watch\"",
     "deploy": "yarn run deploy:run -e $(git branch --show-current)",
     "deploy:run": "yarn run prepare:ghpages && touch ghpages/.nojekyll && yarn run deploy:gh",
@@ -23,16 +24,14 @@
   "types": "dist/index.d.ts",
   "dependencies": {
     "@internetarchive/analytics-manager": "^0.1.2",
-    "@internetarchive/collection-name-cache": "^0.2.16",
     "@internetarchive/feature-feedback": "^0.1.4",
     "@internetarchive/field-parsers": "^0.1.4",
     "@internetarchive/histogram-date-range": "^1.2.0",
     "@internetarchive/ia-activity-indicator": "^0.0.4",
     "@internetarchive/ia-dropdown": "^1.3.6",
     "@internetarchive/infinite-scroller": "1.0.1",
-    "@internetarchive/local-cache": "^0.2.1",
     "@internetarchive/modal-manager": "^0.2.8",
-    "@internetarchive/search-service": "^1.2.4",
+    "@internetarchive/search-service": "^1.3.0",
     "@internetarchive/shared-resize-observer": "^0.2.0",
     "@lit/localize": "^0.11.2",
     "dompurify": "^2.3.6",

src/app-root.ts

@@ -9,11 +9,9 @@ import {
   SearchType,
   StringField,
 } from '@internetarchive/search-service';
-import { LocalCache } from '@internetarchive/local-cache';
 import { html, css, LitElement, PropertyValues, nothing } from 'lit';
 import { customElement, property, query, state } from 'lit/decorators.js';
 import { SharedResizeObserver } from '@internetarchive/shared-resize-observer';
-import { CollectionNameCache } from '@internetarchive/collection-name-cache';
 
 import type { ModalManagerInterface } from '@internetarchive/modal-manager';
 import type { AnalyticsManagerInterface } from '@internetarchive/analytics-manager';
@@ -28,13 +26,6 @@ export class AppRoot extends LitElement {
 
   private resizeObserver = new SharedResizeObserver();
 
-  private localCache = new LocalCache();
-
-  private collectionNameCache = new CollectionNameCache({
-    searchService: this.searchService,
-    localCache: this.localCache,
-  });
-
   @state() private toggleSlots: boolean = false;
 
   @state() private currentPage?: number;
@@ -440,7 +431,6 @@ export class AppRoot extends LitElement {
           .baseImageUrl=${'https://archive.org'}
           .searchService=${this.searchService}
           .resizeObserver=${this.resizeObserver}
-          .collectionNameCache=${this.collectionNameCache}
           .showHistogramDatePicker=${true}
           .loggedIn=${this.loggedIn}
           .modalManager=${this.modalManager}