refactor: separate toolkit into core library (aws#4475)

In order to share code between multiple extensions we need to create a core library. Rather than create a core lib package, we decided to rename the current toolkit as a core library and create a barebones replacement that will generate the extension by importing from the core lib. This is easier than searching for common components, unwiring them, and hoisting them into an empty core lib package.

List of commits:

* initial restructuring and code moving

* rename toolkit to core

* rename stuff to core, 'inherit' fields from core package.json

* rename toolkit2 to toolkit, cleanup

* debugging works (non chrome/webpack only)

* run in browser (but debugging doesnt work)

* delete resources, package.nls.json

* move/update auxillary files, e.g. types/, templates/, etc

* update vscode configurations, build scripts, core lib tests.

* update infra configs (e.g. root package.json, codecov)

* remove unnecessary changes

* update AWS.Token import error comments

* fix web tests by adding webCompile to core

* generalize .vscodeignore, remove dev artifacts

* fix package.json scripts

* update documentation

* fix import from mainline commit, fix prepublish script

* remove compile from postinstall in core lib

* try compiling on install with bigger heap size

* try to fix flaky linux unit tests

* add more docs + workspace file for easier opening

* fix up configs

* update docs to specify how to import from core lib

.eslintignore

@@ -12,8 +12,8 @@ src/codewhisperer/client/codewhispererclient.d.ts
 src/codewhisperer/client/codewhispereruserclient.d.ts
 src/amazonqFeatureDev/client/featuredevproxyclient.d.ts
 src/testFixtures/**
-packages/toolkit/src/shared/telemetry/clienttelemetry.d.ts
-packages/toolkit/src/codewhisperer/client/codewhispererclient.d.ts
-packages/toolkit/src/codewhisperer/client/codewhispereruserclient.d.ts
-packages/toolkit/src/amazonqFeatureDev/client/featuredevproxyclient.d.ts
-packages/toolkit/src/testFixtures/**
+packages/core/src/shared/telemetry/clienttelemetry.d.ts
+packages/core/src/codewhisperer/client/codewhispererclient.d.ts
+packages/core/src/codewhisperer/client/codewhispereruserclient.d.ts
+packages/core/src/amazonqFeatureDev/client/featuredevproxyclient.d.ts
+packages/core/src/testFixtures/**

.github/CODEOWNERS

@@ -1,5 +1,5 @@
 * @aws/aws-ides-team
-packages/toolkit/src/codewhisperer/ @aws/codewhisperer-team
-packages/toolkit/src/amazonqFeatureDev/ @aws/earlybird
-packages/toolkit/src/codewhispererChat/ @aws/aws-mynah
-packages/toolkit/src/amazonq/ @aws/aws-mynah
+packages/core/src/codewhisperer/ @aws/codewhisperer-team
+packages/core/src/amazonqFeatureDev/ @aws/earlybird
+packages/core/src/codewhispererChat/ @aws/aws-mynah
+packages/core/src/amazonq/ @aws/aws-mynah

.github/dependabot.yml

@@ -7,7 +7,7 @@
 version: 2
 updates:
     - package-ecosystem: 'npm'
-      directory: './packages/toolkit/' # Location of package manifests.
+      directory: './packages/core/' # Location of package manifests.
       target-branch: 'master' # Avoid updates to "staging".
       commit-message:
           prefix: 'deps'
@@ -25,7 +25,7 @@ updates:
                   - '@smithy*'
                   - 'smithy*'
     - package-ecosystem: 'github-actions'
-      directory: './packages/toolkit/'
+      directory: './packages/core/'
       target-branch: 'master' # Avoid updates to "staging".
       commit-message:
           prefix: 'deps'

.gitignore

@@ -19,34 +19,37 @@ __pycache__
 /dependency-graph.svg
 
 # Auto generated definitions
-packages/toolkit/src/**/*.gen.ts
+packages/*/src/**/*.gen.ts
 src.gen/*
+
 # Telemetry definition for testing adding telemetry
-packages/toolkit/src/shared/telemetry/vscodeTelemetry.json
+packages/core/src/shared/telemetry/vscodeTelemetry.json
 
 # Test reports
 .test-reports
 
 # Auto generated type definitions
-packages/toolkit/src/shared/telemetry/clienttelemetry.d.ts
-packages/toolkit/src/codewhisperer/client/codewhispererclient.d.ts
-packages/toolkit/src/codewhisperer/client/codewhispereruserclient.d.ts
-packages/toolkit/src/amazonqFeatureDev/client/featuredevproxyclient.d.ts
+packages/core/src/shared/telemetry/clienttelemetry.d.ts
+packages/core/src/codewhisperer/client/codewhispererclient.d.ts
+packages/core/src/codewhisperer/client/codewhispereruserclient.d.ts
+packages/core/src/amazonqFeatureDev/client/featuredevproxyclient.d.ts
 
 # Generated by tests
-packages/toolkit/src/testFixtures/**/bin
-packages/toolkit/src/testFixtures/**/obj
+packages/core/src/testFixtures/**/bin
+packages/core/src/testFixtures/**/obj
 
 # Generated by copyFiles.ts
 packages/*/README*
 packages/*/CHANGELOG.md
 packages/*/LICENSE
 packages/*/NOTICE
+packages/toolkit/package.nls.json
+packages/toolkit/resources
 
 # Icons
-packages/toolkit/resources/icons/cloud9/generated/**
-packages/toolkit/resources/fonts/aws-toolkit-icons.woff
-packages/toolkit/resources/css/icons.css
+packages/*/resources/icons/cloud9/generated/**
+packages/*/resources/fonts/aws-toolkit-icons.woff
+packages/*/resources/css/icons.css
 
 # local configuration
 .local.env

.prettierignore

@@ -11,5 +11,5 @@ src.gen/**
 src/shared/telemetry/service-2.json
 src/testFixtures/**
 
-packages/toolkit/src/shared/telemetry/service-2.json
-packages/toolkit/src/testFixtures/**
+packages/core/src/shared/telemetry/service-2.json
+packages/core/src/testFixtures/**

CONTRIBUTING.md

@@ -9,8 +9,9 @@ codebase and sending pull requests.
 ## Getting Started
 
 This project is set up as a typescript monorepo. The documentation throughout this project
-is referring to the subproject in [`packages/toolkit/`](./packages/toolkit/). For more information,
-see [ARCHITECTURE.md](./docs/ARCHITECTURE.md#monorepo-structure)
+is referring to the subprojects [`packages/toolkit/`](./packages/toolkit/) and [`packages/core/`](./packages/core/).
+Please see [ARCHITECTURE.md](./docs/ARCHITECTURE.md#monorepo-structure) to understand the
+structure of thispackage before contributing.
 
 ### Find things to do
 
@@ -40,8 +41,8 @@ Then clone the repository and install NPM packages:
 
 ### Run
 
-Due to the monorepo structure of the project, you can run the extension in VSCode by either opening the
-`aws-toolkit-vscode/packages/toolkit` folder directly, or adding it as a root folder in the VSCode Workspace.
+Due to the monorepo structure of the project, you must have the `aws-toolkit-vscode/packages/toolkit` folder open as root folder in the workspace.
+The easiest way to open the project: File > Open Workspace from File > choose `aws-toolkit-vscode/aws-toolkit-vscode.code-workspace`
 
 To run the extension from VSCode as a Node.js app:
 
@@ -161,9 +162,8 @@ See [web.md](./docs/web.md) for working with the web mode implementation of the
 See [TESTPLAN.md](./docs/TESTPLAN.md) to understand the project's test
 structure, mechanics and philosophy.
 
-You can run tests directly from VSCode. Due to the monorepo structure of the project, you must either open
-the `aws-toolkit-vscode/packages/toolkit/` folder directly, or add it as a root folder in the VSCode Workspace.
-Then:
+You can run tests directly from VSCode. Due to the monorepo structure of the project, you must have the `aws-toolkit-vscode/packages/toolkit` folder open as root folder in the workspace.
+The easiest way to open the project: File > Open Workspace from File > choose `aws-toolkit-vscode/aws-toolkit-vscode.code-workspace`
 
 1. Select `View > Debug`, or select the Debug pane from the sidebar.
 2. From the dropdown at the top of the Debug pane, select the `Extension Tests` configuration.

aws-toolkit-vscode.code-workspace

@@ -0,0 +1,19 @@
+{
+	"folders": [
+		{
+			"path": "."
+		},
+		{
+			"path": "packages/core"
+		},
+		{
+			"path": "packages/toolkit"
+		},
+		{
+			"path": "packages/amazonq"
+		}
+	],
+	"settings": {
+		"typescript.tsdk": "node_modules/typescript/lib"
+	}
+}

buildspec/linuxE2ETests.yml

@@ -8,6 +8,11 @@ env:
         AWS_TOOLKIT_TEST_NO_COLOR: '1'
         # Suppress noisy apt-get/dpkg warnings like "debconf: unable to initialize frontend: Dialog").
         DEBIAN_FRONTEND: 'noninteractive'
+        # Required dir overrides, otherwise the test will likely fail due to too long path names.
+        # E.g. WARNING: IPC handle <path> is longer than 107 chars, try a shorter --user-data-dir
+        # followed by Error: Could not delete obsolete instance handle Error: ENOENT: no such file or directory, unlink <path>
+        AWS_TOOLKIT_TEST_CACHE_DIR: '/tmp/.vscode-test/'
+        AWS_TOOLKIT_TEST_USER_DIR: '/tmp/.vscode-test/user-data/'
 
 phases:
     install:

buildspec/linuxIntegrationTests.yml

@@ -11,6 +11,11 @@ env:
         AWS_TOOLKIT_TEST_NO_COLOR: '1'
         # Suppress noisy apt-get/dpkg warnings like "debconf: unable to initialize frontend: Dialog").
         DEBIAN_FRONTEND: 'noninteractive'
+        # Required dir overrides, otherwise the test will likely fail due to too long path names.
+        # E.g. WARNING: IPC handle <path> is longer than 107 chars, try a shorter --user-data-dir
+        # followed by Error: Could not delete obsolete instance handle Error: ENOENT: no such file or directory, unlink <path>
+        AWS_TOOLKIT_TEST_CACHE_DIR: '/tmp/.vscode-test/'
+        AWS_TOOLKIT_TEST_USER_DIR: '/tmp/.vscode-test/user-data/'
 
 phases:
     install:

buildspec/linuxTests.yml

@@ -8,6 +8,11 @@ env:
     shell: bash
     variables:
         AWS_TOOLKIT_TEST_NO_COLOR: '1'
+        # Required dir overrides, otherwise the test will likely fail due to too long path names.
+        # E.g. WARNING: IPC handle <path> is longer than 107 chars, try a shorter --user-data-dir
+        # followed by Error: Could not delete obsolete instance handle Error: ENOENT: no such file or directory, unlink <path>
+        AWS_TOOLKIT_TEST_CACHE_DIR: '/tmp/.vscode-test/'
+        AWS_TOOLKIT_TEST_USER_DIR: '/tmp/.vscode-test/user-data/'
 
 phases:
     install:
@@ -25,12 +30,12 @@ phases:
     build:
         commands:
             - export HOME=/home/codebuild-user
-            # TODO: remove this after lint.yml is enabled in codebuild.
-            - |
-                if [ "$VSCODE_TEST_VERSION" = 'insiders' ] ; then
-                    npm run testCompile
-                    npm run lint
-                fi
+            # TODO: Enable if we run into issues with GHA linting or remove.
+            # - |
+            #     if [ "$VSCODE_TEST_VERSION" = 'insiders' ] ; then
+            #         npm run testCompile
+            #         npm run lint
+            #     fi
             - |
                 {
                     # Ensure that "foo | run_and_report" fails correctly.

buildspec/shared/linux-pre_build.sh

@@ -22,4 +22,5 @@ if [ "$TOOLKITS_CODEARTIFACT_DOMAIN" ] && [ "$TOOLKITS_CODEARTIFACT_REPO" ] && [
 fi
 
 # TODO: move this to the "install" phase?
+export NODE_OPTIONS=--max-old-space-size=8192
 npm 2>&1 ci | run_and_report 2 'npm WARN deprecated' 'Deprecated dependencies must be updated.'

codecov.yml

@@ -38,7 +38,7 @@ github_checks:
 flags:
     codewhisperer:
         paths:
-            - packages/toolkit/src/codewhisperer/
+            - packages/core/src/codewhisperer/
     amazonqFeatureDev:
         paths:
-            - packages/toolkit/src/amazonqFeatureDev/
+            - packages/core/src/amazonqFeatureDev/

designs/credentials/credentials-management.md

@@ -31,11 +31,11 @@ A formatted version of the Credentials Provider Id may be surfaced to users, how
 
 When the user connects to AWS in the Toolkit, a Credentials Provider is requested, which is then used to obtain credentials. The toolkit requests a Credentials Provider by checking which credentials provider factories support the provider's credentials type. The factories of interest are queried to see which (if any) have the requested Credentials Provider.
 
-At the time this document was written, Shared Credentials are the only supported Credentials. Additional credentials providers will reside at [/src/credentials/providers](/packages/toolkit/src/credentials/providers) as they are implemented.
+At the time this document was written, Shared Credentials are the only supported Credentials. Additional credentials providers will reside at [/src/credentials/providers](/packages/core/src/credentials/providers) as they are implemented.
 
 ### Shared Credentials Profiles
 
-Profiles are retrieved from the user's shared credentials files. The profile is handled and validated differently based on which fields are present. Handling and validation logic can be found in [sharedCredentialsProvider.ts](/packages/toolkit/src/credentials/providers/sharedCredentialsProvider.ts).
+Profiles are retrieved from the user's shared credentials files. The profile is handled and validated differently based on which fields are present. Handling and validation logic can be found in [sharedCredentialsProvider.ts](/packages/core/src/credentials/providers/sharedCredentialsProvider.ts).
 
 Only profiles that are considered valid are provided to the toolkit. When validation issues are detected, they are written to the logs to help users understand why the toolkit is having difficulties with a profile. Users running the 'Connect to AWS' command will not see invalid profiles in the list of Credentials.
 

designs/modify-resources-attached-to-code-pipeline.md

@@ -150,7 +150,7 @@ The confirmation prompt can be implemented using one of two UI facilities:
 
 ### API Client
 
-The `ResourceGroupsTaggingAPI` service client will be used. The client will be set up in a manner consistent with the Lambda and CloudFormation clients, allowing the clients to be stubbed out in unit tests. See the Toolkit's Lambda Client [Interface](/packages/toolkit/src/shared/clients/lambdaClient.ts) and [Implementation](/packages/toolkit/src/shared/clients/defaultLambdaClient.ts) as an example. A new client factory will be added to [ToolkitClientBuilder](/packages/toolkit/src/shared/clients/toolkitClientBuilder.ts) and [DefaultToolkitClientBuilder](/packages/toolkit/src/shared/clients/defaultToolkitClientBuilder.ts).
+The `ResourceGroupsTaggingAPI` service client will be used. The client will be set up in a manner consistent with the Lambda and CloudFormation clients, allowing the clients to be stubbed out in unit tests. See the Toolkit's Lambda Client [Interface](/packages/core/src/shared/clients/lambdaClient.ts) and [Implementation](/packages/core/src/shared/clients/defaultLambdaClient.ts) as an example. A new client factory will be added to [ToolkitClientBuilder](/packages/core/src/shared/clients/toolkitClientBuilder.ts) and [DefaultToolkitClientBuilder](/packages/core/src/shared/clients/defaultToolkitClientBuilder.ts).
 
 ### <a id="fallback-handling"></a>Fallback Handling
 

docs/ARCHITECTURE.md

@@ -4,13 +4,29 @@ An overview of the architecture for various components within the Toolkit.
 
 ## Monorepo Structure
 
-This project is currently set up as a typescript monorepo with a single subproject.
-We are currently working on splitting the Toolkit into various subprojects to help with
-sharing code between modules, Web mode development, etc. For now, there is just
-one monolithic subproject with all the extension functionality: [`packages/toolkit/`](./packages/toolkit/).
+**The easiest way to open the project for running/debugging:** File > Open Workspace from File > choose `aws-toolkit-vscode/aws-toolkit-vscode.code-workspace`
+
+This project is currently set up as a typescript monorepo with the following subprojects:
+
+-   [`packages/core/`](./packages/core/)
+    -   Currently, this package contains almost all of the functionality required for the extension.
+        It was created by moving all of the code from `packages/tookit/` to here.
+        We are currently working on separating non-shareable code out of the core library into the relevant
+        subproject. Running tests for most of the toolkit extension logic occurs in this subproject.
+-   [`packages/toolkit/`](./packages/toolkit/)
+    -   Currently, this package is a barebones wrapper that calls activation functions from the core library.
+        As we discover what code cannot be shared across other projects, it will be moved to this package.
+        Running and packaging the extension occurs from this subproject.
+
+If you are considering contributing, please consider whether your implementation should live in the core
+library or in `packages/toolkit`. If your work could be re-used by other packages (e.g. auth mechanisms,
+utilities), then it may belong in the core library. If instead you are adding something toolkit specific
+(eg. an integration to a new AWS service in the Explorer Tree), consider putting it in `packages/toolkit`.
+To import from the core library, please export your desired code using `index.ts` files, and add an appropriate `exports` statement
+in `packages/core/package.json`.
 
 Unless otherwise stated, the documentation throughout this project is referring to the code and
-functionality of that subproject.
+functionality in `packages/core/` and `packages/toolkit`.
 
 Current quirks of the current monorepo status that should be resolved/evaluated in later versions (TODO):
 
@@ -26,20 +42,27 @@ Current quirks of the current monorepo status that should be resolved/evaluated
         for more info on how `node_modules/` hoisting works.
     -   Because of `node_modules/` hoisting, references to this folder in code access the root project modules folder. This may be
         an issue if more subprojects are added and the contents of the root and local modules folders differ.
--   [`globalSetup.test.ts`](../packages/toolkit/src/test/globalSetup.test.ts) should be configured to work as a library/run tests for all subprojects.
+-   [`globalSetup.test.ts`](../packages/core/src/test/globalSetup.test.ts) should be configured to work as a library/run tests for all subprojects.
 -   Subproject `tsconfig.json`s should extend a root `tsconfig.packages.json`.
 -   Linting tests should run at the root level, not subproject level.
--   `packages/toolkit/scripts/` should be generalized and moved to the root of the project as needed.
+-   `packages/*/scripts/` should be generalized and moved to the root of the project as needed.
 -   LICENSE, README.md, and other non-code artifacts that must be packaged into the .vsix are currently
     being copied into the packaging subproject directory from the root project directory as part of the `copyFiles` task.
 -   Pre-release only publishes packages/toolkit extension directly. It should be extended to other added extensions. See [`release.yml`](../.github/workflows/release.yml)
 -   VSCode does not support inheriting/extending `.vscode/` settings: https://github.com/microsoft/vscode/issues/15909
 
+Additional quirks introduced by creating a core library from the original extension code:
+
+-   Tests are ran from `packages/core/`
+-   Extension runs from `packages/toolkit`
+-   Extension tests run from the core lib. Since some of the tests require an extension context/sandbox, we initiate a "fake" extension to run these tests. This is also why there are vscode extensionproperties in the package.json
+-   Some of original extension code (that now lives in `packages/core`) depends on the package.json, specifically the contributes section. This section is very large AND needs to be present in both the core library and toolkit extension package.jsons. The core library code needs access to this section to create types, set up SAM debuggers, etc. The toolkit needs this section during packaging/debugging so that the extension can run in vscode. The short term solution was to creat a [build script](../packages/toolkit/scripts/build/handlePackageJson.ts) to copy necessary fields over to the toolkit extension during packaging and debugging.
+
 ## Commands
 
 Many parts of the VS Code API relies on the use of 'commands' which are simply functions bound to a global ID. For small projects, this simplicity is great. But the base API doesn't offer a lot of common functionality that a large project might want: logging, error handling, etc.
 
-For the Toolkit, common command functionality is implemented in [Commands](../packages/toolkit/src/shared/vscode/commands2.ts). The goal with this abstraction is to increase consistency across the Toolkit for anything related to commands.
+For the Toolkit, common command functionality is implemented in [Commands](../packages/core/src/shared/vscode/commands2.ts). The goal with this abstraction is to increase consistency across the Toolkit for anything related to commands.
 
 ### Examples
 
@@ -123,7 +146,7 @@ _See also [CODE_GUIDELINES.md](./CODE_GUIDELINES.md#exceptions)._
 
 Large applications often have a correspondingly large number of failure points. For feature-level logic, these failures are often non-recoverable. The best we can do is show the user that something went wrong and maybe offer guidance on how to fix it.
 
-Because this is such a common pattern, shared error handling logic is defined by `ToolkitError` found [here](../packages/toolkit/src/shared/errors.ts). This class provides the basic structure for errors throughout the Toolkit.
+Because this is such a common pattern, shared error handling logic is defined by `ToolkitError` found [here](../packages/core/src/shared/errors.ts). This class provides the basic structure for errors throughout the Toolkit.
 
 ### Chaining
 
@@ -133,7 +156,7 @@ By adding additional information as the exception bubbles up, we can create a be
 
 ### Handlers
 
-Any code paths exercised via `Commands` will have errors handled by `handleError` in [extensions.ts](../packages/toolkit/src/extension.ts). A better API for error handling across more than just commands will be added in a future PR.
+Any code paths exercised via `Commands` will have errors handled by `handleError` in [extensions.ts](../packages/core/src/extension.ts). A better API for error handling across more than just commands will be added in a future PR.
 
 ### Best Practices