From 733adb946120100985898a9d1f821a85e86dad32 Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Fri, 24 May 2024 22:32:00 +0530 Subject: [PATCH 01/17] feat: added design doc for sparse checkout Signed-off-by: Asish Kumar --- .../design-doc/sparse_checkout_asishkumar.md | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 research/design-doc/sparse_checkout_asishkumar.md diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md new file mode 100644 index 00000000..181f375a --- /dev/null +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -0,0 +1,79 @@ +# KPM sparse checkout + +**Author**: Asish Kumar + +## Abstract + +`kpm` manages third-party libraries through Git repositories, requiring a `kcl.mod` file at the root directory. It treats the entire Git repository as a single `kcl` package, which is inefficient for monorepos containing multiple `kcl` packages. Often, a `kcl` project depends on just one package within a monorepo, but `kpm` downloads the entire repository. Therefore, `kpm` needs to allow adding a subdirectory of a Git repository as a dependency, enabling it to download only the necessary parts and improve performance. + +## User Interface + +I will add a new flag called `--subdir` in `kpm add` command. This flag will specify the path to the desired subdirectory within the Git repository. Below is the syntax for the enhanced kpm add command: + +``` +kpm add --subdir +``` + +The `--subdir` flag will be optional. If the flag is not provided, `kpm` will download the entire repository as it does now. If the flag is provided, `kpm` will download only the specified subdirectory. The `kcl.mod` file will be generated with the path to the subdirectory. + +Example usage: + +``` +kpm add --subdir 1.21/* k8s +``` + +This command will download the `1.21` directory and all its contents from the `k8s` repository hosted in https://github.com/kcl-lang/modules + + +The `kcl.mod` file of the users project will also contain an array of path to the subdirectories. + +``` +[dependencies] +bbb = { path = "../bbb", subdir = ["test-*", "test-*"]} +``` + +## Design + +The path to the directory will be passed to `CloneOptions` in [pkg/git/git.go](https://github.com/kcl-lang/kpm/blob/d20b1acdc988f600c8f8465ecd9fe04225e19149/pkg/git/git.go#L19) as subDir. + +### using go-getter + +As mentioned in the [go-getter](https://pkg.go.dev/github.com/hashicorp/go-getter#readme-subdirectories) docs, we can append our subDir from `CloneOptions` (only if subDir is not empty) in `WithRepoURL` function. + +### using go-git + +This process will involve using the `sparse-checkout` feature of git. + +1. Initialize a new git repository in the local `.kcl/kpm/` directory using [PlainInit](https://pkg.go.dev/github.com/go-git/go-git#PlainInit). The repository name will be the PackageName_version. + +2. Create a new worktree using [Worktree](https://pkg.go.dev/github.com/go-git/go-git/v5#Repository.Worktree) + +3. Enable the sparse-checkout feature using [SparseCheckout](https://pkg.go.dev/github.com/go-git/go-git/v5#Worktree.SparseCheckout). The second argument will be a slice of strings containing the subdirectory path. + +4. Add the remote repository using [AddRemote](https://pkg.go.dev/github.com/go-git/go-git/v5#Repository.CreateRemote) + +5. Pull the repository using [Pull](https://pkg.go.dev/github.com/go-git/go-git/v5#Worktree.Pull) + +Whenever we want to access the subdirectory using any command, we can refer to `kcl.mod` file of the project and iterate over the `subdir` array to get the path to the subdirectory. The `kcl.mod` file will automatically get updated whenever `kpm add` command is run. + + +### Additional modifications + +To avoid creating a new root for each subdirectory download, I can add some check functions. + +## References + +1. https://medium.com/@marcoscannabrava/git-download-a-repositorys-specific-subfolder-ceeabc6023e2 +2. https://pkg.go.dev/github.com/go-git/go-git/v5 +3. https://pkg.go.dev/github.com/hashicorp/go-getter + + + + + + + + + + + From 6352f5bdca2126d06b4ee63de3dff658824f7015 Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Tue, 28 May 2024 07:55:04 +0530 Subject: [PATCH 02/17] fix: format sparse checkout design Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index 181f375a..ade1da87 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -65,15 +65,4 @@ To avoid creating a new root for each subdirectory download, I can add some chec 1. https://medium.com/@marcoscannabrava/git-download-a-repositorys-specific-subfolder-ceeabc6023e2 2. https://pkg.go.dev/github.com/go-git/go-git/v5 -3. https://pkg.go.dev/github.com/hashicorp/go-getter - - - - - - - - - - - +3. https://pkg.go.dev/github.com/hashicorp/go-getter \ No newline at end of file From a52c590b02cdba9e607ae7cc29a5ef2d520c444e Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Tue, 28 May 2024 08:01:12 +0530 Subject: [PATCH 03/17] fix: formatted sparse checkout design Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index ade1da87..f531b95b 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -24,7 +24,6 @@ kpm add --subdir 1.21/* k8s This command will download the `1.21` directory and all its contents from the `k8s` repository hosted in https://github.com/kcl-lang/modules - The `kcl.mod` file of the users project will also contain an array of path to the subdirectories. ``` @@ -56,7 +55,6 @@ This process will involve using the `sparse-checkout` feature of git. Whenever we want to access the subdirectory using any command, we can refer to `kcl.mod` file of the project and iterate over the `subdir` array to get the path to the subdirectory. The `kcl.mod` file will automatically get updated whenever `kpm add` command is run. - ### Additional modifications To avoid creating a new root for each subdirectory download, I can add some check functions. @@ -65,4 +63,4 @@ To avoid creating a new root for each subdirectory download, I can add some chec 1. https://medium.com/@marcoscannabrava/git-download-a-repositorys-specific-subfolder-ceeabc6023e2 2. https://pkg.go.dev/github.com/go-git/go-git/v5 -3. https://pkg.go.dev/github.com/hashicorp/go-getter \ No newline at end of file +3. https://pkg.go.dev/github.com/hashicorp/go-getter From f4e65e2f9961d1807fbfa8505d3038e0ff01ad59 Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Tue, 28 May 2024 21:18:03 +0530 Subject: [PATCH 04/17] update design doc with additional info Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index f531b95b..c9fa8a28 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -55,9 +55,11 @@ This process will involve using the `sparse-checkout` feature of git. Whenever we want to access the subdirectory using any command, we can refer to `kcl.mod` file of the project and iterate over the `subdir` array to get the path to the subdirectory. The `kcl.mod` file will automatically get updated whenever `kpm add` command is run. -### Additional modifications +### Additional information -To avoid creating a new root for each subdirectory download, I can add some check functions. +1. To avoid creating a new root for each subdirectory download, I can add some check functions. + +2. The subdir flag is only for git options. If we pass it as a flag after oci, for example: `kpm add k8s --subdir 1.21/*`, it will not work. We can add a check [here](https://github.com/kcl-lang/kpm/blob/92158183556d39545bc0734a1e24284344ff3d9e/pkg/cmd/cmd_add.go#L154) that will give a warning if the subdir flag is passed. Furthermore, the subdir flag will only work for git repositories since it will insert the flag value into the field variable of the [Git](https://github.com/kcl-lang/kpm/blob/92158183556d39545bc0734a1e24284344ff3d9e/pkg/package/modfile.go#L375) struct. ## References From 3927f04f2fa8455f4f596f8eadee9e288c3d81b4 Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Wed, 29 May 2024 14:56:28 +0530 Subject: [PATCH 05/17] address comments sparse checkout design Signed-off-by: Asish Kumar --- .../design-doc/sparse_checkout_asishkumar.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index c9fa8a28..55bc4072 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -8,27 +8,27 @@ ## User Interface -I will add a new flag called `--subdir` in `kpm add` command. This flag will specify the path to the desired subdirectory within the Git repository. Below is the syntax for the enhanced kpm add command: +I will add a new flag called `--subdir` in `kcl mod add` command. This flag will specify the path to the desired subdirectory within the Git repository. Below is the syntax for the enhanced kpm add command: ``` -kpm add --subdir +kcl mod add --git --subdir ``` -The `--subdir` flag will be optional. If the flag is not provided, `kpm` will download the entire repository as it does now. If the flag is provided, `kpm` will download only the specified subdirectory. The `kcl.mod` file will be generated with the path to the subdirectory. +The `--subdir` flag will be optional. If the flag is not provided, `kpm` will download the entire repository as it does now. If the flag is provided, `kpm` will download only the specified subdirectory of the git repo. Example usage: ``` -kpm add --subdir 1.21/* k8s +kcl mod add --git https://github.com/kcl-lang/modules --subdir add-certificates-volume ``` -This command will download the `1.21` directory and all its contents from the `k8s` repository hosted in https://github.com/kcl-lang/modules +This command will download the `add-certificates-volume` subdirectory from the `modules` repository and append it in the subdir array of the `kcl.mod` file. -The `kcl.mod` file of the users project will also contain an array of path to the subdirectories. +The `kcl.mod` file will look like this: ``` [dependencies] -bbb = { path = "../bbb", subdir = ["test-*", "test-*"]} +bbb = { git = "https://github.com/kcl-lang/modules", subdir = ["add-certificates-volume"]} ``` ## Design @@ -53,7 +53,7 @@ This process will involve using the `sparse-checkout` feature of git. 5. Pull the repository using [Pull](https://pkg.go.dev/github.com/go-git/go-git/v5#Worktree.Pull) -Whenever we want to access the subdirectory using any command, we can refer to `kcl.mod` file of the project and iterate over the `subdir` array to get the path to the subdirectory. The `kcl.mod` file will automatically get updated whenever `kpm add` command is run. +Whenever we want to access the subdirectory using any command, we can refer to `kcl.mod` file of the project and iterate over the `subdir` array to get the path to the subdirectory. The `kcl.mod` file will automatically get updated whenever `kcl mod add` command is run. ### Additional information From f113f096d26857a11872dc52deb1be5acc64e0da Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Wed, 29 May 2024 22:07:15 +0530 Subject: [PATCH 06/17] feat: updated user interface url for sparse checkout design doc Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index 55bc4072..43cf784f 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -28,7 +28,7 @@ The `kcl.mod` file will look like this: ``` [dependencies] -bbb = { git = "https://github.com/kcl-lang/modules", subdir = ["add-certificates-volume"]} +bbb = { git = "https://github.com/kcl-lang/flask-demo-kcl-manifests.git", commit = "ade147b", subdir = ["something"]} ``` ## Design From d7da73df1689ffe74e98f29e95e102c4d975f37d Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Thu, 30 May 2024 12:15:47 +0530 Subject: [PATCH 07/17] explain why subdir is a list Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index 43cf784f..b2a8a4f8 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -28,9 +28,11 @@ The `kcl.mod` file will look like this: ``` [dependencies] -bbb = { git = "https://github.com/kcl-lang/flask-demo-kcl-manifests.git", commit = "ade147b", subdir = ["something"]} +bbb = { git = "https://github.com/kcl-lang/modules", commit = "ade147b", subdir = ["add-ndots"]} ``` +The subdir is a list because in the future if user wants to add another subdir from the same git repo then it can be added without overwritting the current subdir. + ## Design The path to the directory will be passed to `CloneOptions` in [pkg/git/git.go](https://github.com/kcl-lang/kpm/blob/d20b1acdc988f600c8f8465ecd9fe04225e19149/pkg/git/git.go#L19) as subDir. From 8b66da98e463a2a937e789ca3fd630cdfc8b9f52 Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Thu, 30 May 2024 12:29:21 +0530 Subject: [PATCH 08/17] new user interface Signed-off-by: Asish Kumar --- .../design-doc/sparse_checkout_asishkumar.md | 42 +++---------------- 1 file changed, 6 insertions(+), 36 deletions(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index b2a8a4f8..cffdef32 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -8,21 +8,13 @@ ## User Interface -I will add a new flag called `--subdir` in `kcl mod add` command. This flag will specify the path to the desired subdirectory within the Git repository. Below is the syntax for the enhanced kpm add command: +The user can just provide the subdir git url. An example command will look like this: ``` -kcl mod add --git --subdir +kcl mod add --git https://github.com/kcl-lang/modules/tree/main/argoproj --tag ``` -The `--subdir` flag will be optional. If the flag is not provided, `kpm` will download the entire repository as it does now. If the flag is provided, `kpm` will download only the specified subdirectory of the git repo. - -Example usage: - -``` -kcl mod add --git https://github.com/kcl-lang/modules --subdir add-certificates-volume -``` - -This command will download the `add-certificates-volume` subdirectory from the `modules` repository and append it in the subdir array of the `kcl.mod` file. +kpm would parse the git url and extract the subdirectory path using `GetPath()` function from github.com/kubescape/go-git-url package. It will then download the subdirectory and append it in the subdir array of the `kcl.mod` file. The `kcl.mod` file will look like this: @@ -33,7 +25,7 @@ bbb = { git = "https://github.com/kcl-lang/modules", commit = "ade147b", subdir The subdir is a list because in the future if user wants to add another subdir from the same git repo then it can be added without overwritting the current subdir. -## Design +## Design The path to the directory will be passed to `CloneOptions` in [pkg/git/git.go](https://github.com/kcl-lang/kpm/blob/d20b1acdc988f600c8f8465ecd9fe04225e19149/pkg/git/git.go#L19) as subDir. @@ -41,30 +33,8 @@ The path to the directory will be passed to `CloneOptions` in [pkg/git/git.go](h As mentioned in the [go-getter](https://pkg.go.dev/github.com/hashicorp/go-getter#readme-subdirectories) docs, we can append our subDir from `CloneOptions` (only if subDir is not empty) in `WithRepoURL` function. -### using go-git - -This process will involve using the `sparse-checkout` feature of git. - -1. Initialize a new git repository in the local `.kcl/kpm/` directory using [PlainInit](https://pkg.go.dev/github.com/go-git/go-git#PlainInit). The repository name will be the PackageName_version. - -2. Create a new worktree using [Worktree](https://pkg.go.dev/github.com/go-git/go-git/v5#Repository.Worktree) - -3. Enable the sparse-checkout feature using [SparseCheckout](https://pkg.go.dev/github.com/go-git/go-git/v5#Worktree.SparseCheckout). The second argument will be a slice of strings containing the subdirectory path. - -4. Add the remote repository using [AddRemote](https://pkg.go.dev/github.com/go-git/go-git/v5#Repository.CreateRemote) - -5. Pull the repository using [Pull](https://pkg.go.dev/github.com/go-git/go-git/v5#Worktree.Pull) - -Whenever we want to access the subdirectory using any command, we can refer to `kcl.mod` file of the project and iterate over the `subdir` array to get the path to the subdirectory. The `kcl.mod` file will automatically get updated whenever `kcl mod add` command is run. - -### Additional information - -1. To avoid creating a new root for each subdirectory download, I can add some check functions. - -2. The subdir flag is only for git options. If we pass it as a flag after oci, for example: `kpm add k8s --subdir 1.21/*`, it will not work. We can add a check [here](https://github.com/kcl-lang/kpm/blob/92158183556d39545bc0734a1e24284344ff3d9e/pkg/cmd/cmd_add.go#L154) that will give a warning if the subdir flag is passed. Furthermore, the subdir flag will only work for git repositories since it will insert the flag value into the field variable of the [Git](https://github.com/kcl-lang/kpm/blob/92158183556d39545bc0734a1e24284344ff3d9e/pkg/package/modfile.go#L375) struct. - ## References 1. https://medium.com/@marcoscannabrava/git-download-a-repositorys-specific-subfolder-ceeabc6023e2 -2. https://pkg.go.dev/github.com/go-git/go-git/v5 -3. https://pkg.go.dev/github.com/hashicorp/go-getter +2. https://pkg.go.dev/github.com/hashicorp/go-getter +3. https://pkg.go.dev/github.com/kubescape/go-git-url From cb6d79d04efd19096c80f1928b70dfcd01ee8b3e Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Thu, 6 Jun 2024 22:36:02 +0530 Subject: [PATCH 09/17] update sparse_checkout_asishkumar.md Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index cffdef32..57fcd81d 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -8,13 +8,13 @@ ## User Interface -The user can just provide the subdir git url. An example command will look like this: +The user can just provide the git url of the subdir they want to install. An example command will look like this: ``` kcl mod add --git https://github.com/kcl-lang/modules/tree/main/argoproj --tag ``` -kpm would parse the git url and extract the subdirectory path using `GetPath()` function from github.com/kubescape/go-git-url package. It will then download the subdirectory and append it in the subdir array of the `kcl.mod` file. +kpm would parse the git url and extract the subdirectory path using `GetPath()` function from github.com/kubescape/go-git-url subdir. It will then download the subdirectory and append it in the subdir array of the `kcl.mod` file. The `kcl.mod` file will look like this: @@ -27,11 +27,9 @@ The subdir is a list because in the future if user wants to add another subdir f ## Design -The path to the directory will be passed to `CloneOptions` in [pkg/git/git.go](https://github.com/kcl-lang/kpm/blob/d20b1acdc988f600c8f8465ecd9fe04225e19149/pkg/git/git.go#L19) as subDir. +The path to the directory will be passed to `CloneOptions` in [pkg/git/git.go](https://github.com/kcl-lang/kpm/blob/d20b1acdc988f600c8f8465ecd9fe04225e19149/pkg/git/git.go#L19) as subdir. -### using go-getter - -As mentioned in the [go-getter](https://pkg.go.dev/github.com/hashicorp/go-getter#readme-subdirectories) docs, we can append our subDir from `CloneOptions` (only if subDir is not empty) in `WithRepoURL` function. +As mentioned in the [go-getter](https://pkg.go.dev/github.com/hashicorp/go-getter#readme-subdirectories) docs, we can append our subdir from `CloneOptions` (only if subdir is not empty) in `WithRepoURL` function. ## References From 0192fc0ccf27fa1df61c5b6644b19b425fb0008c Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Thu, 6 Jun 2024 22:43:52 +0530 Subject: [PATCH 10/17] formatted document Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index 57fcd81d..a0c12390 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -29,7 +29,7 @@ The subdir is a list because in the future if user wants to add another subdir f The path to the directory will be passed to `CloneOptions` in [pkg/git/git.go](https://github.com/kcl-lang/kpm/blob/d20b1acdc988f600c8f8465ecd9fe04225e19149/pkg/git/git.go#L19) as subdir. -As mentioned in the [go-getter](https://pkg.go.dev/github.com/hashicorp/go-getter#readme-subdirectories) docs, we can append our subdir from `CloneOptions` (only if subdir is not empty) in `WithRepoURL` function. +As mentioned in the [go-getter](https://pkg.go.dev/github.com/hashicorp/go-getter#readme-subdirectories) docs, we can append our subdir from `CloneOptions` (only if subdir is not empty) in `WithRepoURL` function. ## References From f99daee38ffacea76cdee036923332e9016a23ba Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Sat, 6 Jul 2024 14:49:27 +0530 Subject: [PATCH 11/17] remove subdir Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index a0c12390..938e2ef5 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -14,17 +14,15 @@ The user can just provide the git url of the subdir they want to install. An exa kcl mod add --git https://github.com/kcl-lang/modules/tree/main/argoproj --tag ``` -kpm would parse the git url and extract the subdirectory path using `GetPath()` function from github.com/kubescape/go-git-url subdir. It will then download the subdirectory and append it in the subdir array of the `kcl.mod` file. +kpm would parse the git url and extract the subdirectory path using `GetPath()` function from github.com/kubescape/go-git-url subdir. The `kcl.mod` file will look like this: ``` [dependencies] -bbb = { git = "https://github.com/kcl-lang/modules", commit = "ade147b", subdir = ["add-ndots"]} +bbb = { git = "https://github.com/kcl-lang/modules", commit = "ade147b"} ``` -The subdir is a list because in the future if user wants to add another subdir from the same git repo then it can be added without overwritting the current subdir. - ## Design The path to the directory will be passed to `CloneOptions` in [pkg/git/git.go](https://github.com/kcl-lang/kpm/blob/d20b1acdc988f600c8f8465ecd9fe04225e19149/pkg/git/git.go#L19) as subdir. From 2e83c64086c0891e4019f04e0286df0a1a7bccc4 Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Mon, 19 Aug 2024 13:51:34 +0530 Subject: [PATCH 12/17] update with new changes Signed-off-by: Asish Kumar --- .../design-doc/sparse_checkout_asishkumar.md | 29 +++++++------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index 938e2ef5..eba1dc41 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -2,35 +2,28 @@ **Author**: Asish Kumar -## Abstract +## Solution -`kpm` manages third-party libraries through Git repositories, requiring a `kcl.mod` file at the root directory. It treats the entire Git repository as a single `kcl` package, which is inefficient for monorepos containing multiple `kcl` packages. Often, a `kcl` project depends on just one package within a monorepo, but `kpm` downloads the entire repository. Therefore, `kpm` needs to allow adding a subdirectory of a Git repository as a dependency, enabling it to download only the necessary parts and improve performance. +In order to add the feature of sparse-checkout, kpm will add the package specified in the kcl.mod file and later when running `kcl mod run` it will checkout that destination directory which contains that package recursively. ## User Interface -The user can just provide the git url of the subdir they want to install. An example command will look like this: +In order to use a specified package within a repository, the user will have to specify the package during the `kcl mod add` command. For example ``` -kcl mod add --git https://github.com/kcl-lang/modules/tree/main/argoproj --tag +kcl mod add --git https://github.com/officialasishkumar/check-kcl.git --commit 831fada36e155c8758f07f293c8267c869af69d3 --package k8s ``` -kpm would parse the git url and extract the subdirectory path using `GetPath()` function from github.com/kubescape/go-git-url subdir. +This command will recursively search for the package name within all the existing kcl.mod files in the repository and load it by specifying the package name in `kcl.mod` under `dependencies`. -The `kcl.mod` file will look like this: +The user can then run `kcl mod run` to run the code: -``` -[dependencies] -bbb = { git = "https://github.com/kcl-lang/modules", commit = "ade147b"} -``` +`kcl mod run` -## Design +This will checkout the destination directory which contains that package within the repository. -The path to the directory will be passed to `CloneOptions` in [pkg/git/git.go](https://github.com/kcl-lang/kpm/blob/d20b1acdc988f600c8f8465ecd9fe04225e19149/pkg/git/git.go#L19) as subdir. - -As mentioned in the [go-getter](https://pkg.go.dev/github.com/hashicorp/go-getter#readme-subdirectories) docs, we can append our subdir from `CloneOptions` (only if subdir is not empty) in `WithRepoURL` function. +## Design -## References +In order to use this feature, a new field `package` will be added to the `kcl.mod` file. This field will contain the package name that the user wants to use. -1. https://medium.com/@marcoscannabrava/git-download-a-repositorys-specific-subfolder-ceeabc6023e2 -2. https://pkg.go.dev/github.com/hashicorp/go-getter -3. https://pkg.go.dev/github.com/kubescape/go-git-url +Earlier the download only happens, in case of git, when there is a `kcl.mod` file in the root. Enabling this feature, will allow download of git repository even when there is no `kcl.mod` file in the root but this will only work if a package flag is passed. From 4de7d6935406a1f1ac5da889121f66bfc9374e98 Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Mon, 19 Aug 2024 15:13:47 +0530 Subject: [PATCH 13/17] update on package flag not provided Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index eba1dc41..22359ad3 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -16,6 +16,8 @@ kcl mod add --git https://github.com/officialasishkumar/check-kcl.git --commit 8 This command will recursively search for the package name within all the existing kcl.mod files in the repository and load it by specifying the package name in `kcl.mod` under `dependencies`. +The package flag is optional. If not provided kpm will work normally as before and you will need to have a `kcl.mod` file in the root of the repository. + The user can then run `kcl mod run` to run the code: `kcl mod run` From 19667bd814237339605593ff95e1823995ad2c93 Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Tue, 20 Aug 2024 13:21:27 +0530 Subject: [PATCH 14/17] final conclusion Signed-off-by: Asish Kumar --- .../design-doc/sparse_checkout_asishkumar.md | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index 22359ad3..985b22c3 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -16,16 +16,34 @@ kcl mod add --git https://github.com/officialasishkumar/check-kcl.git --commit 8 This command will recursively search for the package name within all the existing kcl.mod files in the repository and load it by specifying the package name in `kcl.mod` under `dependencies`. -The package flag is optional. If not provided kpm will work normally as before and you will need to have a `kcl.mod` file in the root of the repository. +The package flag is optional. If not provided kpm will work normally as before and you will need to have a `kcl.mod` file in the root of the repository. + +There is also an option to manually write the package flag in `kcl.mod` file. For example: + +``` +[dependencies] +agent = { git = "https://github.com/kcl-lang/modules.git", commit = "ee03122b5f45b09eb48694422fc99a0772f6bba8", package = "agent" } +``` + +This will work the same way as before only thing to note is you need to have a `kcl.mod` file in the root of the repository when running the command: + +``` +kcl mod add --git --commit +``` The user can then run `kcl mod run` to run the code: `kcl mod run` -This will checkout the destination directory which contains that package within the repository. +This will checkout the destination directory which contains that package within the repository. You can then use the loaded dependencies in your code. ## Design In order to use this feature, a new field `package` will be added to the `kcl.mod` file. This field will contain the package name that the user wants to use. Earlier the download only happens, in case of git, when there is a `kcl.mod` file in the root. Enabling this feature, will allow download of git repository even when there is no `kcl.mod` file in the root but this will only work if a package flag is passed. + + +# Implementation and conclusion + +This feature is implemented by [PR#453](https://github.com/kcl-lang/kpm/pull/453) and all the new tests are passing that check the verdict of the feature. The idea implemented in this PR was mentioned in https://github.com/kcl-lang/kpm/pull/335#issuecomment-2151338180. From 87c62be9a7ce7084e0c54d588c1f18e850d1025d Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Wed, 21 Aug 2024 12:43:58 +0530 Subject: [PATCH 15/17] update implementation and conclusion Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index 985b22c3..8dd32c9c 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -46,4 +46,9 @@ Earlier the download only happens, in case of git, when there is a `kcl.mod` fil # Implementation and conclusion -This feature is implemented by [PR#453](https://github.com/kcl-lang/kpm/pull/453) and all the new tests are passing that check the verdict of the feature. The idea implemented in this PR was mentioned in https://github.com/kcl-lang/kpm/pull/335#issuecomment-2151338180. +The idea implemented in the following PR was mentioned in https://github.com/kcl-lang/kpm/pull/335#issuecomment-2151338180. + +Here are the merged PRs: + +- https://github.com/kcl-lang/kpm/pull/453 +- https://github.com/kcl-lang/kpm/pull/457 From 5247cb1328cf5758f8416d5910c34f322b0ac131 Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Wed, 21 Aug 2024 18:39:10 +0530 Subject: [PATCH 16/17] add more details about other subcommands Signed-off-by: Asish Kumar --- .../design-doc/sparse_checkout_asishkumar.md | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index 8dd32c9c..96ab47a8 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -37,6 +37,24 @@ The user can then run `kcl mod run` to run the code: This will checkout the destination directory which contains that package within the repository. You can then use the loaded dependencies in your code. +The user can also run the following commands with package in there `kcl.mod` file: + +``` +kcl mod metadata +``` + +``` +kcl mod metadata --update +``` + +``` +kcl mod metadata --vendor +``` + +``` +kcl mod graph +``` + ## Design In order to use this feature, a new field `package` will be added to the `kcl.mod` file. This field will contain the package name that the user wants to use. @@ -52,3 +70,5 @@ Here are the merged PRs: - https://github.com/kcl-lang/kpm/pull/453 - https://github.com/kcl-lang/kpm/pull/457 + +The changes made are tested by unit tests. From 8c768c372c16b56096ff40f1610f75bec4fc22e7 Mon Sep 17 00:00:00 2001 From: Asish Kumar Date: Thu, 22 Aug 2024 15:08:55 +0530 Subject: [PATCH 17/17] kcl run Signed-off-by: Asish Kumar --- research/design-doc/sparse_checkout_asishkumar.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/research/design-doc/sparse_checkout_asishkumar.md b/research/design-doc/sparse_checkout_asishkumar.md index 96ab47a8..4625921b 100644 --- a/research/design-doc/sparse_checkout_asishkumar.md +++ b/research/design-doc/sparse_checkout_asishkumar.md @@ -31,10 +31,12 @@ This will work the same way as before only thing to note is you need to have a ` kcl mod add --git --commit ``` -The user can then run `kcl mod run` to run the code: +The user can then run `kcl mod run` or `kcl run` to run the code: `kcl mod run` +`kcl run` + This will checkout the destination directory which contains that package within the repository. You can then use the loaded dependencies in your code. The user can also run the following commands with package in there `kcl.mod` file: