-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[LFX Term 3]: Full storage path for KCL third-party dependencies #384
Comments
PreTest
|
HI @zong-zhe @Peefy I am excited to contribute to this issue. After going through the solution mentioned I have see following key take-aways:
In order to proceed forward with the pretest where I need to expand kpm's ability to download , chekout from a git bare repo. Which part of documentation I can refer for configuring this for |
Hi @Manoramsharma 😃 Welcome! Let me provide more details.
kpm and git
kpm and go-getter |
Okay, so Now for our case, development of
@zong-zhe am I thinking in a right direction? |
Proposal to Enhance kpm's current git moduleTo enhance the kpm current git module, I propose the following solutions to expand its capabilities: 1. Expand Ability to Download Git Bare Repositories1.1 Define a Function for Cloning Bare RepositoriesI will add a function to clone Git repositories as bare repositories. This function will use Git commands to perform the cloning and store the bare repository in the appropriate cache directory. Example Implementation: package git
import (
"fmt"
"os/exec"
"path/filepath"
)
// CloneBareRepo clones a Git repository as a bare repository.
func CloneBareRepo(repoURL string, cacheDir string) (string, error) {
repoHash := calculateHash(repoURL) // Function to calculate a unique hash for the repo URL
repoName := extractRepoName(repoURL) // Function to extract repository name from URL
cachePath := filepath.Join(cacheDir, fmt.Sprintf("%s-%s", repoName, repoHash))
if _, err := exec.LookPath("git"); err != nil {
return "", fmt.Errorf("git not found in PATH")
}
if err := exec.Command("git", "clone", "--bare", repoURL, cachePath).Run(); err != nil {
return "", fmt.Errorf("error cloning repository: %v", err)
}
return cachePath, nil
}
// Helper functions
func calculateHash(url string) string {
// Implement hashing function for URL
}
func extractRepoName(url string) string {
// Implement repo name extraction
} 1.2 Integrate with the Unified Dependency SystemI will ensure that this new function integrates with the unified dependency system. This may involve updating the system to manage and utilize the cache effectively. package unified
// UnifiedDependencySystem is an abstraction for managing dependencies.
type UnifiedDependencySystem struct {
cacheDir string
}
// Get handles fetching the Git repository and managing the cache.
func (uds *UnifiedDependencySystem) Get(repoURL string, commitID string, targetDir string) error {
bareRepoPath, err := git.CloneBareRepo(repoURL, uds.cacheDir)
if err != nil {
return err
}
return git.CheckoutFromBareRepo(bareRepoPath, commitID, targetDir)
} 2. Expand Ability to Checkout Source Code from Bare Repositories2.1 Define a Function for Checking Out Code from Bare RepositoriesI will add a function to check out a specific commit or branch from a bare repository. Example Implementation: package git
import (
"fmt"
"os"
"os/exec"
"path/filepath"
)
// CheckoutFromBareRepo checks out a specific commit from a bare repository.
func CheckoutFromBareRepo(bareRepoPath string, commitID string, targetDir string) error {
if err := os.MkdirAll(targetDir, 0755); err != nil {
return fmt.Errorf("error creating target directory: %v", err)
}
cmd := exec.Command("git", "worktree", "add", targetDir, commitID)
cmd.Dir = bareRepoPath
if err := cmd.Run(); err != nil {
return fmt.Errorf("error checking out commit: %v", err)
}
return nil
} 2.2 Update the Unified Dependency SystemI will ensure that the unified dependency system correctly utilizes the new checkout functionality and integrates with the local cache. package unified
// UnifiedDependencySystem is an abstraction for managing dependencies.
type UnifiedDependencySystem struct {
cacheDir string
}
// Get handles fetching and checking out the Git repository.
func (uds *UnifiedDependencySystem) Get(repoURL string, commitID string, targetDir string) error {
bareRepoPath, err := git.CloneBareRepo(repoURL, uds.cacheDir)
if err != nil {
return err
}
return git.CheckoutFromBareRepo(bareRepoPath, commitID, targetDir)
} 3. Handle Cache and Metadata Management3.1 Implement Cache ManagementI will add functionality to manage and update cached repositories, including metadata. package cache
// CacheManager handles operations related to caching dependencies.
type CacheManager struct {
cacheDir string
}
// UpdateCache updates the local cache with new metadata and repositories.
func (cm *CacheManager) UpdateCache(repoURL string) error {
// Implement caching logic here
return nil
} 3.2 Ensure ConsistencyI will ensure that the local storage system (e.g., |
Hi @Manoramsharma 😃 You might need to learn if |
You may need to add some detail here, because KCL's dependencies are not only those from git, you also need to consider the dependencies from oci together, abstracting them from the same behavior. You also need to consider combining the downloader to support the download behavior of different dependencies. https://github.com/kcl-lang/kpm/blob/main/pkg/downloader/downloader.go |
This part I think may not need to be a separate module, it can be used as part of the unified file system. |
Hi @zong-zhe, I have carried out some research and found that But our current git implementation includes Should I utilize these functions to extend our git implementaion for kpm adding support to clone bare repository? I need your reviews on this. |
Hi @Manoramsharma 😃 I reviewed your previous work, and I think that you may need to adjust some of your work.
|
@zong-zhe are we talking of something similar to the Or I can add an extra |
Hi @Manoramsharma 😃 What I mean is that the |
implemented it in #418 cc @zong-zhe |
Hi @zong-zhe you can review my latest PR where I removed the Also I have one question regarding the checkout operation from the bare repo, the bare is something with no workingtree so how it is possible to checkout the particular commit or branch from the repository with no workingtree? |
Hi @Manoramsharma 😃 This part of my description is inaccurate, which has caused a misunderstanding. I apologize for it. My original idea was to use the bare repository as a cache, then clone different dependencies into the corresponding directory from the bare repo and checkout to the corresponding branch, commit or tag. |
Hi @zong-zhe after some research about the bare repository and its implementation as the cache support for our objective I have designed an overall proposal discussing the approach. Before moving forward with the exact changes to the codebase I have few questions in my mind:
Looking forward to have your feedback on the same. Thanks for your support !! |
Hi @Manoramsharma 😃
kpm supports adding dependencies from git repositories, which are specified by the user.
kpm's git capability is directly inherited from the locally installed git client. As long as
I don't understand the question. |
Hi @Gmin2 I think you are not following the issue quite well. I have already designed an overall roadmap for achieving this particular enhancement of KPM to manage third-party dependencies based on caching package repositories as bare repo to the local path. All discussions including the overall approach to achieve a unified dependency management to low-level works in the form of pretest are already included in my proposal PR . You can have a look at the proposal here #423 . I am consistently working on the proposal refining and achieving the expected outcome by taking regular reviews from the maintainers on my work. I don't think you need to invest your time on this particular issue as I have already made a good progress on this. Thanks |
designing was not part of the pretest, it was not mentioned in the pretest i thought we had to give the design in the cover letter |
1. Problem
2. Solution
In order to solve the problem, we redesigned the storage local system of the third-party dependencies with reference to rust
2.1 Git
Under the subdir in
kpm/git
, the storage local system isNOTE:
<NAME>-<HASH>
<NAME>
is the name of git repo,<HASH>
is calculated by the git url.<HASH>
is mainly used to distinguish between different git repositories, such as github and gitlab, and the reason for using a hash value is mainly to mask the inconsistency of the symbols in the url and local path.2.2 Oci
Under the subdir in
kpm/oci
, the storage local system is4. Implementation
The text was updated successfully, but these errors were encountered: