[Design] KCL CLI UI and Workflow

[Peefy]

Summary

This document mainly provides the overall command line user interface of kcl lang, including the types and forms of commands, as well as how users can use the command line provided by kcl lang to complete scene functions.

2023.10.12 v0.1 Peefy
Last Update: 2023.10.17 v0.2 Peefy

Motivation

As shown in https://kcl-lang.io/docs/community/release-policy/kcl, we have already provided many languages, toolchain CLIs, and APIs related to KCL and KPM in advance, such as kcl, kcl-go, kcl-lint, kcl-fmt, kpm kcl-openapi, etc. Developer may be confused when to use them. We need to provide a unified CLI, API framework and implementation to maintain scalability, and provide unified documentation in v0.7.0.

Goals

• Unified user workflow: The primary goal is to establish a unified user workflow that caters to both KCL package producers and consumers. This means providing a consistent set of commands, options, and conventions that can be easily followed by users. The unified workflow should streamline the process of creating, managing, and using KCL packages.
• Seamless integration with KCL tools and APIs: The design should ensure seamless integration with existing KCL tools and APIs, such as kcl, kcl-go, kcl-lint, kcl-fmt, and kcl-openapi. Users should be able to leverage the command line interface to access and interact with these tools and APIs efficiently.
• Clear and comprehensive documentation: The design should facilitate the creation of clear and comprehensive documentation for the KCL command line interface.
• Scalability: The design should be scalable and adaptable to future updates and enhancements. It should accommodate new features, commands, and options that may be introduced in future versions of KCL. The goal is to ensure that the command line interface remains relevant and effective as the KCL ecosystem evolves.
• Consistency and standardization: The design should promote consistency and standardization across different KCL tools and languages.

Non-Goals

• Integration with non-KCL related tools: The focus of the KCL command line interface is to provide a seamless experience for KCL package producers and consumers. Integrating with tools or systems that are not directly related to KCL functionality is outside the scope of this design.
• Implementation-specific details: This design does not delve into the implementation details of individual KCL tools or APIs. While it may mention the existence of various tools, the specific implementation and technical details of each tool are considered outside the scope of this design.
• This design document does not consider the specific interaction process with the registry web page, such as the artifact hub. Users can complete the entire workflow independently through the kpm command.

Proposals

• Standardize command structure: Define a consistent structure for commands across different KCL tools and APIs. This includes specifying the format for command invocation, options, and arguments. The aim is to provide a clear and predictable command structure that is easy for users to understand and follow.

• Introduce a unified CLI tool: Develop a unified CLI tool, called "kcl", which serves as the entry point for accessing various KCL tools and functionalities. The kcl CLI will provide a set of high-level commands that cover common package management tasks, such as package creation, installation, publishing, and dependency management. This will simplify the user workflow and reduce the need for users to remember and use different command names for different tools.

• Implement comprehensive help and documentation: Include detailed help messages and documentation for each command, providing clear explanations of command usage, available options, and examples. The help messages should follow a consistent format and should be easily accessible to users when needed. This will enable users to quickly understand and utilize the capabilities of each command without relying on external documentation or resources.

• Enable extensibility: Design the CLI in a modular and extensible manner, allowing for the integration of additional tools, APIs, and functionality in the future. Provide clear guidelines and interfaces for extending the CLI, enabling developers to contribute new commands, plugins, or integrations. This will foster community involvement and allow the CLI to evolve and adapt to the changing needs of the KCL ecosystem.

• Support automation and scripting including GitHub action components, etc.

Design Details

KCL Tool

Language & Tooling Commands

kcl
   run           compile kcl package from a url or filepath
   build        build the kcl package
   check      check the current package, but don't build target files
   doc          documetation tool
   fmt          format tool
   lint           lint tool
   test          unit/integration/benchmark test tool
   lsp           language server tool
   clean       remove object files and cached files

Package & Registry Related Commands (mod and registry workspace)

kcl
   mod init            initialize new module in current directory
   mod search      search a command from regisry
   mod add           add new dependency
   mod remove     remove dependency
   mod update      update dependency
   mod pkg            package a kcl package into tar
   mod metadata  output the resolved dependencies of a package
   mod push          push kcl package to OCI registry.
   mod pull            pull kcl package from OCI registry.
   registry login    login to a registry
   registry logout  logout from a registry

Integration Commands

kcl
   import     migration other data and schema to kcl e.g., openapi, jsonschema, json, yaml
   export     convert kcl schema to other schema e.g., openapi

Plugin Commands (plugin workspace)

kcl
   plugin install     install one or more kcl command plugins
   plugin list        list installed command plugins
   plugin uninstall   uninstall one or more command plugins
   plugin update update one or more command plugins

Version and Help Commands

kcl
   help, h   Shows a list of commands or help for one command
   version Shows the command version

Note: To ensure smooth migration, we set kcl as the abbreviation for the kcl run command and kpm as the abbreviation/alias for the kcl mod subcommand.

alias kpm='kcl mod'

Subcommands Lifecycle

• The lifecycle of subcommands includes three stages: initialization, execution, and cleanup. During the initialization phase, some preparatory work can be carried out, such as checking dependencies, configuring the environment, etc. The execution phase is the main functional execution phase of subcommands, and after execution, some cleaning work may be required, such as closing resources, outputting results, etc. The lifecycle of subcommands is managed and controlled by the command-line framework.
• Issuing new subcommands can be achieved through a registration mechanism. Developers can register new subcommands into the command-line framework and provide corresponding information such as name, description, entry point, etc. The next time the command-line tool starts, the command-line framework will load and make new subcommands available.
• KCL defaults to providing functions that cannot use the subcommand line, as there are not many additional features for most language toolchains, and it is best to maintain a unified interface and workflow. For third-party command-line tools, KCL can be integrated.
• Core sub commands can not be used directly, and plugin commands can be used directly. The purpose of this selection is to minimize the number of tools used, while plugin commands are installed externally and can be built and used separately.
• Publish a subcommand means that we need to produce download links for binary compressed packages on various platforms (Windows, Mac, Linux), and maintain the plugin.yaml file in the subcommand Git repository to declare the download URL and version of plugin commands. The kcl plugin command can recognize the plugin. yaml file and install the corresponding plugin commands.

CLI Structure

Commands should follow a consistent structure of kcl [command] [options] [arguments]. Options should use standard conventions like short flags (e.g., -v for verbose) and long flags (e.g., --version).

• Examples

kcl
kcl run
kcl run <path/to/package>
kcl run <url>
kcl doc generate
kcl test
kcl mod push

The interface should provide informative and user-friendly error messages for incorrect command usage or failures. The command line interface should provide extensive inline help and documentation for each command, accessible through the following command

kcl help [command]

Help Message

The help information in a command line typically includes the following:

• Command functionality and syntax: Including the command name, the format of its parameters and the purpose and usage of the command.
• Option description: Listing the possible options of the command and their meanings.
• Examples: Providing usage examples of the command to help users understand how it is used.
• Notes: Indicating any limitations, caveats, or other relevant information about the command.
• Related commands: Listing other commands or functionalities related to the current command.
• More help: Providing ways to access additional help information, such as links to official documentation or other resources.

Environment Variables

• KCL_MODULE_ROOT = "~/.kcl/modules": Set an alternative location for KCL package location stored from the remote registry/hub.
• KCL_MODULE_CACHE_PATH = = "~/.kcl/modules/cache": Set an alternative location for storing cached files.
• KCL_MODULE_CONFIG_PATH = = "~/.kcl/modules/config.json": Set an alternative location for the user configuration information.
• KCL_DEFAULT_REGISTRY = "ghcr.io": Set the default package registry.
• KCL_DEFAULT_REPO = "kcl-lang": Set the default package repo.
• KCL_DEFAULT_OCI_PLAIN_HTTP = "OFF", Set an alternative switch for https/http

Unified SDK & API

The kcl-go, kpm, kcl-playground, and kcl-openapi repos will be unified into one kcl-go repo for the following reasons:

• kcl-go confuses the responsibilities of CLI and SDK, blurs the boundary between kpm and kcl go, and kpm has all the APIs of kcl-go, allowing users to use kpm instead of kcl-go and kpm.
• The language/tool capabilities provided by these repositories are not equal (kcl openAPI, kcl playground only assumes a small part of the unified CLI&API), and they are unidirectional dependencies without the need for abstract common components. Users generally do not use a particular repository alone.

CLI Framework

Design Detail is in #741, we won't go into detail here.

User Stories

Producer

As a KCL package producer, I want to be able to create a new KCL package easily using the command line interface. I should be able to specify the package name, version, and other relevant metadata during the creation process.

kcl mod init mypackage

As a KCL package producer, I want to be able to add dependencies to my KCL package using the command line interface. I should be able to specify the dependencies and their versions, and the CLI should handle the installation and management of these dependencies.

kcl mod add k8s

As a KCL package producer, I want to be able to test, lint and format my KCL package code using the command line interface. The CLI should provide commands to run linters and formatters on my package code, helping me maintain code quality and consistency.

• Format check

kcl fmt

kcl fmt --check

• Lint check

kcl mod lint

• Testing

kcl test

As a KCL package producer, I want to be able to publish my KCL package to a registry or repository using the command line interface. The CLI should handle the necessary authentication and package publishing process, allowing me to easily share my package with others.

kcl registry login & kcl mod push

Consumer

As a KCL package consumer, I want to be able to search for available KCL packages using the command line interface. The CLI should provide a command to search for packages based on keywords, allowing me to find relevant packages for my project.

kcl mod search

As a KCL package consumer, I want to be able to install KCL packages into my project using the command line interface. The CLI should provide a command to specify the package name and version, and handle the installation and dependency resolution process automatically.

• add dependency

kcl mod add k8s

• download the package

kcl mod pull k8s

As a KCL package consumer, I want to be able to update installed KCL packages to newer versions using the command line interface. The CLI should provide a command to check for package updates and handle the update process, ensuring that my project is using the latest versions of the packages.

kcl mod update [package]

As a KCL package consumer, I want to be able to generate documentation for installed KCL packages using the command line interface. The CLI should provide a command to generate documentation based on the package code and metadata, helping me understand and utilize the package's functionality.

kcl doc generate

Risks and Mitigations

• Comprehensive documentation: Develop clear and comprehensive documentation that explains the benefits, features, and usage of the KCL command line interface. Provide examples and tutorials that demonstrate its value and ease of use.
• Outreach and education: Conduct outreach activities to promote the KCL command line interface and its benefits among the KCL community.
• Seamless integration with existing workflows: Ensure that the KCL command line interface seamlessly integrates with existing development workflows and tools. Provide easy migration paths from existing tools and workflows to the KCL command line interface, highlighting the advantages and improved efficiency it offers. The current kcl command and kpm run command are consistent, and kcl run additionally extends OCI related capabilities, such as directly executing code from the remote end.

Alternatives

Language-specific CLIs

Language-specific CLIs: Instead of a unified command line interface, each tool (e.g., kcl-fmt, kcl-lint) could have its own dedicated CLI tool. This would allow for more specialization and tailored experiences for each language, but it may result in fragmentation and confusion for users who work with multiple KCL languages and tools. It would require users to learn and remember different commands and workflows for each language-specific CLI.

Integration with existing package managers

Integration with existing package managers: Rather than introducing a new CLI tool (kcl), the KCL functionality could be integrated with existing package managers like npm, pip, or Maven. This would leverage the familiarity and usage of these package managers and provide a unified experience for users who are already using these tools. However, it may introduce dependencies and limitations imposed by the existing package managers, potentially limiting the flexibility and extensibility of the KCL ecosystem.

Different CLI Workspaces

• Mixing subcommand spaces can lead to conflicts and confusion, especially in terms of command names or parameter options. To avoid conflicts, it is necessary to design good naming conventions and parameter selection rules. The command-line framework should provide a consistent parsing and validation mechanism to ensure that the parameters of each subcommand are correctly parsed and used.
• Choosing pm as the top-level command feature may be because package management is a core feature that involves tasks such as library installation, update, and uninstallation, which are typically considered the foundation of the entire toolchain.
• Designing the pm section as a subcommand of kcl pm is also a possible option. This design can better organize and group related commands, but may increase the nesting level and command length of the commands.

Tasks

• CLI Framework Design: [Design] KCL CLI Framework Design #741
• CLI Framework Impl: https://github.com/kcl-lang/cli
• Subcommand Integration: https://github.com/kcl-lang/cli
• Document: feat: add cli install scripts and v0.7 documents including new CLIs and registry modules. kcl-lang.io#193

[zong-zhe]

In the UI design, it is necessary to consider environment variables. Users may need to adjust kpm through environment variables in some scenarios. For example, the current commands do not provide a mechanism for adjusting the registry, so users may need to change the default registry through environment variables.

[zong-zhe]

Something I've considered in my previous job that might be useful

Considerations for deciding which functionalities to support through environment variables:

• Less frequently used features, such as changing the registry.
• Status-related functionalities, such as supporting HTTP protocol for the registry. I also considered adding "--quiet" in env, but due to its lower priority at the moment, it hasn't been implemented. We can consider it in the UI design.
• Default values for constants, such as the file path for saving downloaded packages or some binary.

An additional implementation consideration

• Add prefix to the environment variables with KPM_ or KCL_. Adding a prefix can effectively reduce the chances of conflicts with other environment variables.

[Peefy]

Issues that need further clarification

• What is the lifecycle of subcommands (Version Select, Install, Uninstall, ...)
• How to publish the new subcommand?
• Is there a precedent for kpm, which is both a function of pm and an entrance to a unified interface? Will there be conflicts between mixed sub command spaces? Will it have an impact in the future? Another way of thinking, why is the function of top level not fmt or lint, but pm? Why is the pm section not a subcommand similar to kclCtl pm<action>?
• Can sub commands be used directly?
• kcl-go mixes the responsibilities of SDK and CLI, and further clarification is needed.