V1.0.0 (#132)

* Add connection configuration documentation (#131) * Add new configuration document for connection abuseipdb (#94) * Add new configuration document for connection alicloud (#95) * Add new configuration document for connection azure (#97) * Add new configuration document for connection bitbucket (#98) * Add new configuration document for connection clickup (#99) * Add new configuration document for connection datadog (#100) * Add new configuration document for connection discord (#101) * Add new configuration document for connection freshdesk (#102) * Add new configuration document for connection github (#103) * Add new configuration document for connection gitlab (#104) * Add new configuration document for connection ip2locationio (#105) * Add new configuration document for connection ipstack (#106) * Add new configuration document for connection jira (#107) * Add new configuration document for connection jumpcloud (#108) * Add new configuration document for connection mastodon (#109) * Add new configuration document for connection microsoft_teams (#110) * Add new configuration document for connection okta (#111) * Add new configuration document for connection opsgenie (#112) * Add new configuration document for connection openai (#113) * Add new configuration document for connection pagerduty (#114) * Add new configuration document for connection sendgrid (#115) * Add new configuration document for connection servicenow (#116) * Add new configuration document for connection slack (#117) * Add new configuration document for connection turbot_guardrails (#118) * Add new configuration document for connection turbot_pipes (#119) * Add new configuration document for connection uptime_robot (#120) * Add new configuration document for connection urlscan (#121) * Add new configuration document for connection vault (#122) * Add new configuration document for connection virus_total (#123) * Add new configuration document for connection zendesk (#124) * Add new configuration document for connection pagerduty (#125) * Add new configuration document for connection aws (#96) * Add new configuration document for connection gcp (#128) * Add new configuration document for connection slack (#127) * Create Index and Template for connection documents (#129) * add .gitignore * add connection_import docs, update sidebar for connection, connnection_import reference docs * add deprecation warnings for credentials * change references, examples from "credential" to "connection". Updated run/connection examples with new connection type syntax * update build/mod-variables for new connection, notifier types and enums * update flowpipe hcl variable, param reference for new connection, notifier types and enums * update example in flowpipe-hcl/variable for new capabilities-- connection type and enum * added exit codes to cli docs * add new db connection types, update docs for mod database args (WIP) * updates for connection docs for duckdb, postgres, sqlite, steampipe * add new connections to sidebar * updates for connection docs for duckdb, postgres, sqlite, steampipe * updates for connection docs for duckdb, postgres, sqlite, steampipe * change references to version 0.10.0 -> 1.0.0 * add mysql connection docs, update docs for other db connections --------- Co-authored-by: Omero Saienni <[email protected]>
turbot · Oct 22, 2024 · 157e600 · 157e600
1 parent 0039f11
commit 157e600
Show file tree

Hide file tree

Showing 91 changed files with 2,497 additions and 186 deletions.
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,55 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+/output
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# local env files
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+# Editor cache and lock files
+*.swp
+*.swo
+
+# vercel
+.vercel
+
+# Jetbrains
+.idea
+
+# Auto-generated robots.txt and sitemap.xml
+/public/robots.txt
+/public/sitemap.xml
+/public/blog/feed.xml
+/public/changelog/feed.xml
+/public/images/docs
+/public/images/reference_examples
+/steampipe-docs-local
+/tmp
+content/docs
+
+.vscode
diff --git a/docs/build/input.md b/docs/build/input.md
@@ -7,7 +7,7 @@ sidebar_label: Ask for Input
 
 Flowpipe allows you to optimize and automate your operational and business processes, but there are times when human interaction is required.  For example, a manager may be required to approve a privilege escalation request.  Furthermore, we live in an era of constant communication across multiple channels.  People prefer to use specific tools (Email, Slack, etc.) for communication and expect these interactions to occur in their tools.  The Flowpipe `input` step primitive provides the ability to prompt for human input via multiple channels and wait for a response.
 
-To interact with the various tools and services, Flowpipe also includes an `integration` configuration resource.  This allows you to configure Flowpipe for 2-way communication with your tool (Slack, email, etc).  `integration` is a configuration resource (defined in `.fpc` files) like `credential`, thus the configuration is installation level, not mod level; you can define the integrations for your installation once and refer to them from any mods.
+To interact with the various tools and services, Flowpipe also includes an `integration` configuration resource.  This allows you to configure Flowpipe for 2-way communication with your tool (Slack, email, etc).  `integration` is a configuration resource (defined in `.fpc` files) like `connection`, thus the configuration is installation level, not mod level; you can define the integrations for your installation once and refer to them from any mods.
 
 Sending notifications is a common pattern, and often users will want to route a request to more than one user, group, or channel, and via more than one delivery mechanism.  For instance, you may want to request approval via Slack AND email.  The `notifier` resource allows you to define a list of integrations to send notifications to. Like `integration`, a `notifier` is an installation-level configuration resource.
 

diff --git a/docs/build/mod-variables.md b/docs/build/mod-variables.md
@@ -12,36 +12,153 @@ sidebar_label: Pass Variables
 ##  Input Variables
 
 ### Defining Input Variables
-Flowpipe mods support input variables that are similar to [Terraform input variables](https://www.terraform.io/docs/language/values/variables.html):
-
-You declare them with a `variable` block:
+Flowpipe mods support input variables that are similar to [Terraform input variables](https://www.terraform.io/docs/language/values/variables.html). You declare them with a `variable` block:
 ```hcl
 variable "instance_id" {
   type = string
 }
+```
+
+The [type](#types) is optional, though it is generally recommended that you provide the type explicitly to provide clarity to the user and to avoid unexpected run time errors due to unforeseen type coercions.
 
+You can optionally define a `default` value and `description` for the variable.  
+
+```hcl
 variable "mandatory_tag_keys" {
   type        = list(string)
   description = "A list of mandatory tag keys to check for (case sensitive)."
   default     = ["Environment", "Owner"]
 }
+```
+
+If a variable does not define a `default`, the user will be prompted for its value when flowpipe starts.  If a variable defines a `default`, the user is not prompted and the default value is used if the variable is not explicitly set.
+
+You may also provide a `description` of the variable.  The description helps to provide information about the intent and format of the variable to the user of the mod.  The description is included when the user is prompted for a variable's value.
+
+
 
+#### Types
+
+Flowpipe supports the standard HCL `string`, `number`, and `bool` type primitives.
+
+```hcl
+variable "my_string" {
+  type = string
+}
+
+variable "counter" {
+  type = number
+}
+
+variable "force" {
+  type = bool
+}
+```
+
+
+The keyword `any` may be used to indicate that any type is acceptable.  If no type is defined, it is assumed to be `any`:
+
+```hcl
+variable "myvar" {
+  type = any
+}
+```
+
+
+Collection types (`list`, `map`, `set`, `object`) may also be used:
+
+```hcl
+variable "people" {
+  type    = list(string)
+  default = ["Kramer", "Elaine", "George"]
+}
+
+variable "employers" {
+  type    = map(string)
+
+  default = {
+    Kramer = "Kramerica Industries"
+    Elaine = "J. Peterman, Inc"
+    George = "Vandelay Industries"
+  }
+}
+```
+
+You can even create deeply nested types:
+
+```hcl
+variable "base_tag_rules" {
+  type = object({
+    add           = optional(map(string))
+    remove        = optional(list(string))
+    remove_except = optional(list(string))
+    update_keys   = optional(map(list(string)))
+    update_values = optional(map(map(list(string))))
+  })
+  description = "Base rules to apply to resources unless overridden when merged with any provided resource-specific rules."
+  default     = {
+    add           = {}
+    remove        = []
+    remove_except = []
+    update_keys   = {}
+    update_values = {}
+  }
+}
+
+```
+
+In addition to the standard `string`, `number`, and `bool` primitives, Flowpipe `connection` and `notifier` config primitives may be specified. 
+
+```hcl
+variable "conn" {
+  type = connection
+}
+
+variable "approver" {
+  type = notifier
+}
+```
+
+You can even specify a variable as a specific *type* of connection:
+```hcl
+variable "aws_conn" {
+  type    = connection.aws
+  default = connection.aws.default
+}
+
+variable "github_conn" {
+  type    = connection.github
+  default = connection.github.default
+}
+```
+
+As with any type, collections may also be defined:
+
+```hcl
+variable "aws_connections" {
+  type    = list(connection.aws)
+  default = [connection.aws.default]
+}
+
+variable "approvers" {
+  type    = list(notifier)
+  default = ["default"]
+}
+```
+
+
+#### Enum
+
+Flowpipe supports an `enum` argument to provide a set of allowed values; no other values are allowed.
+
+```hcl
+variable "log_level" {
+  type    = string
+  default = "info"
+  enum    = ["emerg", "alert", "crit", "err", "warning", "notice", "info", "debug" ]
+}
 ```
 
-You can optionally define:
-- `default` - A default value.  If no value is passed, the user is not prompted and the default is used.
-- `type` - The data type of the variable.  This may be a simple type or a collection.
-  - The supported type primitives are:
-    - `string`
-    - `number`
-    - `bool`
-  - Collections types may also be used:
-    - `list(<TYPE>)`
-    - `set(<TYPE>)`
-    - `map(<TYPE>)`
-    - `object({<ATTR NAME> = <TYPE>, ... })`
-  - The keyword `any` may be used to indicate that any type is acceptable 
-- `description` - A description of the variable.  This text is included when the user is prompted for a variable's value.
 
 ### Using Input Variables
 Variables may be referenced as `var.<NAME>`.  Variables are often used as defaults for pipeline parameters:
@@ -75,6 +192,12 @@ When running Flowpipe, you can pass variables in several ways.  You can pass ind
 flowpipe pipeline run list_org_workspaces --var=org="my_org"
 ```
 
+For `connection` or `notifier` variables, pass them by name:
+```bash
+flowpipe pipeline run list_org_workspaces --var conn=connection.aws.prod --var approver=notifier.admins
+```
+
+
 When passing list variables, they must be enclosed in single quotes:
 
 ```bash

diff --git a/docs/flowpipe-hcl/mod.md b/docs/flowpipe-hcl/mod.md
@@ -41,10 +41,17 @@ mod "aws" {
 ## Example - Composite Mod with Dependency
 
 ```hcl
+variable "database" {
+  type        = connection.steampipe
+  description = "Steampipe database connection string."
+  default     = connection.steampipe.default
+}
+
 mod "deactivate_expired_aws_iam_access_keys" {
   title       = "Deactivate expired AWS IAM keys"
   description = "Deactivates AWS IAM keys that have been active for a certain period of time."
-
+  database    = var.database
+  
   require {
     flowpipe {
       min_version = "0.1.0"
@@ -68,7 +75,8 @@ mod "deactivate_expired_aws_iam_access_keys" {
 | Name | Type | Required? | Description
 |-|-|-|-
 | `categories` | List(String) | Optional | A list of labels, used to categorize mods (such as on the Flowpipe Hub).
-| `color` | String |Optional |  A hexadecimal RGB value to use as the color scheme for the mod on hub.flowpipe.io.  
+| `color`      | String | Optional |  A hexadecimal RGB value to use as the color scheme for the mod on hub.flowpipe.io.  
+| `database`    | String | Optional | The database to use as the default for [query steps](/docs/flowpipe-hcl/step/query) and [query triggers](/docs/flowpipe-hcl/trigger/query). The `database` may be a connection reference (`connection.steampipe.default`), a connection string (`postgres://[email protected]:9193/steampipe`), or a Pipes workspace (`acme/anvils`). If not set, the default is the local Steampipe database instance.
 | `description` |  String | Optional | A string containing a short description. 
 | `documentation` | String (Markdown)| Optional | A markdown string containing a long form description, used as documentation for the mod on hub.flowpipe.io. 
 | `icon` |  String | Optional | The URL of an icon to use for the mod on hub.flowpipe.io.

diff --git a/docs/flowpipe-hcl/pipeline.md b/docs/flowpipe-hcl/pipeline.md
@@ -57,6 +57,7 @@ pipeline "get_astronauts" {
 ## Parameters
 
 One or more `param` blocks may optionally be used in a pipeline to define parameters that the pipeline accepts. 
+
 ```hcl
 param "url" {
   type    = string
@@ -79,8 +80,9 @@ step "http" "whos_in_space" {
 |---------------|---------|--------------------------
 | `default`     | Any     | A value to use if no argument is passed for this parameter when the query is run.
 | `description` | String  | A description of the parameter.
+| `enum`        | Set     | A set of allowed values for the param; no other values are allowed.
 | `tags`         | Map    | A map of key:value metadata for the benchmark, used to categorize, search, and filter.  The structure is up to the mod author and varies by benchmark and provider. 
-| `type`        | String   | The data type of the parameter: `string`, `number`, `bool`, `list`, `map`, `any` (default `any`). 
+| `type`        | String   | The data type of the parameter: a primitive type `string`, `number`, `bool`, `connection`, `connection.{type}`, `notifier`, or a collection type `list()`, `map()`, or `any` (default `any`). 
 
 ----