diff --git a/README.md b/README.md index 1f1a939d..44eb6398 100644 --- a/README.md +++ b/README.md @@ -1,33 +1,154 @@ -# Talisman - -Talisman is a tool to validate code changes that are to be pushed out -of a local Git repository on a developer's workstation. By hooking -into the pre-push hook provided by Git, it validates the outgoing -changeset for things that look suspicious - such as potential SSH +
+

+ + Talisman

+
+

A tool to detect and prevent secrets from getting checked in

+ +[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT) [![Go Report Card](https://goreportcard.com/badge/thoughtworks/talisman)](https://goreportcard.com/report/thoughtworks/talisman) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/thoughtworks/talisman/issues) + + +## Table of Contents +- [What is Talisman?](#what-is-talisman) +- [Installation](#installation) + - [As a global hook template (Recommended)](#installation-as-a-global-hook-template) + - [To a single repository](#installation-to-a-single-project) + - [As a CLI to find file types](#installation-as-a-cli) +- [Talisman in action](#talisman-in-action) + - [Validations](#validations) + - [Ignoring files](#ignoring-files) +- [Uninstallation](#uninstallation) + - [From a global hook template](#uninstallation-from-a-global-hook-template) + - [From a single repository](#uninstallation-from-a-single-repository) +- [Contributing to Talisman](#contributing-to-talisman) + - [Developing locally](#developing-locally) + - [Releasing](#releasing) + +# What is Talisman? +Talisman is a tool that installs a hook to your repository to ensure that potential secrets or sensitive information do not leave the developer's workstation. + +It validates the outgoing changeset for things that look suspicious - such as potential SSH keys, authorization tokens, private keys etc. -The aim is for this tool to do this through a variety of means -including file names and file content. We hope to have it be an -effective check to prevent potentially harmful security mistakes from -happening due to secrets which get accidentally checked in to a -repository. -The implementation as it stands is very bare bones and only has the -skeleton structure required to add the full range of functionality we -wish to incorporate. However, we encourage folks that want to -contribute to have a look around and contribute ideas/suggestions or -ideally, code that implements your ideas and suggestions! +# Installation + +Talisman supports MAC OSX, Linux and Windows. -## Installation +Talisman can be installed and used in one of three different ways: -Talisman can either be installed and used in three different ways -1. As a git hook into a single git repository -2. As a git hook as a global [git hook template](https://git-scm.com/docs/git-init#_template_directory) +1. As a git hook as a global [git hook template](https://git-scm.com/docs/git-init#_template_directory) +2. As a git hook into a single git repository 3. As a CLI with the `--pattern` argument to find files -As a git hook, Talisman can be set up a as a pre-push or pre-commit hook on git repositories. +Talisman can be set up as either a pre-commit or pre-push hook on the git repositories. + +Find the instructions below. + +## [Recommended approach] +## Installation as a global hook template + +We recommend installing Talisman as a git hook template, as that will cause +Talisman to be present, not only in your existing git repositories, but also in any new repository that you 'init' or +'clone'. + +1. Run the following command on your terminal, to download and install the binary at $HOME/.talisman/bin + + As a pre-commit hook: + + ``` +curl --silent https://raw.githubusercontent.com/thoughtworks/talisman/master/global_install_scripts/install.bash > /tmp/install_talisman.bash && /bin/bash /tmp/install_talisman.bash +``` + + OR + + As a pre-push hook: + + ``` +curl --silent https://raw.githubusercontent.com/thoughtworks/talisman/master/global_install_scripts/install.bash > /tmp/install_talisman.bash && /bin/bash /tmp/install_talisman.bash pre-push +``` + +2. If you do not have TALISMAN\_HOME set up in your `$PATH`, you will be asked an appropriate place to set it up. Choose the option number where you set the profile source on your machine. + + + Remember to execute *source* on the path file or restart your terminal. +If you choose to set the `$PATH` later, please export TALISMAN\_HOME=$HOME/.talisman/bin to the path. -### Installation to a single project + +3. Choose a base directory where Talisman should scan for all git repositories, and setup a git hook (pre-commit or pre-push, as chosen in step 1) as a symlink. + This script will not clobber pre-existing hooks. If you have existing hooks, [look for ways to chain Talisman into them.] (#handling-existing-hooks) + + +### Handling existing hooks +Installation of Talisman globally does not clobber pre-existing hooks on repositories.
+If the installation script finds any existing hooks, it will only indicate so on the console.
+To achieve running multiple hooks we suggest (but not limited to) the following two tools + +#### 1. Pre-commit (Linux/Unix) +Use [pre-commit](https://pre-commit.com) tool to manage all the existing hooks along with Talisman. +In the suggestion, it will prompt the following code to be included in .pre-commit-config.yaml + +``` + - repo: local + hooks: + - id: talisman-precommit + name: talisman + entry: bash -c 'if [ -n "${TALISMAN_HOME:-}" ]; then ${TALISMAN_HOME}/talisman_hook_script pre-commit; else echo "TALISMAN does not exist. Consider installing from https://github.com/thoughtworks/talisman . If you already have talisman installed, please ensure TALISMAN_HOME variable is set to where talisman_hook_script resides, for example, TALISMAN_HOME=${HOME}/.talisman/bin"; fi' + language: system + pass_filenames: false + types: [text] + verbose: true +``` + +#### 2. Husky (Linux/Unix/Windows) +[husky](https://github.com/typicode/husky/blob/master/DOCS.md) is an npm module for managing git hooks. +In order to use husky, make sure you have already set TALISMAN_HOME to `$PATH`. + ++ **Existing Users** + + If you already are using husky, add the following lines to husky pre-commit in package.json + + ###### Windows + + ``` + "bash -c '\"%TALISMAN_HOME%\\${TALISMAN_BINARY_NAME}\" -githook pre-commit'" +``` + + ###### Linux/Unix + + ``` + $TALISMAN_HOME/talisman_hook_script pre-commit +``` ++ **New Users** + + If you want to use husky with multiple hooks along with talisman, add the following snippet to you package json. +###### Windows + + ``` + { + "husky": { + "hooks": { + "pre-commit": "bash -c '\"%TALISMAN_HOME%\\${TALISMAN_BINARY_NAME}\" -githook pre-commit'" && "other-scripts" + } + } + } +``` + + ###### Linux/Unix + + ``` + { + "husky": { + "hooks": { + "pre-commit": "$TALISMAN_HOME/talisman_hook_script pre-commit" && "other-scripts" + } + } + } +``` + + + +## Installation to a single project ```bash # Download the talisman binary @@ -41,26 +162,8 @@ cd my-git-project ~/install-talisman.sh ``` -### Installation as a CLI - -1. Download the Talisman binary from the [Releases page](https://github.com/thoughtworks/talisman/releases) corresponding to your system type -2. Place the binary somewhere (either directly in your repository, or by putting it somewhere in your system and adding it to your `$PATH`) -3. Run talisman with the `--pattern` argument (matches glob-like patterns, [see more](https://github.com/bmatcuk/doublestar#patterns)) -```bash -# finds all .go and .md files in the current directory (recursively) -talisman --pattern="./**/*.{go,md}" -``` - -### Installation as a global hook template (recommended) -We recommend installing it as a git hook template, as that will cause -Talisman to be present, not only in your existing git repositories, but also in any new repository that you 'init' or -'clone'. - -Use the [Global scripts Readme](global_install_scripts/Readme.md) to guide you through the installation process. - -### Installation - -#### Usage with the [pre-commit](https://pre-commit.com) git hooks framework +### Handling existing hooks +Talisman will need to be chained with any existing git hooks.You can use [pre-commit](https://pre-commit.com) git hooks framework to handle this. Add this to your `.pre-commit-config.yaml` (be sure to update `rev` to point to a real git revision!) @@ -74,9 +177,19 @@ a real git revision!) # - id: talisman-push ``` -## Talisman in action +## Installation as a CLI +1. Download the Talisman binary from the [Releases page](https://github.com/thoughtworks/talisman/releases) corresponding to your system type +2. Place the binary somewhere (either directly in your repository, or by putting it somewhere in your system and adding it to your `$PATH`) +3. Run talisman with the `--pattern` argument (matches glob-like patterns, [see more](https://github.com/bmatcuk/doublestar#patterns)) -After the installation is successful, Talisman will run checks for obvious secrets automatically before each push: +```bash +# finds all .go and .md files in the current directory (recursively) +talisman --pattern="./**/*.{go,md}" +``` + +# Talisman in action + +After the installation is successful, Talisman will run checks for obvious secrets automatically before each commit or push (as chosen during installation): ```bash $ git push @@ -86,7 +199,18 @@ The following errors were detected in danger.pem error: failed to push some refs to 'git@github.com:jacksingleton/talisman-demo.git' ``` -### Ignoring Files +## Validations +The following detectors execute against the changesets to detect secrets/sensitive information: + +* **Encoded values** - scans for encoded secrets in Base64, hex etc. +* **File content** - scans for suspicious content in file that could be potential secrets or passwords +* **File size** - scans for large files that may potentially contain keys or other secrets +* **Entropy** - scans for content with high entropy that are likely to contain passwords +* **Credit card numbers** - scans for content that could be potential credit card numbers +* **File names** - scans for file names and extensions that could indicate them potentially containing secrets, such as keys, credentials etc. + + +## Ignoring Files If you're *really* sure you want to push that file, you can add it to a `.talismanignore` file in the project root: @@ -124,6 +248,32 @@ At the moment, you can ignore * `filename` * `filesize` +# Uninstallation +The uninstallation process depends on how you had installed Talisman. +You could have chosen to install as a global hook template or at a single repository. + +Please follow the steps below based on which option you had chosen at installation. + +## Uninstallation from a global hook template +To uninstall talisman globally from your machine, run: + +``` +curl --silent https://raw.githubusercontent.com/thoughtworks/talisman/master/global_install_scripts/uninstall.bash > /tmp/uninstall_talisman.bash && /bin/bash /tmp/uninstall_talisman.bash +``` +This will + +1. ask you for the base dir of all your repos, find all git repos inside it and remove talisman hooks +2. remove talisman hook from .git-template +3. remove talisman from the central install location ($HOME/.talisman/bin).
+ +You will have to manually remove TALISMAN_HOME from your environment variables + +## Uninstallation from a single repository +When you installed Talisman, it must have created a pre-commit or pre-push hook (as selected) in your repository during installation. + +You can remove the hook manually by deleting the Talisman pre-commit or pre-push hook from .git/hooks folder in repository. + +# Contributing to Talisman ## Developing locally @@ -147,9 +297,8 @@ To build Talisman, we can use [gox](https://github.com/mitchellh/gox): Convenince scripts ```./build``` and ```./clean``` perform build and clean-up as mentioned above. -#### Contributing to Talisman -##### Releasing +## Releasing * Follow the instructions at the end of 'Developing locally' to build the binaries * Bump the [version in install.sh](https://github.com/thoughtworks/talisman/blob/d4b1b1d11137dbb173bf681a03f16183a9d82255/install.sh#L10) according to [semver](https://semver.org/) conventions diff --git a/global_install_scripts/Readme.md b/global_install_scripts/Readme.md deleted file mode 100644 index 9d539644..00000000 --- a/global_install_scripts/Readme.md +++ /dev/null @@ -1,92 +0,0 @@ -## Installation - -You can install Talisman as a global hook in your machine as a pre-commit or a pre-push hook, run: - -``` -curl --silent https://raw.githubusercontent.com/thoughtworks/talisman/master/global_install_scripts/install.bash > /tmp/install_talisman.bash && /bin/bash /tmp/install_talisman.bash -``` - -To install as pre-push hook, run: -``` -curl --silent https://raw.githubusercontent.com/thoughtworks/talisman/master/global_install_scripts/install.bash > /tmp/install_talisman.bash && /bin/bash /tmp/install_talisman.bash pre-push -``` - -This will -1. dowload the appropriate version of talisman for your machine and install it at $HOME/.talisman/bin -2. create a bash script talisman_hook_script at $HOME/.talisman/bin to run talisman -3. ask you for an appropriate place to set the TALISMAN\_HOME in path. TALISMAN\_HOME=$HOME/.talisman/bin [This should be the file which sets the profile source on your machine (.bash_Profile or .bashrc or .profile). Remember to execute 'source ' or restart your terminal. You could also choose to set the variable yourself later] -4. setup hook in .git-template (symlink to hook script at $HOME/.talisman/bin) - any new repo (git init OR git clone) will automatically get the hook -5. ask you for the base dir of all your repos, find all git repos inside it and setup hooks (as symlink) -This script will not clobber pre-existing hooks - -#### Handling existing hooks -Installation of Talisman globally does not clobber pre-existing hooks on repositories. -If the installation script finds any existing hooks, it will only indicate so on the console. -To achieve running multiple hooks we suggest the following two tools - -##### 1. Pre-commit (Linux/Unix) - 1. use [pre-commit](https://pre-commit.com) tool to manage all the existing hooks along with Talisman. - 2. In the suggestion, it will prompt the following code to be included in .pre-commit-config.yaml -``` - - repo: local - hooks: - - id: talisman-precommit - name: talisman - entry: bash -c 'if [ -n "${TALISMAN_HOME:-}" ]; then ${TALISMAN_HOME}/talisman_hook_script pre-commit; else echo "TALISMAN does not exist. Consider installing from https://github.com/thoughtworks/talisman . If you already have talisman installed, please ensure TALISMAN_HOME variable is set to where talisman_hook_script resides, for example, TALISMAN_HOME=${HOME}/.talisman/bin"; fi' - language: system - pass_filenames: false - types: [text] - verbose: true -``` - -##### 2. Husky (Linux/Unix/Windows) - 1. [husky](https://github.com/typicode/husky/blob/master/DOCS.md) is a npm module for managing git hooks. - 2. In order to use husky, make sure you SET TALISMAN_HOME. - - ##### Existing Users - If you already are using husky, add the following lines to husky pre-commit in package.json -###### Windows -``` - "bash -c '\"%TALISMAN_HOME%\\${TALISMAN_BINARY_NAME}\" -githook pre-commit'" -``` -###### Linux/Unix -``` - $TALISMAN_HOME/talisman_hook_script pre-commit -``` -##### New Users -If you want to use husky with multiple hooks along with talisman, add the following snippet to you package json. - - ###### Windows -``` - { - "husky": { - "hooks": { - "pre-commit": "bash -c '\"%TALISMAN_HOME%\\${TALISMAN_BINARY_NAME}\" -githook pre-commit'" && "other-scripts" - } - } - } -``` - ###### Linux/Unix -``` - { - "husky": { - "hooks": { - "pre-commit": "$TALISMAN_HOME/talisman_hook_script pre-commit" && "other-scripts" - } - } - } -``` - - -## Uninstallation -To uninstall talisman globally from your machine, run: -``` -curl --silent https://raw.githubusercontent.com/thoughtworks/talisman/master/global_install_scripts/uninstall.bash > /tmp/uninstall_talisman.bash && /bin/bash /tmp/uninstall_talisman.bash -``` -This will -1. ask you for the base dir of all your repos, find all git repos inside it and remove talisman hooks -2. remove talisman hook from .git-template -3. remove talisman from the central install location ($HOME/.talisman/bin) -You will have to manually remove TALISMAN_HOME from your environment variables - -