From d9427e05e6ed41546e1318c2838cbcd9ceeb810a Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Fri, 22 Nov 2024 18:05:56 -0500 Subject: [PATCH 1/2] Concept: Pipelines and Command Lists --- concepts/pipelines/.meta/config.json | 8 ++ concepts/pipelines/about.md | 1 + concepts/pipelines/introduction.md | 180 +++++++++++++++++++++++++++ concepts/pipelines/links.json | 10 ++ config.json | 5 + 5 files changed, 204 insertions(+) create mode 100644 concepts/pipelines/.meta/config.json create mode 100644 concepts/pipelines/about.md create mode 100644 concepts/pipelines/introduction.md create mode 100644 concepts/pipelines/links.json diff --git a/concepts/pipelines/.meta/config.json b/concepts/pipelines/.meta/config.json new file mode 100644 index 00000000..9a53fe0d --- /dev/null +++ b/concepts/pipelines/.meta/config.json @@ -0,0 +1,8 @@ +{ + "authors": [ + "glennj" + ], + "contributors": [ + ], + "blurb": "Compose more complex bash commands with pipelines and command lists" +} diff --git a/concepts/pipelines/about.md b/concepts/pipelines/about.md new file mode 100644 index 00000000..46409041 --- /dev/null +++ b/concepts/pipelines/about.md @@ -0,0 +1 @@ +# TODO diff --git a/concepts/pipelines/introduction.md b/concepts/pipelines/introduction.md new file mode 100644 index 00000000..0bfc1f78 --- /dev/null +++ b/concepts/pipelines/introduction.md @@ -0,0 +1,180 @@ +# Pipelines and Command Lists + +We have seen how to write simple commands, where the command is followed by arguments. +Now we will see how to make more complex commands by composing simple commands. + +## Pipelines + +This is one of the "killer features" of shell programming. +Pipelines allow you create sophisticated transformations on a stream of text. + +To produce a sorted list of users: + +```bash +cat /etc/passwd | cut -d : -f 1 | sort +``` + +The pipe symbol (`|`) connects the output of one command to the input of another. +`cut` reads the output of `cat`, and `sort` reads the output of `cut`. + +~~~~exercism/advanced +* By default, each command in a pipeline runs in a separate subshell. + (A subshell is a child process that is a copy of the currently running shell.) + +* All the commands in a subshell execute in parallel. + +* There is a performance cost to running pipelines. + If you find yourself with long pipelines of similar commands, consider combining them in a single command. + For example, pipelines using multiple instances of grep+cut+sed+awk+tr can be combined into a single awk command for efficiency. + +* The exit status of a pipeline is the exit status of the last command in the pipeline. + However, there is a shell setting that can control this. + The "pipefail" setting (enabled with `set -o pipefail`) will use the _**last** non-zero exit status_ of the pipeline's commands as the pipeline's exit status, unless all commands succeeded. +~~~~ + +## Command Lists + +A command list is a sequence of pipelines separated by `;` (or newline), `&&` or `||`. + +* `A ; B` is a command list where `B` executes after `A` has completed. +* `A && B`, where `B` executes only if `A` succeeds (exits with status zero). +* `A || B`, where `B` executes only if `A` fails (exits with status non-zero). + +The exit status of a command list is the exit status of the last command that executes. + +The `&&` and `||` operators can be chained so that the next command conditionally executes based on the status of the preceding commands. +For example + +```bash +A && B && C && D || E +``` + +* B executes if A succeeds, +* C executes if A and B succeed, +* D executes if A and B and C succeed, +* E executes if **any** of A, B, C or D fails. + +~~~~exercism/caution +Use these logical operators sparingly. +They can quickly lead to unreadable code, or logic that is hard to comprehend. + +For example, do you think these are the same? + +```bash +if A; then B; else C; fi +``` + +```bash +A && B || C +``` + +The difference is: when does C execute? + +* In the first snippet, the if statement, C will only execute if A fails. +* In the second snippet, C executes if A fails _or if A succeeds but B fails_! +~~~~ + +### Uses of Command Lists + +Here are a couple of examples where command lists can simplify bash code. + +#### Reading blocks of lines from a file + +Suppose you have a data file that represents points of a triangle as the length of the three sides but each on a separate line. + +```bash +$ cat triangle.dat +3 +4 +5 +9 +12 +14 +``` + +You can use a while loop where the condition is three separate read commands: + +```bash +while read a && read b && read c; do + if is_pythagorean "$a" "$b" "$c"; then + echo "$a:$b:$c is pythagorean" + else + echo "$a:$b:$c is not pythagorean" + fi +done < triangle.dat +``` + +Assuming `is_pythagorean` is a command that determines if the three sides satisfy the Pythagoran equation, the output would be + +```none +3:4:5 is pythagorean +9:12:14 is not pythagorean +``` + +#### Assertions + +Many programming languages have a form of assertion where an exception is thrown if some condition fails + +``` +assert(x == 5, "x must be 5"); +``` + +We can use an OR operator in bash to simulate that function: + +```bash +die () { + echo "$*" >&2 + exit 1 +} + +[[ $x -eq 5 ]] || die "x must be equal to 5" +[[ $y -gt 5 ]] || die "y must be greater than 5" +``` + +## Style Considerations + +Long command lists become hard to read quite quickly. +Liberal use of newlines can help a lot. + +Here's an example: a word is added to an array if two conditions are met + +```bash +[[ "$word" != "$topic" ]] && [[ "$key" == "$(sorted "$topic")" ]] && anagrams+=("$candidate") +``` + +Bash allows you to add a newline after a pipe or a logical operator. + +```bash +[[ "$word" != "$topic" ]] && +[[ "$key" == "$(sorted "$topic")" ]] && +anagrams+=("$candidate") +``` + +But the operator kind of gets lost at the end of the line. +Using a _line continuation_ means you can put the operator first, which makes it more obvious that the list is being continued: + +```bash +[[ "$word" != "$topic" ]] \ +&& [[ "$key" == "$(sorted "$topic")" ]] \ +&& anagrams+=("$candidate") +``` + +~~~~exercism/note +A _line continuation_ is the two character sequence "backslash"+"newline". +When bash sees that sequence, it is simply removed from the code, thereby _continuing_ the current line with the next line. +Take care to not allow any spaces between the backslash and the newline. +~~~~ + +Here's another example + +```bash +printf "%s\n" "${numbers[@]}" | bc --mathlib | sort --general-numeric-sort +``` + +or + +```bash +printf "%s\n" "${numbers[@]}" \ +| bc --mathlib \ +| sort --general-numeric-sort +``` diff --git a/concepts/pipelines/links.json b/concepts/pipelines/links.json new file mode 100644 index 00000000..caff37f3 --- /dev/null +++ b/concepts/pipelines/links.json @@ -0,0 +1,10 @@ +[ + { + "url": "https://www.gnu.org/software/bash/manual/bash.html#Pipelines", + "description": "Pipelines" + }, + { + "url": "https://www.gnu.org/software/bash/manual/bash.html#Lists", + "description": "Lists of Commands" + } +] diff --git a/config.json b/config.json index b769bf2e..00451a2c 100644 --- a/config.json +++ b/config.json @@ -1243,6 +1243,11 @@ "uuid": "ae9f3e82-bcdd-4c09-9788-bc543235fd52", "slug": "looping", "name": "Looping" + }, + { + "uuid": "f64a17aa-cbdb-49de-b50c-f3bf14a4e03d", + "slug": "pipelines", + "name": "Pipelines and Command Lists" } ], "key_features": [ From 24f5020360c6e7dcae2e27dffacfa9878b9195ab Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Mon, 25 Nov 2024 21:17:56 -0500 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Isaac Good --- concepts/pipelines/introduction.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/concepts/pipelines/introduction.md b/concepts/pipelines/introduction.md index 0bfc1f78..178394e3 100644 --- a/concepts/pipelines/introduction.md +++ b/concepts/pipelines/introduction.md @@ -1,6 +1,6 @@ # Pipelines and Command Lists -We have seen how to write simple commands, where the command is followed by arguments. +We have seen how to write simple commands, where a command is followed by arguments. Now we will see how to make more complex commands by composing simple commands. ## Pipelines @@ -25,22 +25,22 @@ The pipe symbol (`|`) connects the output of one command to the input of another * There is a performance cost to running pipelines. If you find yourself with long pipelines of similar commands, consider combining them in a single command. - For example, pipelines using multiple instances of grep+cut+sed+awk+tr can be combined into a single awk command for efficiency. + For example, pipelines using multiple instances of `grep`, `cut`, `sed`, `awk`, and `tr` can generally be combined into a single `awk` command for efficiency. * The exit status of a pipeline is the exit status of the last command in the pipeline. However, there is a shell setting that can control this. - The "pipefail" setting (enabled with `set -o pipefail`) will use the _**last** non-zero exit status_ of the pipeline's commands as the pipeline's exit status, unless all commands succeeded. + The "pipefail" setting (enabled with `set -o pipefail`) will use the _**last** non-zero exit status_ of the commands in a pipeline as the pipeline's exit status, unless all commands succeeded. ~~~~ ## Command Lists A command list is a sequence of pipelines separated by `;` (or newline), `&&` or `||`. -* `A ; B` is a command list where `B` executes after `A` has completed. +* `A; B` is a command list where `B` executes after `A` has completed. * `A && B`, where `B` executes only if `A` succeeds (exits with status zero). * `A || B`, where `B` executes only if `A` fails (exits with status non-zero). -The exit status of a command list is the exit status of the last command that executes. +The exit status of a command list is the exit status of the last command that was executed. The `&&` and `||` operators can be chained so that the next command conditionally executes based on the status of the preceding commands. For example @@ -68,9 +68,9 @@ if A; then B; else C; fi A && B || C ``` -The difference is: when does C execute? +They differ in when C is executed. -* In the first snippet, the if statement, C will only execute if A fails. +* In the first snippet (the if statement), C will only execute if A fails. * In the second snippet, C executes if A fails _or if A succeeds but B fails_! ~~~~ @@ -136,7 +136,7 @@ die () { Long command lists become hard to read quite quickly. Liberal use of newlines can help a lot. -Here's an example: a word is added to an array if two conditions are met +Consider this example where a word is added to an array if two conditions are met. ```bash [[ "$word" != "$topic" ]] && [[ "$key" == "$(sorted "$topic")" ]] && anagrams+=("$candidate") @@ -150,7 +150,7 @@ Bash allows you to add a newline after a pipe or a logical operator. anagrams+=("$candidate") ``` -But the operator kind of gets lost at the end of the line. +However, the operator can be easy to miss at the end of the line. Using a _line continuation_ means you can put the operator first, which makes it more obvious that the list is being continued: ```bash @@ -160,7 +160,7 @@ Using a _line continuation_ means you can put the operator first, which makes it ``` ~~~~exercism/note -A _line continuation_ is the two character sequence "backslash"+"newline". +A _line continuation_ is the two character sequence "backslash" and "newline" (`\` + `\n`). When bash sees that sequence, it is simply removed from the code, thereby _continuing_ the current line with the next line. Take care to not allow any spaces between the backslash and the newline. ~~~~