diff --git a/book/how_nushell_code_gets_run.md b/book/how_nushell_code_gets_run.md index 6121f449c1..204484c705 100644 --- a/book/how_nushell_code_gets_run.md +++ b/book/how_nushell_code_gets_run.md @@ -1,325 +1,367 @@ # How Nushell Code Gets Run -As you probably noticed, Nushell behaves quite differently from other shells and dynamic languages. In [Thinking in Nu](thinking_in_nu.md#think-of-nushell-as-a-compiled-language), we advise you to _think of Nushell as a compiled language_ but we do not give much insight into why. This section hopefully fills the gap. +In [Thinking in Nu](./thinking_in_nu.md#think-of-nushell-as-a-compiled-language), we encouraged you to _"Think of Nushell as a compiled language"_ due to the way in which Nushell code is processed. We also covered several code examples that won't work in Nushell due that process. -First, let's give a few example which you might intuitively try but which do not work in Nushell. +The underlying reason for this is a strict separation of the **_parsing and evaluation_** stages that **_disallows `eval`-like functionality_**. In this section, we'll explain in detail what this means, why we're doing it, and what the implications are. The explanation aims to be as simple as possible, but it might help if you've programmed in another language before. -1. Sourcing a dynamic path (note that a constant would work, see [parse-time evaluation](#parse-time-evaluation)) +[[toc]] -```nu -let my_path = 'foo' -source $"($my_path)/common.nu" -``` - -2. Write to a file and source it in a single script - -```nu -"def abc [] { 1 + 2 }" | save output.nu -source "output.nu" -``` - -3. Change a directory and source a path within (even though the file exists) - -```nu -if ('spam/foo.nu' | path exists) { - cd spam - source-env foo.nu -} -``` - -The underlying reason why all of the above examples won't work is a strict separation of **parsing and evaluation** steps by **disallowing eval function**. In the rest of this section, we'll explain in detail what it means, why we're doing it, and what the implications are. The explanation aims to be as simple as possible, but it might help if you've written a program in some language before. - -## Parsing and Evaluation +## Interpreted vs. Compiled Languages ### Interpreted Languages -Let's start with a simple "hello world" Nushell program: +Nushell, Python, and Bash (and many others) are _"interpreted"_ languages. + +Let's start with a simple "Hello, World!" Nushell program: ```nu # hello.nu -print "Hello world!" +print "Hello, World!" ``` -When you run `nu hello.nu`, Nushell's interpreter directly runs the program and prints the result to the screen. This is similar (on the highest level) to other languages that are typically interpreted, such as Python or Bash. If you write a similar "hello world" program in any of these languages and call `python hello.py` or `bash hello.bash`, the result will be printed to the screen. We can say that interpreters take the program in some representation (e.g., a source code), run it, and give you the result: +Of course, this runs as expected using `nu hello.nu`. A similar program written in Python or Bash would look (and behave) nearly the same. + +In _"interpreted languages"_ code usually gets handled something like this: ``` -source code --> interpreting --> result +Source Code → Interpreter → Result ``` -Under the hood, Nushell's interpreter is split into two parts, like this: +Nushell follows this pattern, and its "Interpreter" is split into two parts: -``` -1. source code --> parsing --> Intermediate Representation (IR) -2. IR --> evaluating --> result -``` +1. `Source Code → Parser → Intermediate Representation (IR)` +2. `IR → Evaluation Engine → Result` + +First, the source code is analyzed by the Parser and converted into an intermediate representation (IR), which in Nushell's case is just a collection of data structures. Then, these data structures are passed to the Engine for evaluation and output of the results. -First, the source code is analyzed by the parser and converted into an intermediate representation (IR), which in Nushell's case are just some data structures. Then, these data structures are passed to the engine which evaluates them and produces the result. This is nothing unusual. For example, Python's source code is typically converted into [bytecode](https://en.wikipedia.org/wiki/Bytecode) before evaluation. +This, as well, is common in interpreted languages. For example, Python's source code is typically [converted into bytecode](https://github.com/python/cpython/blob/main/InternalDocs/interpreter.md) before evaluation. ### Compiled Languages -On the other side are languages that are typically "compiled", such as C, C++, or Rust. Assuming a simple ["hello world"](https://doc.rust-lang.org/stable/book/ch01-02-hello-world.html) in Rust +On the other side are languages that are typically "compiled", such as C, C++, or Rust. For example, here's a simple _"Hello, World!"_ in Rust: ```rust // main.rs fn main() { - println!("Hello, world!"); + println!("Hello, World!"); } ``` -you first need to _compile_ the program into [machine code instructions](https://en.wikipedia.org/wiki/Machine_code) and store the binary file to a disk (`rustc main.rs`). Then, to produce a result, you need to run the binary (`./main`), which passes the instructions to the CPU: +To "run" this code, it must be: -``` -1. source code --> compiler --> machine code -2. machine code --> CPU --> result -``` +1. Compiled into [machine code instructions](https://en.wikipedia.org/wiki/Machine_code) +2. The compilation results stored as a binary file one the disk + +The first two steps are handled with `rustc main.rs`. + +3. Then, to produce a result, you need to run the binary (`./main`), which passes the instructions to the CPU -You can see the compile-run sequence is not that much different from the parse-evaluate sequence of an interpreter. You begin with a source code, parse (or compile) it into some IR (or machine code), then evaluate (or run) the IR to get a result. You could think of machine code as just another type of IR and the CPU as its interpreter. +So: -One big difference, however, between interpreted and compiled languages is that interpreted languages typically implement an _eval function_ while compiled languages do not. What does it mean? +1. `Source Code ⇒ Compiler ⇒ Machine Code` +2. `Machine Code ⇒ CPU ⇒ Result` + +::: important +You can see that the compile-run sequence is not much different from the parse-evaluate sequence of an interpreter. You begin with source code, parse (or compile) it into some state (e.g., bytecode, IR, machine code), then evaluate (or run) the IR to get a result. You could think of machine code as just another type of IR and the CPU as its interpreter. + +One big difference, however, between interpreted and compiled languages is that interpreted languages typically implement an _`eval` function_ while compiled languages do not. What does this mean? +::: + +## Dynamic vs. Static Languages + +::: tip Terminology +In general, the difference between a dynamic and static language is how much of the source code is resolved during Compilation (or Parsing) vs. Evaluation/Runtime: + +- _"Static"_ languages perform more code analysis (e.g., type-checking, [data ownership](https://doc.rust-lang.org/stable/book/ch04-00-understanding-ownership.html)) during Compilation/Parsing. + +- _"Dynamic"_ languages perform more code analysis, including `eval` of additional code, during Evaluation/Runtime. + +For the purposes of this discussion, the primary difference between a static and dynamic language is whether or not it has an `eval` function. + +::: ### Eval Function -Most languages considered as "dynamic" or "interpreted" have an eval function, for example Python (it has two, [eval](https://docs.python.org/3/library/functions.html#eval) and [exec](https://docs.python.org/3/library/functions.html#exec)) or [Bash](https://linux.die.net/man/1/bash). It is used to take source code and interpret it within a running interpreter. This can get a bit confusing, so let's give a Python example: +Most dynamic, interpreted languages have an `eval` function. For example, [Python `eval`](https://docs.python.org/3/library/functions.html#eval) (also, [Python `exec`](https://docs.python.org/3/library/functions.html#exec)) or [Bash `eval`](https://linux.die.net/man/1/bash). + +The argument to an `eval` is _"source code inside of source code"_, typically conditionally or dynamically computed. This means that, when an interpreted language encounters an `eval` in source code during Parse/Eval, it typically interrupts the normal Evaluation process to start a new Parse/Eval on the source code argument to the `eval`. -```python +Here's a simple Python `eval` example to demonstrate this (potentially confusing!) concept: + +```python:line-numbers # hello_eval.py -print("Hello world!") -eval("print('Hello eval!')") +print("Hello, World!") +eval("print('Hello, Eval!')") ``` -When you run the file (`python hello_eval.py`), you'll see two messages: "Hello world!" and "Hello eval!". Here is what happened: +When you run the file (`python hello_eval.py`), you'll see two messages: _"Hello, World!"_ and _"Hello, Eval!"_. Here is what happens: + +1. The entire program is Parsed +2. (Line 3) `print("Hello, World!")` is Evaluated +3. (Line 4) In order to evaluate `eval("print('Hello, Eval!')")`: + 1. `print('Hello, Eval!')` is Parsed + 2. `print('Hello, Eval!')` is Evaluated + +::: tip More fun +Consider `eval("eval(\"print('Hello, Eval!')\")")` and so on! +::: -1. Parse the whole source code -2. Evaluate `print("Hello world!")` -3. To evaluate `eval("print('Hello eval!')")`: - 1. Parse `print('Hello eval!')` - 2. Evaluate `print('Hello eval!')` +Notice how the use of `eval` here adds a new "meta" step into the execution process. Instead of a single Parse/Eval, the `eval` creates additional, "recursive" Parse/Eval steps instead. This means that the bytecode produced by the Python interpreter can be further modified during the evaluation. -Of course, you can have more fun and try `eval("eval(\"print('Hello eval!')\")")` and so on... +Nushell does not allow this. -You can see the eval function adds a new "meta" layer into the code execution. Instead of parsing the whole source code, then evaluating it, there is an extra parse-eval step during the evaluation. This means that the IR produced by the parser (whatever it is) can be further modified during the evaluation. +As mentioned above, without an `eval` function to modify the bytecode during the interpretation process, there's very little difference (at a high level) between the Parse/Eval process of an interpreted language and that of the Compile/Run in compiled languages like C++ and Rust. -We've seen that without `eval`, the difference between compiled and interpreted languages is actually not that big. This is exactly what we mean by [thinking of Nushell as a compiled language](https://www.nushell.sh/book/thinking_in_nu.html#think-of-nushell-as-a-compiled-language): Despite Nushell being an interpreted language, its lack of `eval` gives it characteristics and limitations typical for traditional compiled languages like C or Rust. We'll dig deeper into what it means in the next section. +::: tip Takeaway +This is why we recommend that you _"think of Nushell as a compiled language"_. Despite being an interpreted language, its lack of `eval` gives it some of the characteristic benefits as well as limitations common in traditional static, compiled languages. +::: + +We'll dig deeper into what it means in the next section. ## Implications Consider this Python example: -```python +```python:line-numbers exec("def hello(): print('Hello eval!')") hello() ``` -_Note: We're using `exec` instead of `eval` because it can execute any valid Python code, not just expressions. The principle is similar, though._ +::: note +We're using `exec` in this example instead of `eval` because it can execute any valid Python code rather than being limited to `eval` expressions. The principle is similar in both cases, though. +::: + +During interpretation: -What happens: +1. The entire program is Parsed +2. In order to Evaluate Line 1: + 1. `def hello(): print('Hello eval!')` is Parsed + 2. `def hello(): print('Hello eval!')` is Evaluated +3. (Line 2) `hello()` is evaluated. -1. Parse the whole source code -2. To evaluate `exec("def hello(): print('Hello eval!')")`: - 1. Parse `def hello(): print('Hello eval!')` - 2. Evaluate `def hello(): print('Hello eval!')` -3. Evaluate `hello()` +Note, that until step 2.2, the interpreter has no idea that a function `hello` even exists! This makes [static analysis](https://en.wikipedia.org/wiki/Static_program_analysis) of dynamic languages challenging. In this example, the existence of the `hello` function cannot be checked just by parsing (compiling) the source code. The interpreter must evaluate (run) the code to discover it. -Note, that until step 2.2, the interpreter has no idea a function `hello` exists! This makes static analysis of dynamic languages challenging. In the example, the existence of `hello` function cannot be checked just by parsing (compiling) the source code. You actually need to go and evaluate (run) the code to find out. While in a compiled language, missing function is a guaranteed compile error, in a dynamic interpreted language, it is a runtime error (which can slip unnoticed if the line calling `hello()` is, for example, behind an `if` condition and does not get executed). +- In a static, compiled language, a missing function is guaranteed to be caught at compile-time. +- In a dynamic, interpreted language, however, it becomes a _possible_ runtime error. If the `eval`-defined function is conditionally called, the error may not be discovered until that condition is met in production. +::: important In Nushell, there are **exactly two steps**: -1. Parse the whole source code -2. Evaluate the whole source code +1. Parse the entire source code +2. Evaluate the entire source code -This is the complete parse-eval sequence. +This is the complete Parse/Eval sequence. +::: -Not having `eval`-like functionality prevents `eval`-related bugs from happening. Calling a non-existent function is 100% guaranteed parse-time error in Nushell. Furthermore, after the parse step, we have a deep insight into the program and we're 100% sure it is not going to change during evaluation. This trivially allows for powerful and reliable static analysis and IDE integration which is challenging to achieve with more dynamic languages. In general, you have more peace of mind when scaling Nushell programs to bigger applications. +::: tip Takeaway +By not allowing `eval`-like functionality, Nushell prevents these types of `eval`-related bugs. Calling a non-existent definition is guaranteed to be caught at parse-time in Nushell. -_Before going into examples, one note about the "dynamic" and "static" terminology. Stuff that happens at runtime (during evaluation, after parsing) is considered "dynamic". Stuff that happens before running (during parsing / compilation) is called "static". Languages that have more stuff (such as `eval`, type checking, etc.) happening at runtime are sometimes called "dynamic". Languages that analyze most of the information (type checking, [data ownership](https://doc.rust-lang.org/stable/book/ch04-00-understanding-ownership.html), etc.) before evaluating the program are sometimes called "static". The whole debate can get quite confusing, but for the purpose of this text, the main difference between a "static" and "dynamic" language is whether it has or has not the eval function._ +Furthermore, after parsing completes, we can be certain the bytecode (IR) won't change during evaluation. This gives us a deep insight into the resulting bytecode (IR), allowing for powerful and reliable static analysis and IDE integration which can be challenging to achieve with more dynamic languages. -## Common Mistakes +In general, you have more peace of mind that errors will be caught earlier when scaling Nushell programs. +::: -By insisting on strict parse-evaluation separation, we lose much of a flexibility users expect from dynamic interpreted languages, especially other shells, such as bash, fish, zsh and others. This leads to the examples at the beginning of this page not working. Let's break them down one by one +## The Nushell REPL -_Note: The following examples use [`source`](/commands/docs/source.md), but similar conclusions apply to other commands that parse Nushell source code, such as [`use`](/commands/docs/use.md), [`overlay use`](/commands/docs/overlay_use.md), [`hide`](/commands/docs/hide.md) or [`source-env`](/commands/docs/source-env.md)._ +As with most any shell, Nushell has a _"Read→Eval→Print Loop"_ ([REPL](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop)) that is started when you run `nu` without any file. This is often thought of, but isn't quite the same, as the _"commandline"_. -### 1. Sourcing a dynamic path +::: tip Note +In this section, the `> ` character at the beginning of a line in a code-block is used to represent the commandline **_prompt_**. For instance: ```nu -let my_path = 'foo' -source $"($my_path)/common.nu" +> some code... ``` -Let's break down what would need to happen for this to work: +Code after the prompt in the following examples is executed by pressing the Enter key. For example: -1. Parse `let my_path = 'foo'` and `source $"($my_path)/config.nu"` -2. To evaluate `source $"($my_path)/common.nu"`: - 1. Parse `$"($my_path)/common.nu"` - 2. Evaluate `$"($my_path)/common.nu"` to get the file name - 3. Parse the contents of the file - 4. Evaluate the contents of the file +```nu +> print "Hello world!" +# => Hello world! -You can see the process is similar to the `eval` functionality we talked about earlier. Nesting parse-evaluation cycles into the evaluation is not allowed in Nushell. +> ls +# => prints files and directories... +``` -To give another perspective, here is why it is helpful to _think of Nushell as a compiled language_. Instead of +The above means: -```nu -let my_path = 'foo' -source $"($my_path)/common.nu" -``` +- From inside Nushell (launched with `nu`): + 1. Type `print "Hello world!"` + 1. Press Enter + 1. Nushell will display the result + 1. Type `ls` + 1. Press Enter + 1. Nushell will display the result -imagine it being written in some typical compiled language, such as C++ +::: -```cpp -#include +When you press Enter after typing a commandline, Nushell: -std::string my_path("foo"); -#include -``` +1. **_(Read):_** Reads the commandline input +1. **_(Evaluate):_** Parses the commandline input +1. **_(Evaluate):_** Evaluates the commandline input +1. **_(Evaluate):_** Merges the environment (such as the current working directory) to the internal Nushell state +1. **_(Print):_** Displays the results (if non-`null`) +1. **_(Loop):_** Waits for another input -or Rust +In other words, each REPL invocation is its own separate parse-evaluation sequence. By merging the environment back to the Nushell's state, we maintain continuity between the REPL invocations. -```rust -let my_path = "foo"; -use format!("{}::common", my_path); +Compare a simplified version of the [`cd` example](./thinking_in_nu.md#example-change-to-a-different-directory-cd-and-source-a-file) from _"Thinking in Nu"_: + +```nu +cd spam +source-env foo.nu ``` -If you've ever written a simple program in any of these languages, you can see these examples do not make a whole lot of sense. You need to have all the source code files ready and available to the compiler beforehand. +There we saw that this cannot work (as a script or other single expression) because the directory will be changed _after_ the parse-time [`source-env` keyword](/commands/docs/source-env.md) attempts to read the file. -### 2. Write to a file and source it in a single script +Running these commands as separate REPL entries, however, works: ```nu -"def abc [] { 1 + 2 }" | save output.nu -source "output.nu" +> cd spam +> source-env foo.nu +# Yay, works! ``` -Here, the sourced path is static (= known at parse-time) so everything should be fine, right? Well... no. Let's break down the sequence again: +To see why, let's break down what happens in the example: -1. Parse the whole source code - 1. Parse `"def abc [] { 1 + 2 }" | save output.nu` - 2. Parse `source "output.nu"` - 1.2.1. Open `output.nu` and parse its contents -2. Evaluate the whole source code - 1. Evaluate `"def abc [] { 1 + 2 }" | save output.nu` to generate `output.nu` - 2. ...wait what??? +1. Read the `cd spam` commandline. +2. Parse the `cd spam` commandline. +3. Evaluate the `cd spam` commandline. +4. Merge environment (including the current directory) into the Nushell state. +5. Read and Parse `source-env foo.nu`. +6. Evaluate `source-env foo.nu`. +7. Merge environment (including any changes from `foo.nu`) into the Nushell state. -We're asking Nushell to read `output.nu` before it even exists. All the source code needs to be available to Nushell at parse-time, but `output.nu` is only generated during evaluation. Again, it helps here to _think of Nushell as a compiled language_. +When `source-env` tries to open `foo.nu` during the parsing in Step 5, it can do so because the directory change from Step 3 was merged into the Nushell state during Step 4. As a result, it's visible in the following Parse/Eval cycles. -### 3. Change a directory and source a path within +### Multiline REPL Commandlines -(We assume the `spam/foo.nu` file exists.) +Keep in mind that this only works for **_separate_** commandlines. -```nu -if ('spam/foo.nu' | path exists) { - cd spam - source-env foo.nu -} -``` +In Nushell, it's possible to group multiple commands into one commandline using: -This one is similar to the previous example. `cd spam` changes the directory _during evaluation_ but [`source-env`](/commands/docs/source-env.md) attempts to open and read `foo.nu` during parsing. +- A semicolon: -## REPL + ```nu + cd spam; source-env foo.nu + ``` -[REPL](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop) is what happens when you run `nu` without any file. You launch an interactive prompt. By +- A newline: -```nu -> some code... -``` + ``` + > cd span + source-env foo.nu + ``` -we denote a REPL entry followed by pressing Enter. For example + Notice there is no "prompt" before the second line. This type of multiline commandline is usually created with a [keybinding](./line_editor.md#keybindings) to insert a Newline when Alt+Enter or Shift+ Enter is pressed. -```nu -> print "Hello world!" -Hello world! +These two examples behave exactly the same in the Nushell REPL. The entire commandline (both statements) are processed a single Read→Eval→Print Loop. As such, they will fail the same way that the earlier script-example did. -> ls -# prints files and directories... -``` +::: tip +Multiline commandlines are very useful in Nushell, but watch out for any out-of-order Parser-keywords. +::: -means the following: +## Parse-time Constant Evaluation -1. Launch `nu` -2. Type `print "Hello world!"`, press Enter -3. Type [`ls`](/commands/docs/ls.md), press Enter +While it is impossible to add parsing into the evaluation stage and yet still maintain our static-language benefits, we can safely add _a little bit_ of evaluation into parsing. -Hopefully, that's clear. Now, when you press Enter, these things happen: +::: tip Terminology +In the text below, we use the term _"constant"_ to refer to: -1. Parse the line input -2. Evaluate the line input -3. Merge the environment (such as the current working directory) to the internal Nushell state -4. Wait for another input +- A `const` definition +- The result of any command that outputs a constant value when provide constant inputs. + ::: -In other words, each REPL invocation is its own separate parse-evaluation sequence. By merging the environment back to the Nushell's state, we maintain continuity between the REPL invocations. +By their nature, **_constants_** and constant values are known at Parse-time. This, of course, is in sharp contrast to _variable_ declarations and values. + +As a result, we can utilize constants as safe, known arguments to parse-time keywords like [`source`](/commands/docs/source.md), [`use`](/commands/docs/use.md), and related commands. -To give an example, we showed that +Consider [this example](./thinking_in_nu.md#example-dynamically-creating-a-filename-to-be-sourced) from _"Thinking in Nu"_: ```nu -cd spam -source-env foo.nu +let my_path = "~/nushell-files" +source $"($my_path)/common.nu" ``` -does not work because the directory will be changed _after_ [`source-env`](/commands/docs/source-env.md) attempts to read the file. Running these commands as separate REPL entries, however, works: - -```nu -> cd spam +As noted there, we **_can_**, however, do the following instead: -> source-env foo.nu -# yay, works! +```nu:line-numbers +const my_path = "~/nushell-files" +source $"($my_path)/common.nu" ``` -To see why, let's break down what happens in the example: +Let's analyze the Parse/Eval process for this version: -1. Launch `nu` -2. Parse `cd spam` -3. Evaluate `cd spam` -4. **Merge environment (including the current directory) into the Nushell state** -5. Parse `source-env foo.nu` -6. Evaluate `source-env foo.nu` -7. Merge environment (including the current directory) into the Nushell state +1. The entire program is Parsed into IR. -When [`source-env`](/commands/docs/source-env.md) tries to open `foo.nu` during the parsing in step 5., it can do so because the directory change from step 3. was merged into the Nushell state in step 4. and therefore is visible in the following parse-evaluation cycles. + 1. Line 1: The `const` definition is parsed. Because it is a constant assignment (and `const` is also a parser-keyword), that assignment can also be Evaluated at this stage. Its name and value are stored by the Parser. + 2. Line 2: The `source` command is parsed. Because `source` is also a parser-keyword, it is Evaluated at this stage. In this example, however, it can be **_successfully_** parsed since its argument is **_known_** and can be retrieved at this point. + 3. The source-code of `~/nushell-files/common.nu` is parsed. If it is invalid, then an error will be generated, otherwise the IR results will be included in evaluation in the next stage. -### Parse-time Evaluation +2. The entire IR is Evaluated: + 1. Line 1: The `const` definition is Evaluated. The variable is added to the runtime stack. + 2. Line 2: The IR result from parsing `~/nushell-files/common.nu` is Evaluated. -While it is impossible to add parsing into the evaluation, we can add _a little bit_ of evaluation into parsing. This feature has been added [only recently](https://github.com/nushell/nushell/pull/7436) and we're going to expand it as needed. +::: important -One pattern that this unlocks is being able to [`source`](/commands/docs/source.md)/[`use`](/commands/docs/use.md)/etc. a path from a "variable". We've seen that +- An `eval` adds additional parsing during evaluation +- Parse-time constants do the opposite, adding additional evaluation to the parser. + ::: -```nu -let some_path = $nu.default-config-dir -source $"($some_path)/common.nu" -``` +Also keep in mind that the evaluation allowed during parsing is **_very restricted_**. It is limited to only a small subset of what is allowed during a regular evaluation. -does not work, but we can do the following: +For example, the following is not allowed: ```nu -const some_path = $nu.default-config-dir -source $"($some_path)/config.nu" +const foo_contents = (open foo.nu) ``` -We can break down what is happening again: +Put differently, only a small subset of commands and expressions can generate a constant value. For a command to be allowed: -1. Parse the whole source code - 1. Parse `const some_path = $nu.default-config-dir` - 1. Evaluate\* `$nu.default-config-dir` to `/home/user/.config/nushell` and store it as a `some_path` constant - 2. Parse `source $"($some_path)/config.nu"` - 1. Evaluate\* `$some_path`, see that it is a constant, fetch it - 2. Evaluate\* `$"($some_path)/config.nu"` to `/home/user/.config/nushell/config.nu` - 3. Parse the `/home/user/.config/nushell/config.nu` file -2. Evaluate the whole source code - 1. Evaluate `const some_path = $nu.default-config-dir` (i.e., add the `/home/user/.config/nushell` string to the runtime stack as `some_path` variable) - 2. Evaluate `source $"($some_path)/config.nu"` (i.e., evaluate the contents of `/home/user/.config/nushell/config.nu`) +- It must be designed to output a constant value +- All of its inputs must also be constant values, literals, or composite types (e.g., records, lists, tables) of literals. -This still does not violate our rule of not having an eval function, because an eval function adds additional parsing to the evaluation step. With parse-time evaluation we're doing the opposite. +In general, the commands and resulting expressions will be fairly simple and **_without side effects_**. Otherwise, the parser could all-too-easily enter an unrecoverable state. Imagine, for instance, attempting to assign an infinite stream to a constant. The Parse stage would never complete! -Also, note the \* in steps 1.1.1. and 1.2.1. The evaluation happening during parsing is very restricted and limited to only a small subset of what is normally allowed during a regular evaluation. For example, the following is not allowed: +::: tip +You can see which Nushell commands can return constant values using: ```nu -const foo_contents = (open foo.nu) +help commands | where is_const ``` -By allowing _everything_ during parse-time evaluation, we could set ourselves up to a lot of trouble (think of generating an infinite stream in a subexpression...). Generally, only a simple expressions _without side effects_ are allowed, such as string literals or integers, or composite types of these literals (records, lists, tables). +::: -Compiled ("static") languages also tend to have a way to convey some logic at compile time, be it C's preprocessor, Rust's macros, or [Zig's comptime](https://kristoff.it/blog/what-is-zig-comptime). One reason is performance (if you can do it during compilation, you save the time during runtime) which is not as important for Nushell because we always do both parsing and evaluation, we do not store the parsed result anywhere (yet?). The second reason is similar to Nushell's: Dealing with limitations caused by the absence of the eval function. +For example, the `path join` command can output a constant value. Nushell also defines several useful paths in the `$nu` constant record. These can be combined to create useful parse-time constant evaluations like: + +```nu +const my_startup_modules = $nu.default-config-dir | path join "my-mods" +use $"($my_startup_modules)/my-utils.nu" +``` + +::: note Additional Notes +Compiled ("static") languages also tend to have a way to convey some logic at compile time. For instance: + +- C's preprocessor +- Rust macros +- [Zig's comptime](https://kristoff.it/blog/what-is-zig-comptime), which was an inspiration for Nushell's parse-time constant evaluation. + +There are two reasons for this: + +1. _Increasing Runtime Performance:_ Logic in the compilation stage doesn't need to be repeated during runtime. + + This isn't currently applicable to Nushell, since the parsed results (IR) are not stored beyond Evaluation. However, this has certainly been considered as a possible future feature. + +2. As with Nushell's parse-time constant evaluations, these features help (safely) work around limitations caused by the absence of an `eval` function. + ::: ## Conclusion -Nushell operates in a scripting language space typically dominated by "dynamic" "interpreted" languages, such as Python, bash, zsh, fish, etc. While Nushell is also "interpreted" in a sense that it runs the code immediately, instead of storing the intermediate representation (IR) to a disk, one feature sets it apart from the pack: It does not have an **eval function**. In other words, Nushell cannot parse code and manipulate its IR during evaluation. This gives Nushell one characteristic typical for "static" "compiled" languages, such as C or Rust: All the source code must be visible to the parser beforehand, just like all the source code must be available to a C or Rust compiler. For example, you cannot [`source`](/commands/docs/source.md) or [`use`](/commands/docs/use.md) a path computed "dynamically" (during evaluation). This is surprising for users of more traditional scripting languages, but it helps to _think of Nushell as a compiled language_. +Nushell operates in a scripting language space typically dominated by _"dynamic"_, _"interpreted"_ languages, such as Python, Bash, Zsh, Fish, and many others. Nushell is also _"interpreted"_ since code is run immediately (without a separate, manual compilation). + +However, is not _"dynamic"_ in that it does not have an `eval` construct. In this respect, it shares more in common with _"static"_, compiled languages like Rust or Zig. + +This lack of `eval` is often surprising to many new users and is why it can be helpful to think of Nushell as a compiled, and static, language. diff --git a/book/thinking_in_nu.md b/book/thinking_in_nu.md index 7efe53ca43..24c675f1af 100644 --- a/book/thinking_in_nu.md +++ b/book/thinking_in_nu.md @@ -1,95 +1,440 @@ # Thinking in Nu -To help you understand - and get the most out of - Nushell, we've put together this section on "thinking in Nushell". By learning to think in Nushell and use the patterns it provides, you'll hit fewer issues getting started and be better setup for success. +Nushell is different! It's common (and expected!) for new users to have some existing "habits" or mental models coming from other shells or languages. -So what does it mean to think in Nushell? Here are some common topics that come up with new users of Nushell. +The most common questions from new users typically fall into one of the following topics: -## Nushell isn't bash +[[toc]] -Nushell is both a programming language and a shell. Because of this, it has its own way of working with files, directories, websites, and more. We've modeled this to work closely with what you may be familiar with other shells. Pipelines work by attaching two commands together: +## Nushell isn't Bash + +### It can sometimes look like Bash + +Nushell is both a programming language and a shell. Because of this, it has its own way of working with files, directories, websites, and more. You'll find that some features in Nushell work similar to those you're familiar with in other shells. For instance, pipelines work by combining two (or more) commands together, just like in other shells. + +For example, the following commandline works the same in both Bash and Nushell on Unix/Linux platforms: + +```nu +curl -s https://api.github.com/repos/nushell/nushell/contributors | jq '.[].login' +# => returns contributors to Nushell, ordered by number of contributions +``` + +Nushell has many other similarities with Bash (and other shells) and many commands in common. + +::: tip +Bash is primarily a command interpreter which runs external commands. Nushell provides many of these as cross-platform, built-in commands. + +While the above commandline works in both shells, in Nushell there's just no need to use the `curl` and `jq` commands. Instead, Nushell has a built-in [`http get` command](/commands/docs/http_get.md) and handles JSON data natively. For example: ```nu -> ls | length +http get https://api.github.com/repos/nushell/nushell/contributors | select login contributions ``` -Nushell, for example, also has support for other common capabilities like getting the exit code from previously run commands. +::: + +::: warning Thinking in Nushell +Nushell borrows concepts from many shells and languages. You'll likely find many of Nushell's features familiar. +::: -While it does have these amenities, Nushell isn't bash. The bash way of working, and the POSIX style in general, is not one that Nushell supports. For example, in bash, you might use: +### But it's not Bash + +Because of this, however, it's sometimes easy to forget that some Bash (and POSIX in general) style constructs just won't work in Nushell. For instance, in Bash, it would be normal to write: ```sh -> echo "hello" > output.txt +# Redirect using > +echo "hello" > output.txt +# But compare (greater-than) using the test command +test 4 -gt 7 +echo $? +# => 1 +``` + +In Nushell, however, the `>` is used as the greater-than operator for comparisons. This is more in line with modern programming expectations. + +```nu +4 > 10 +# => false +``` + +Since `>` is an operator, redirection to a file in Nushell is handled through a pipeline command that is dedicated to saving content - `[save](/commands/docs/save.md)`: + +```nu +"hello" | save output.txt +``` + +::: warning Thinking in Nushell +We've put together a list of common Bash'isms and how to accomplish those tasks in Nushell in the [Coming from Bash](./coming_from_bash.md) Chapter. +::: + +## Implicit Return + +Users coming from other shells will likely be very familiar with the `echo` command. Nushell's +[`echo`](/commands/docs/echo.md) might appear the same at first, but it is _very_ different. + +First, notice how the following output _looks_ the same in both Bash and Nushell (and even PowerShell and Fish): + +```nu +echo "Hello, World" +# => Hello, World +``` + +But while the other shells are sending `Hello, World` straight to _standard output_, Nushell's `echo` is +simply _returning a value_. Nushell then _renders_ the return value of a command, or more technically, an _expression_. + +More importantly, Nushell _implicitly returns_ the value of an expression. This is similar to PowerShell or Rust in many respects. + +::: tip +An expression can be more than just a pipeline. Even custom commands (similar to functions in many languages, but we'll cover them more in depth in a [later chapter](./custom_commands.md)) automatically, implicitly _return_ the last value. There's no need for an `echo` or even a [`return` command](/commands/docs/return.md) to return a value - It just _happens_. +::: + +In other words, the string _"Hello, World"_ and the output value from `echo "Hello, World"` are equivalent: + +```nu +"Hello, World" == (echo "Hello, World") +# => true +``` + +Here's another example with a custom command definition: + +```nu +def latest-file [] { + ls | sort-by modified | last +} +``` + +The _output_ of that pipeline (its _"value"_) becomes the _return value_ of the `latest-file` custom command. + +::: warning Thinking in Nushell +Most anywhere you might write `echo `, in Nushell, you can just write `` instead. +::: + +## Single Return Value per Expression + +It's important to understand that an expression can only return a single value. If there are multiple subexpressions inside an expression, only the **_last_** value is returned. + +A common mistake is to write a custom command definition like this: + +```nu:line-numbers +def latest-file [] { + echo "Returning the last file" + ls | sort-by modified | last +} + +latest-file +``` + +New users might expect: + +- Line 2 to output _"Returning the last file"_ +- Line 3 to return/output the file + +However, remember that `echo` **_returns a value_**. Since only the last value is returned, the Line 2 _value_ is discarded. Only the file will be returned by line 3. + +To make sure the first line is _displayed_, use the [`print` command](/commands/docs/print.md): + +```nu +def latest-file [] { +print "Returning last file" +ls | sort-by modified | last +} +``` + +Also compare: + +```nu +40; 50; 60 +``` + +::: tip +A semicolon is the same as a newline in a Nushell expression. The above is the same as a file or multi-line command: + +```nu +40 +50 +60 ``` -In Nushell, we use the `>` as the greater-than operator. This fits better with the language aspect of Nushell. Instead, you pipe to a command that has the job of saving content: +or ```nu -> "hello" | save output.txt +echo 40 +echo 50 +echo 60 ``` -**Thinking in Nushell:** The way Nushell views data is that data flows through the pipeline until it reaches the user or is handled by a final command. You can simply type data, from strings to JSON-style lists and records, and follow it with `|` to send it through the pipeline. Nushell uses commands to do work and produce more data. Learning these commands and when to use them helps you compose many kinds of pipelines. +::: + +In all of the above: + +- The first value is evaluated as the integer 40 but is not returned +- The second value is evaluated as the integer 50 but is not returned +- The third value is evaluated as the integer 60, and since it is the last + value, it is is returned and displayed (rendered). + +::: warning Thinking in Nushell +When debugging unexpected results, be on the lookout for: + +- Subexpressions (e.g., commands or pipelines) that ... +- ... output a (non-`null`) value ... +- ... where that value isn't returned from the parent expression. + +These can be likely sources of issues in your code. +::: + +## Every Command Returns a Value + +Some languages have the concept of "statements" which don't return values. Nushell does not. + +In Nushell, **_every command returns a value_**, even if that value is `null` (the `nothing` type). Consider the following multiline expression: + +```nu:line-numbers +let p = 7 +print $p +$p * 6 +``` + +1. Line 1: The integer 5 is assigned to `$p`, but the return value of the + [`let` command](/commands/docs/let.md) itself is `null`. However, because it is not the last + value in the expression, it is not displayed. +2. Line 2: The return value of the `print` command itself is `null`, but the `print` command + forces its argument (`$p`, which is 5) to be _displayed_. As with Line 1, the `null` return value + is discarded since this isn't the last value in the expression. +3. Line 3: Evaluates to the integer value 42. As the last value in the expression, this is the return + result, and is also displayed (rendered). + +::: warning Thinking in Nushell +Becoming familiar with the output types of common commands will help you understand how +to combine simple commands together to achieve complex results. + +`help ` will show the signature, including the output type(s), for each command in Nushell. +::: ## Think of Nushell as a Compiled Language -An important part of Nushell's design and specifically where it differs from many dynamic languages is that Nushell converts the source you give it into something to run, and then runs the result. It doesn't have an `eval` feature which allows you to continue pulling in new source during runtime. This means that tasks like including files to be part of your project need to be known paths, much like includes in compiled languages like C++ or Rust. +In Nushell, there are exactly two, separate, high-level stages when running code: + +1. _Stage 1 (Parser):_ Parse the **_entire_** source code +2. _Stage 2 (Engine):_ Evaluate the **_entire_** source code + +It can be useful to think of Nushell's parsing stage as _compilation_ in [static](./how_nushell_code_gets_run.md#dynamic-vs-static-languages) languages like Rust or C++. By this, we mean that all of the code that will be evaluated in Stage 2 must be **_known and available_** during the parsing stage. + +::: important +However, this also means that Nushell cannot currently support an `eval` construct as with _dynamic_ languages such as Bash or Python. +::: + +### Features Built on Static Parsing + +On the other hand, the **_static_** results of Parsing are key to many features of Nushell its REPL, such as: + +- Accurate and expressive error messages +- Semantic analysis for earlier and robust detection of error conditions +- IDE integration +- The type system +- The module system +- Completions +- Custom command argument parsing +- Syntax highlighting +- Real-time error highlighting +- Profiling and debugging commands +- (Future) Formatting +- (Future) Saving IR (Intermediate Representation) "compiled" results for faster execution + +### Limitations + +The static nature of Nushell often leads to confusion for users coming to Nushell from languages where an `eval` is available. + +Consider a simple two-line file: + +```text + + +``` + +1. Parsing: + 1. Line 1 is parsed + 2. Line 2 is parsed +2. If parsing was successful, then Evaluation: + 1. Line 1 is evaluated + 2. Line 1 is evaluated + +This helps demonstrate why the following examples cannot run as a single expression (e.g., a script) in Nushell: + +::: note +The following examples use the [`source` command](/commands/docs/source.md), but similar conclusions apply to other commands that parse Nushell source code, such as [`use`](/commands/docs/use.md), [`overlay use`](/commands/docs/overlay_use.md), [`hide`](/commands/docs/hide.md) or [`source-env`](/commands/docs/source-env.md). + +::: + +#### Example: Dynamically Generating Source + +Consider this scenario: + +```nu +"print Hello" | save output.nu +source output.nu +# => Error: nu::parser::sourced_file_not_found +# => +# => × File not found +# => ╭─[entry #5:2:8] +# => 1 │ "print Hello" | save output.nu +# => 2 │ source output.nu +# => · ────┬──── +# => · ╰── File not found: output.nu +# => ╰──── +# => help: sourced files need to be available before your script is run +``` + +This is problematic because: + +1. Line 1 is parsed but not evaluated. In other words, `output.nu` is not created during the parsing stage, but only during evaluation. +2. Line 2 is parsed. Because `source` is a parser-keyword, resolution of the sourced file is attempted during Parsing (Stage 1). But `output.nu` doesn't even exist yet! If it _does_ exist, then it's probably not even the correct file! This results in the error. + +::: note +Typing these as two _separate_ lines in the **_REPL_** will work since the first line will be parsed and evaluated, then the second line will be parsed and evaluated. + +The limitation only occurs when both are parsed _together_ as a single expression, which could be part of a script, block, closure, or other expression. + +See the [REPL](./how_nushell_code_gets_run.md#the-nushell-repl) section in _"How Nushell Code Gets Run"_ for more explanation. +::: + +#### Example: Dynamically Creating a Filename to be Sourced -For example, the following doesn't make sense in Nushell, and will fail to execute if run as a script: +Another common scenario when coming from another shell might be attempting to dynamically create a filename that will be sourced: ```nu -"def abc [] { 1 + 2 }" | save output.nu -source "output.nu" -abc +let my_path = "~/nushell-files" +source $"($my_path)/common.nu" +# => Error: +# => × Error: nu::shell::not_a_constant +# => │ +# => │ × Not a constant. +# => │ ╭─[entry #6:2:11] +# => │ 1 │ let my_path = "~/nushell-files" +# => │ 2 │ source $"($my_path)/common.nu" +# => │ · ────┬─── +# => │ · ╰── Value is not a parse-time constant +# => │ ╰──── +# => │ help: Only a subset of expressions are allowed constants during parsing. Try using the 'const' command or typing the value literally. +# => │ +# => ╭─[entry #6:2:8] +# => 1 │ let my_path = "~/nushell-files" +# => 2 │ source $"($my_path)/common.nu" +# => · ───────────┬─────────── +# => · ╰── Encountered error during parse-time evaluation +# => ╰──── +``` + +Because the `let` assignment is not resolved until evaluation, the parser-keyword `source` will fail during parsing if passed a variable. + +::: details Comparing Rust and C++ +Imagine that the code above was written in a typical compiled language such as C++: + +```cpp +#include + +std::string my_path("foo"); +#include +``` + +or Rust + +```rust +let my_path = "foo"; +use format!("{}::common", my_path); ``` -The [`source`](/commands/docs/source.md) command will grow the source that is compiled, but the [`save`](/commands/docs/save.md) from the earlier line won't have had a chance to run. Nushell runs the whole block as if it were a single file, rather than running one line at a time. In the example, since the output.nu file is not created until after the 'compilation' step, the [`source`](/commands/docs/source.md) command is unable to read definitions from it during parse time. +If you've ever written a simple program in any of these languages, you can see these examples aren't valid in those languages. Like Nushell, compiled languages require that all of the source code files are ready and available to the compiler beforehand. -Another common issue is trying to dynamically create the filename to source from: +::: + +::: tip See Also +As noted in the error message, however, this can work if `my_path` can be defined as a [constant](/book/variables#constant-variables) since constants can be (and are) resolved during parsing. ```nu -> source $"($my_path)/common.nu" +const my_path = "~/nushell-files" +source $"($my_path)/common.nu" +``` + +See [Parse-time Constant Evaluation](./how_nushell_code_gets_run.md#parse-time-constant-evaluation) for more details. +::: + +#### Example: Change to a different directory (`cd`) and `source` a file + +Here's one more — Change to a different directory and then attempt to `source` a file in that directory. + +```nu:line-numbers +if ('spam/foo.nu' | path exists) { + cd spam + source-env foo.nu +} ``` -This doesn't work if `my_path` is a regular runtime variable declared with `let`. This would require the -evaluator to run and evaluate the string, but unfortunately Nushell needs this information at compile-time. +Based on what we've covered about Nushell's Parse/Eval stages, see if you can spot the problem with that example. + +::: details Solution -However, if `my_path` is a [constant](/book/variables#constant-variables), then this -would work, since the string can be evaluated at compile-time: +In line 3, during Parsing, the `source-env` attempts to parse `foo.nu`. However, `cd` doesn't occur until Evaluation. This results in a parse-time error, since the file is not found in the _current_ directory. + +To resolve this, of course, simply use the full-path to the file to be sourced. ```nu -> const my_path = ([$nu.home-path nushell] | path join) -> source $"($my_path)/common.nu" # sources /home/user/nushell/common.nu + source-env spam/foo.nu ``` -**Thinking in Nushell:** Nushell is designed to use a single compile step for all the source you send it, and this is separate from evaluation. This will allow for strong IDE support, accurate error messages, an easier language for third-party tools to work with, and in the future even fancier output like being able to compile Nushell directly to a binary file. +::: + +### Summary -For more in-depth explanation, check [How Nushell Code Gets Run](how_nushell_code_gets_run.md). +::: important +For a more in-depth explanation of this section, see [How Nushell Code Gets Run](how_nushell_code_gets_run.md). +::: + +::: warning Thinking in Nushell +Nushell is designed to use a single Parsing stage for each expression or file. This Parsing stage occurs before and is separate from Evaluation. While this enables many of Nushell's features, it also means that users need to understand the limitations it creates. +::: ## Variables are Immutable by Default -Another common surprise for folks coming from other languages is that Nushell variables are immutable by default. Coming to Nushell, you'll want to spend some time becoming familiar with working in a more functional style, as this tends to help write code that works best with immutable variables. +Another common surprise when coming from other languages is that Nushell variables are immutable by default. While Nushell has optional mutable variables, many of Nushell's commands are based on a functional-style of programming which requires immutability. -**Thinking in Nushell:** If you're used to using mutable variables for different tasks, it will take some time to learn how to do each task in a more functional style. Nushell has a set of built-in capabilities to help with many of these patterns, and learning them will help you write code in a more Nushell-style. The added benefit of speeding up your scripts by running parts of your code in parallel is a nice bonus. +Immutable variables are also key to Nushell's [`par-each` command](/commands/docs/par-each.md), which allows you to operate on multiple values in parallel using threads. See [Immutable Variables](variables.html#immutable-variables) and [Choosing between mutable and immutable variables](variables.html#choosing-between-mutable-and-immutable-variables) for more information. +::: warning Thinking in Nushell +If you're used to relying on mutable variables, it may take some time to relearn how to code in a more functional style. Nushell has many functional features and commands that operate on and with immutable variables. Learning them will help you write code in a more Nushell-idiomatic style. + +A nice bonus is the performance increase you can realize by running parts of your code in parallel with `par-each`. +::: + ## Nushell's Environment is Scoped -Nushell takes multiple design cues from compiled languages. One such cue is that languages should avoid global mutable state. Shells have commonly used global mutation to update the environment, but Nushell steers clear of this approach. +Nushell takes multiple design cues from compiled languages. One such cue is that languages should avoid global mutable state. Shells have commonly used global mutation to update the environment, but Nushell attempts to steer clear of this approach. -In Nushell, blocks control their own environment. Changes to the environment are scoped to the block where they happen. +In Nushell, blocks control their own environment. Changes to the environment are scoped to the block where they occur. -In practice, this lets you write some concise code for working with subdirectories, for example, if you wanted to build each sub-project in the current directory, you could run: +In practice, this lets you write (as just one example) more concise code for working with subdirectories. Here's an example that builds each sub-project in the current directory: ```nu -> ls | each { |row| - cd $row.name - make +ls | each { |row| + cd $row.name + make } ``` -The [`cd`](/commands/docs/cd.md) command changes the `PWD` environment variables, and this variable change does not escape the block, allowing each iteration to start from the current directory and enter the next subdirectory. +The [`cd`](/commands/docs/cd.md) command changes the `PWD` environment variables, but this variable change does not survive past the end of the block. This allows each iteration to start from the current directory and then enter the next subdirectory. + +Having a scoped environment makes commands more predictable, easier to read, and when the time comes, easier to debug. It's also another feature that is key to the `par-each` command we discussed above. + +Nushell also provides helper commands like [`load-env`](/commands/docs/load-env.md) as a convenient way of loading multiple updates to the environment at once. -Having the environment scoped like this makes commands more predictable, easier to read, and when the time comes, easier to debug. Nushell also provides helper commands like [`def --env`](/commands/docs/def.md), [`load-env`](/commands/docs/load-env.md), as convenient ways of doing batches of updates to the environment. +::: tip See Also +[Environment - Scoping](./environment.md#scoping) +::: -_There is one exception here, where [`def --env`](/commands/docs/def.md) allows you to create a command that participates in the caller's environment._ +::: note +[`def --env`](/commands/docs/def.md) is an exception to this rule. It allows you to create a command that changes the parent's environment. +::: -**Thinking in Nushell:** - The coding best practice of no global mutable variables extends to the environment in Nushell. Using the built-in helper commands will let you more easily work with the environment in Nushell. Taking advantage of the fact that environments are scoped to blocks can also help you write more concise scripts and interact with external commands without adding things into a global environment you don't need. +::: warning Thinking in Nushell +Use scoped-environment to write more concise scripts and prevent unnecessary or unwanted global environment mutation. +:::