Different reference types: nullable and non-nullable (#1124)

* Port text from #700

The first commit ports in the text from #700 substantially without change. The only editorial change is to move the clause on "Nullable directives" in lexical-structure before the clause on Pragma directives.

* fix build warnings

This will likely cause new failures, as I turned on nullable annotations and warnings in one of the templates.

Update code samples for the template in use to remove nullable warnings.

I did this as we plan to have nullable annotations on by default in this version of the spec.

* Revert "fix build warnings"

This reverts commit dcb3731.

* test for nullable annotations

If the samples compile cleanly in this mode, I can continue in this PR without breaking all the samples.

* fix NRT grammar

* revert non-type changes.

* In progress draft

First draft at language for variations of nullability and reference types.

* Edit pass

* Apply suggestions from code review

Co-authored-by: Jon Skeet <[email protected]>

* remove null-oblivious type

* a fair amount of cleanup

* fix build issues.

* fix preprocessor symbols and test messages

* one more warning fix.

* respond to feedback.

* respond to feedback.

* Apply suggestions from code review

Co-authored-by: Jon Skeet <[email protected]>

* Notes for next updates

* fix warnings

* rebase and warnings

* word converter warnings

* word converter fx

* fix merge issues

* Updates from September meeting

At our September meeting, we decided that the standard should limit its normative text to the syntax required for nullable reference types, and a basic description explaining that the purpose of the annotations is to provide diagnostics.

* typo

* lint issue

* Add description of the `!` operator

This makes sense to add to this PR logically.

* null forgiveness can't be on array creation

I think this may fix the left recursion error as well. Even if it doesn't, it's still correct.

* Update for implicit declarations

State the nullable type for implicitly typed variables

* Apply suggestions from code review

Co-authored-by: Jon Skeet <[email protected]>

* final updates from Oct 2nd meeting

Additional edits from comments at the Oct 2 meeting. This resolves all open concerns.

* Apply suggestions from code review

Co-authored-by: Nigel-Ecma <[email protected]>

---------

Co-authored-by: Jon Skeet <[email protected]>
Co-authored-by: Nigel-Ecma <[email protected]>

standard/basic-concepts.md

@@ -1030,7 +1030,7 @@ The behavior of the garbage collector can be controlled, to some degree, via sta
 > {
 >     static void Main()
 >     {
->         B? b = new B(new A());
+>         B b = new B(new A());
 >         b = null;
 >         GC.Collect();
 >         GC.WaitForPendingFinalizers();
@@ -1075,19 +1075,19 @@ The behavior of the garbage collector can be controlled, to some degree, via sta
 >
 > class B
 > {
->     public A? Ref;
+>     public A Ref;
 >
 >     ~B()
 >     {
 >         Console.WriteLine("Finalize instance of B");
->         Ref?.F();
+>         Ref.F();
 >     }
 > }
 >
 > class Test
 > {
->     public static A? RefA;
->     public static B? RefB;
+>     public static A RefA;
+>     public static B RefB;
 >
 >     static void Main()
 >     {

standard/conversions.md

@@ -79,6 +79,7 @@ However, dynamic conversions ([§10.2.10](conversions.md#10210-implicit-dynamic-
 An identity conversion converts from any type to the same type or a type that is equivalent at runtime. One reason this conversion exists is so that a type `T` or an expression of type `T` can be said to be convertible to `T` itself. The following identity conversions exist:
 
 - Between `T` and `T`, for any type `T`.
+- Between `T` and `T?` for any reference type `T`.
 - Between `object` and `dynamic`.
 - Between all tuple types with the same arity, and the corresponding constructed `ValueTuple<...>` type, when an identity conversion exists between each pair of corresponding element types.
 - Between types constructed from the same generic type where there exists an identity conversion between each corresponding type argument.

standard/expressions.md

@@ -1273,6 +1273,7 @@ Primary expressions include the simplest forms of expressions.
 primary_expression
     : primary_no_array_creation_expression
     | array_creation_expression
+    | null_forgiving_expression
     ;
 
 primary_no_array_creation_expression
@@ -1821,6 +1822,47 @@ A  *null_conditional_member_access* expression `E` is of the form `P?.A`. The me
 
 A *null_conditional_projection_initializer* is a restriction of *null_conditional_member_access* and has the same semantics. It only occurs as a projection initializer in an anonymous object creation expression ([§12.8.16.7](expressions.md#128167-anonymous-object-creation-expressions)).
 
+### §Null-Forgiving-Expressions Null-forgiving expressions
+
+This operator sets the null state (§Nullabilities-And-Null-States) of the operand to “not null”.
+
+```ANTLR
+null_forgiving_expression
+    : primary_no_array_creation_expression suppression
+    ;
+
+suppression
+    : '!'
+    ;
+```
+
+This operator has no runtime effect; it evaluates to the result of its operand, and that result retains that operand’s classification.
+
+The null-forgiving operator is used to declare that an expression not known to be a value type is not null.
+
+> *Example*: Consider the following:
+>
+> <!-- Example: {template:"code-in-partial-class", name:"NullForgivingExpressions", additionalFiles:["PersonWithName.cs", "SupportNullForgivingExpressions.cs"]} -->
+> <!-- FIX: create and add 2 files. -->
+> ```csharp
+> #nullable enable
+> public static void M()
+> {
+>     Person? p = Find("John");                  // returns Person?
+>     if (IsValid(p))
+>     {
+>        Console.WriteLine($"Found {p!.Name}");  // p can't be null
+>     }
+> }
+>
+> public static bool IsValid(Person? person) =>
+>     person != null && person.Name != null;
+> ```
+>
+> If `IsValid` returns `true`, `p` can safely be dereferenced to access its `Name` property, and the “dereferencing of a possibly null value” warning can be suppressed using `!`. *end example*
+
+The null state (§Nullabilities-And-Null-States) of a *null_forgiving_expression* is “not null.”
+
 ### 12.8.9 Invocation expressions
 
 #### 12.8.9.1 General

standard/lexical-structure.md

@@ -1078,8 +1078,8 @@ The following pre-processing directives are available:
 - `#line`, which is used to control line numbers emitted for errors and warnings ([§6.5.8](lexical-structure.md#658-line-directives)).
 - `#error`, which is used to issue errors ([§6.5.6](lexical-structure.md#656-diagnostic-directives)).
 - `#region` and `#endregion`, which are used to explicitly mark sections of source code ([§6.5.7](lexical-structure.md#657-region-directives)).
-- `#pragma`, which is used to specify optional contextual information to a compiler ([§6.5.10](lexical-structure.md#6510-pragma-directives)).
 - `#nullable`, which is used to specify the nullable context ([§6.5.9](lexical-structure.md#659-nullable-directive)).
+- `#pragma`, which is used to specify optional contextual information to a compiler ([§6.5.10](lexical-structure.md#6510-pragma-directives)).
 
 A pre-processing directive always occupies a separate line of source code and always begins with a `#` character and a pre-processing directive name. White space may occur before the `#` character and between the `#` character and the directive name.
 

standard/statements.md

@@ -366,7 +366,7 @@ implicitly_typed_local_variable_declarator
     ;
 ```
 
-An *implicity_typed_local_variable_declaration* introduces a single local variable, *identifier*. The *expression* or *variable_reference* shall have a compile-time type, `T`. The first alternative declares a variable with type `T` and an initial value of *expression*. The second alternative declares a ref variable with type `ref T` and an initial value of `ref` *variable_reference*.
+An *implicity_typed_local_variable_declaration* introduces a single local variable, *identifier*. The *expression* or *variable_reference* shall have a compile-time type, `T`. The first alternative declares a variable with an initial value of *expression*; its type is `T?` when `T` is a non-nullable reference type, otherwise its type is `T`. The second alternative declares a ref variable with an initial value of `ref` *variable_reference*; its type is `ref T?` when `T` is a non-nullable reference type, otherwise its type is `ref T`.
 
 > *Example*:
 >