EEP XXX: internal export

eeps/eep-0067.md

@@ -0,0 +1,237 @@
+    Author: Marko Minđek <marko.mindek(at)invariant(dot)hr>,
+            Karlo Nikšić <kuna.prime(at)invariant(dot)hr>
+    Status: Draft
+    Type: Standards Track
+    Created: 02-Jan-2024
+    Erlang-Version: OTP-27
+    Post-History:
+****
+EEP XXX: Internal export
+----
+
+Abstract
+========
+
+This EEP introduces a new directive called `internal_export` which
+enables semantic separation of function exports.  This EEP is mostly
+inspired by [EEP 5][].
+
+Rationale
+=========
+
+Erlang application API is an ambigous term.  In theory, it is a set
+of module APIs (exported functions).  In practice, it is a set of
+modules' APIs but without modules considered internal, undocumented
+exports, callback implementation, etc.  While reading the docs,
+you can conclude what is application API, but while reading the
+source, it's not so straightforward.
+
+Not all exports are semantically the same.  A function can be exported
+to be part of the application API, to implement a callback, to test
+some code, or to allow modules within the application to use it.
+Currently, from the user perspective, they are all the same.  There
+is a convention not to use undocumented exports, but there is no way
+to enforce that convention in a codebase.
+
+The goal of this EEP is to give a mechanism to separate application API
+exports from other exports.
+
+A few terms are introduced for convenience:
+
+- internally exported function - a function marked with
+`internal_export` attribute
+- internal module - module containing only internally exported functions
+- application dependencies - a set of applications found in `applications`,
+`include_applications` and `optional_applications` properties in
+`.app` file (and their dependencies, recursively)
+- export scope - a set of modules that can *legally* access the
+exported function
+
+Internally exported functions are still globally exported, but
+application users are discouraged from using them.  The mechanism is similar
+to the deprecation mechanism, where you *can* call deprecated functions,
+although it's not recommended.  You also have a mechanism to check if
+there are static calls to that function so you don't need to manually
+check if each function is deprecated.
+
+Specification
+=============
+
+Syntax for internally exporting a function would be the following:
+
+    -internal_export([f/a, ...]).
+
+where `f` is `atom()` and `a` is `arity()`.
+
+Internally exported functions can *legally* be called from within
+their applications and from their dependencies.  The reason why
+the application's dependencies can *legally* call internally exported
+functions are callbacks to other applications<sup>1</sup>.
+
+Example
+-------
+
+Application `A` contains module `mod_A` which exports `public/1`
+and internally exports `internal/1`.
+
+Application `B` contains module `mod_B` which exports `x/1` and
+internally exports `y/1`.
+
+Application `B` depends on application `A`.
+
+`mod_A:internal/1` can *legally* be called only from within `A`,
+while `mod_B:y/1` can be *legally* called from both `A` and `B`.
+Static call from `A` to `mod_B:y/1` will probably never occur,
+because `A` is not aware of `B`, but it could be that `mod_B:y/1`
+is a callback implementation for `mod_A`.  All the other applications
+can't *legally* call `mod_A:internal/1` and `mod_B:y/1`.
+
+[EEP 5][] Modifications
+=======================
+
+Although the idea is similar, there are a few key differences in design and
+implementation between this EEP and [EEP 5][]:
+
+Application-based approach
+--------------------------
+
+There is an open question of whether to export functions to modules or
+applications.  Exporting to modules is currently proposed in [EEP 5][].
+
+Although the module-based approach is straightforward, it has 2 drawbacks:
+
+1. Maintainance - you have to manually intervene after a module is
+added or renamed.
+2. Exporting to behaviors - you would expect that
+
+        -export_to(gen_server, [init/1, handle_cast/2, handle_call/3]).
+
+would work.
+
+It would be very convenient to export callbacks this way instead of
+
+        %% gen_server API
+        -export([init/1, handle_cast/2, handle_call/3]).
+
+, but that notation would actually be wrong because calls to `gen_server`
+callbacks are done in `gen` module, not `gen_server`.
+Users shouldn't even think about that implementation detail, let alone
+writing code dependent on it.
+
+Implicit scope
+--------------
+
+Currently, comment-based is a widespread approach to clarify
+module/application API, e.g.:
+
+    %% gen_server
+    -export([init/1, handle_cast/2, handle_call/3, code_change/3]).
+    %% system calls
+    -export(...).
+    %% test-only
+    -export(...).
+    %% API
+    -export(...).
+    %% internal exports
+    -export(...).
+
+This of course works, but a comment-based semantical separation of
+exports limits any usage of code analysis tools.  Can we do better?
+
+In [EEP 5][], the responsibility of declaring export scope is on
+the programmer.  This may result in convenient, more semantically
+valuable code, like:
+
+    -export_to([mod_x, mod_y], [f_1/1, f_2/0]).
+
+It denotes what is the purpose of some particular export, i.e.
+which modules can call them.  Notice that this is only convenient when
+limiting function usage to the application itself (conventionally
+called *internal exports*) - it was mentioned before that this approach
+is not appropriate when exporting functions to behavior modules.
+
+This brings up a question: how much control do we really need/want?
+This EEP aims to limit ways users can misuse applications/modules.
+**Missuage doesn't come from within the application itself, or from
+its dependency applications, but from its users!** I.e. besides private
+and global, there is a need for *internal* export scope.
+
+There is no need for the user to specify the scope manually as it can be
+determined at compile-time: internally exported function can be called
+from the application where it is declared or from any of its
+dependency applications.
+
+Implicit scoping is not as semantically valuable as explicit one,
+but it does the main task; **it separates application/module public
+API from internal stuff**.
+
+Syntax
+------
+
+The `export_to` attribute seems completely logical when used with explicit
+scoping, but with implicit scoping, it is unclear to what *_to* refer.
+Name `internal_export` is proposed instead of `export_to`.
+Notice that it only has one argument with the same syntax rules
+as for `export`.
+
+Implementation strategy
+-----------------------
+
+In [EEP 5][], calls would checked in the loader (for static calls) and at
+runtime (for dynamic calls), causing them to fail if an invalid call occurs.
+In contrast, this EEP suggests a code-analysis-based approach, with
+`xref` checking all static calls and disregarding all dynamic calls.
+Calls to all functions remain valid at runtime.  This approach,
+of course, provides no runtime guarantees, but imposes no performance
+hit and requires significantly less work and maintenance.
+
+Reference Implementation
+========================
+
+The current implementation is in [PR 7407][2] in the OTP repository.
+
+Besides this PR, there are forum threads about
+[the need for internal exports][3] and [the implementation][4].
+
+Backward compatibility
+======================
+
+Code that is already using `-internal_export(FAs).` attribute
+would be affected.
+
+References
+==========
+
+[EEP 5]: eep-0005.md
+         "EEP 5: More Versatile Encapsulation with `export_to`"
+
+[1]: Even though they never get called statically, it is semantically
+important to denote those callbacks can *legally* be used from dependency
+applications.  If someday dynamic checks become a thing mechanism
+could stay the same.  That said, the current mechanism can be simplified
+and only check if internally exported is used only from within the
+same application.
+
+[2]: https://github.com/erlang/otp/pull/7407
+     "PR 7407: Feature: internal_export"
+
+[3]: https://erlangforums.com/t/some-thoughts-on-callbacks/1851
+    "Forum thread: Some thoughts on callbacks"
+
+[4]: https://erlangforums.com/t/feature-internal-export/2697/18
+    "Forum thread: Feature: internal_export"
+
+Copyright
+=========
+
+This document is placed in the public domain or under the CC0-1.0-Universal
+license, whichever is more permissive.
+
+[EmacsVar]: <> "Local Variables:"
+[EmacsVar]: <> "mode: indented-text"
+[EmacsVar]: <> "indent-tabs-mode: nil"
+[EmacsVar]: <> "sentence-end-double-space: t"
+[EmacsVar]: <> "fill-column: 70"
+[EmacsVar]: <> "coding: utf-8"
+[EmacsVar]: <> "End:"
+[VimVar]: <> " vim: set fileencoding=utf-8 expandtab shiftwidth=4 softtabstop=4: "