Rethink fn arguments list

[gilch]

Hy currently has &optional, &key, &kwargs, &rest, and &kwonly.

Python does not seem this complicated. It only has * and ** to remember. Clojure only has &, but it can use that with its dict destructure to emulate kwargs. It's an elegant solution, but I don't think it's good enough for Hy, which needs to interoperate closely with Python.

To be fair, Hy is not any worse than Common Lisp's &allow-other-keys &environment &rest &aux &key &whole &body &optional.

Is there any way we can simplify this? Any simplification here would be a very deep change, and bound to be controversial. But it's easier to build advanced macros on simpler, more consistent special forms. The time for breaking changes is sooner, not later.

I think the special form should follow Python as closely as possible. I don't know that I ever use &key, but that kind of thing probably belongs in a macro built on top of fn, not in the special form itself.

If we re-do #* and #** to create their own Hy models, as discussed in hylang/hyrule#52, then we could also use these in the arguments list. This would make Hy more consistent with Python, and with its own other features that use that syntax.

But that would only take care of &rest and &kwargs. Using #*, we can't distinguish &rest from &kwonly without commas. And how do we indicate default arguments without an infix =? It can't just be a list without &optional, or it looks like the destructuring argument bind we still have from Python2. It's also nice to be able to get a None default without typing it explicitly.

So how else could this work?

Option 1

Maybe from this:

(defn foo [a &optional ["b" 1] [c :c] d &rest args &kwonly x [y 2] &kwargs kwargs])

To this:

(defn foo [a  :b 1  :c ':c  :d None  #* args  x  :y 2  #** kwargs])

And the equivalent Python.

def foo(a, b=1, c=HyKeyword("c"), d=None, *args, x, y=2, **kwargs): pass

Here we're using :keyword pairs to stand for =. This makes it look like a Hy function call, which is more consistent and easier to remember. Note that we have to quote keywords we want to pass as data, so they aren't interpreted as a kwarg name. But function calls work the same way. (And given the strange way our keywords work currently #1352, it would actually expand to c='\ufdd0:c'. We might want to change this.)

It seems a lot easier to work with, both for humans, and in macros. There aren't two ways to do the same thing, like &optional and &key. But we don't have our implicit None from &optional anymore. Oh well, neither did Python. We could implement it with a macro later, maybe.

And how should &kwonly work? We can do [#* _ x y], but that's not the same. It silently ignores extra arguments instead of raising an error. One possibility is a new Hy model named #*,, which you'd use instead of #* in those cases. That seems conceptually the same as &kwonly though. Is there a more elegant way to do this?

And finally, what about destructuring something with a default? (e.g. def foo((a, b)=[1, 2]): pass) Actually, Hy can't do this! #1410. And neither can Python3. No loss. Except, I'd like to add this feature in a fn=: macro #1410, #1328. What should it look like? I'm not sure.

Maybe like (defn=: foo [ #:(, a b) [1 2]])? This (ab)uses the one of the (Hypothetical) slice notations I proposed from #541.

[gilch]

Option 2

(defn foo [a {b 1  c :c  d None} args & x {y 2  z 3} kwargs])
(defn akw [{} args & {} kwargs])
(defn only [& x {y 2}])

def foo(a, b=1, c=HyKeyword("c"), d=None, *args, x, y=2, z=3, **kwargs): pass
def akw(*args, *kwargs):pass
def only(*, x, y=2):pass

Here, optional args go in {} after the required positional args (like using &key). The *args part is assumed to follow that. So (defn foo [{} args]) means def foo(*args).

The & now splits the positional args from the kwargs. So anything following & is kwonly. It similarly uses {} for anything optional. So (defn foo [& x {y 2}]) means def foo(*, x, y=2):pass.

And finally the kwargs follows that dict. So (defn foo [{& {} kwargs]) means def foo(**kwargs):pass, and (defn foo [{} args & {} kwargs]) means def foo(*args, **kwargs):pass. Not as nice as Python, but shorter than what we have now: (defn foo [&rest args &kwargs kwargs]).

This gives macros more (and more consistent) structure to work with. I'm not sure how fn=: should look here either, because it's already using brackets a lot.

This seems to handle &kwonly more elegantly than Option 1, but I don't think it's pretty overall.

[gilch]

I think it's better to save the brackets for macro destructuring syntax. Can we avoid them in the special form?

Option ?

(defn foo [a  b 1  c 'c  d None  #* args  x 1  y  z 3  #** kwargs]])
(defn akw [#* args #** kwargs])
(defn only [#*, x  y 2])

def foo(a, b=1, c=HySymbol("c"), d=None, *args, x=1, y, z=3, **kwargs): pass
def akw(*args, *kwargs):pass
def only(*, x, y=2):pass

This version implies defaults by position and Hy model type. Given a symbol foo, if the next element isn't a symbol, it's the default value for foo. You can have symbols as default values by quoting them. #* and #** tag args and kwargs, like Python. & indicates kwonlys, which also have the implied defaults.

We'll probably get rid of the destructuring in the special form. #1410. But what about fn=:? It will have to detect what's a value and what's a target by searching the model at expansion time. If it has any non-quoted symbols, it's a target, e.g. in [q [a b] ['a 'b]], the [a b] part is a target, not the default for q.

[Edit: that's not good enough, since we might want to use a non-quoted symbol as a default to refer to some other value in the environment. [foo ((fn[] bar))] is too awkward for such a common case.]

[gilch]

Option 3

(defn foo [a & b 1  c :c  d None #* args x & y 2  z 3 #** kwargs])
(defn akw [#* args #** kwargs])
(defn only [#*, x & y 2])

In this version, & separates required args from optional args, both for positional and keyword. Kwonly args have to follow the *, as in Python, with #*, for an omitted *args, like Option 1.

[gilch]

Option 4

(defn foo [a &optional b 1  c :c  d None &rest args &kwonly x &kwoptional y 2  z 3 &kwargs kwargs])
(defn akw [&rest args &kwargs kwargs])
(defn only [&kwonly x &kwoptional y 2])

Like Opiton 3, but with familiar &words. Each &word clause is optional, but always goes in that order, like Python.

Why is this any better than the status quo? It looks pretty similar.
There should be one-- and preferably only one --obvious way to do it.
Now any given arguments list has a single canonical form, up to the kwonly ordering, which I think doesn't matter. (If it does, then it'll be hard to avoid brackets--but see Option 1, because this version requries the required kwonly to all appear before the optional kwonly. Python does not require this. But I don't think it changes the meaning of the program.)

It avoids the special cases of #* vs #*,. &optional no longer uses brackets, avoiding another special case of foo vs [foo None]. &key no longer exists, but &optional's args now looks like &key's, but without the extra {} that didn't do anything, another special case (&optional vs &key) avoided.

The single canonical form makes it much easier to deal with in macros.

[gilch]

It looks like the ordering of kwonlys is inspectable. So decorator metaprogramming could potentially use that to mean something important. Hy has a lot of edge cases like this that should be corrected, but let's not make it worse. We're back to Option 1, the only option so far without this problem.

[gilch]

Option 5

(defn foo [a &optional b 1  c :c  d None  &rest args &kwonly x .  y 2  z .  &kwargs kwargs])
(defn akw [&rest args &kwargs kwargs])
(defn only [&kwonly x .  y 2  z .])

&kwonly now implies pairs. The . indicates no default. It's not allowed in calls anyway, so this should be fine. It's also no harder than typing brackets.

[gilch]

Option 6

(defn foo [a &? b 1  c c  d None &* args x .  y 2  z . &** kwargs])
(defn akw [&* args &** kwargs])
(defn only [&* . x .  y 2  z .])

def foo(a, b=1, c=c, d=None, *args, x, y=2, z, **kwargs): pass
def akw(*args, **kwargs): pass
def only(*, x, y=2, z): pass

&? replaces both &optional and &key. It implies pairs. The explicit None is required for optional args, as in Python.

&* replaces both &rest and &kwonly.

• The first symbol following &* is the *args. You may omit the *args with a ..
• The remainder are kwonly pairs. Use the . to omit the default for required kwonlies.

&** replaces &kwargs.

I like this version best so far, and will probably implement fn=: using it.

Maybe we could use something other than &?, &*, and &**, but I don't think it will get any clearer.

[gilch]

Option 6 b

Use ., #*, and #**, instead of &?, &*, and &**, respectively. Probably requires #* and #** to have Hy models. hylang/hyrule#52.

Option 6 c

Use . instead of &?, and change #*/#** to &/&& in all contexts.

[vodik]

I personally started using kwonly arguments without default liberally, and making those harder to understand at a glance does the feature a misservice, imho (requiring the extra .).

Also think its weird that &* can, depending on context, mean either variable number of positional arguments or keyword only arguments. Technically also a problem with Python, but its much easier to distinguish

You also really should be forward looking a bit here as there's a good chance positional only arguments will come. They're already a thing in the CPython core, just without syntax for Python to leverage this - for now. It'll likely look like this def foo(a, /, b): .... You can already see it in use in IPython signatures:

In [1]: range?
Init signature: range(self, /, *args, **kwargs)

I think we really only need a few things:

• A way to specify a default value
• A way to separate positional-only from positional-or-keyword args
• A way to separate positional-or-keyword args from keyword-only args
• A way to specify *args
• A way to specify **kwargs

Building on Option 6, embracing tuples for optional arguments, as they're already done, does simplify things. Then &* can be reserved for keywork-only and &/ for positional only.

And technically, we don't really have to allow the end user to choose the variable name for *args and **kwargs 😉

(defn foo [a [b 1] [c c] [d None] &args x [y 2] z &kwargs])
(defn akw [&args &kwargs])
(defn only [&* x [y 2] z])

And to add, it would be easy to add positional only after the fact:

(defn my-range [self &/ &args &kwargs])

[scauligi]

Necro'ing this as part of a rethinking of :keywords in general.

I'd like to move forward with @vodik's proposal, with some minor tweaks:

(defn foo [i j &/ a [b 1] [c c] [d] &args x [y 2] z &kwargs])
(defn akw [&args :as vs  &kwargs :as kws])
(defn only [&* x [y 2] z])

;; also, a new proposed calling syntax:
(foo "i" "j" "a" "b"  &  x "ecks"  z "zed"  d "dee"  other "extra thing")

def foo(a, b=1, c=c, d=None, *args, x, y=2, z, **kwargs): pass
def vkw(*vs, **kws): pass
def only(*, x, y=2, z): pass

foo("i", "j", "a", "b", x="ecks", z="zed", d="dee", other="extra thing")

This makes function defs/calls have a syntax closer to some of the other
binding forms such as setv or for — which makes sense, since we're
binding values to symbols after all.
It also uses the :as keyword from the import/require syntax if someone really wants to change the common-case names for args and kwargs.

Otherwise, like @vodik's proposal, we're just reusing python's existing * and / syntax for demarcating the arguments list.

In addition, I'd like to propose using symbols as symbols instead of :keywords
as symbols for function calls, as it can lead to confusing behavior (#1765
#1974). It should also make issues such as #1780 easier, as assignment/binding
will only happen to names given via HySymbols.

The new calling syntax should be unambiguous since (just as with python) positional arguments must always come before keyword arguments.

[gilch]

The "new" proposed calling syntax is pretty close to how Hissp does it. Except Hissp uses : as the separator instead of &.

[Kodiologist]

(foo "i" "j" "a" "b" & x "ecks" z "zed" d "dee" other "extra thing")

I'm not a fan. The parameter names look too much like expressions, and it will no longer be possible to intermingle positional and keyword arguments.

[scauligi]

The "new" proposed calling syntax is pretty close to how Hissp does it

I was not aware of Hissp! Good to know.

The parameter names look too much like expressions, and it will no longer be possible to intermingle positional and keyword arguments.

Would it be better to use bracketed pairs after the keyword marker?

(foo "i" "j" "a" & [x "ecks"] [z "zed"] "b" [d "dee"] [other "extra thing"])

Ultimately I'd like to get rid of using HyKeywords for python keyword arguments for the linked reasons above, so that they're not pulling double duty as both control words and symbol names.

[Kodiologist]

No, in my opinion, that's quite noisy, and I don't know what you mean by "pulling double duty as both control words and symbol names". Constructing keyword arguments is the only real purpose of HyKeyword.

[allison-casey]

I also quite like keywords in function calls visually, but would it be possible then to only allow keywords in function calls? Essentially codifying the fact that they only exist to construct keyword arguments?

I feel like a lot of the busy work of hy fn defs would be solved by just getting rid of &optional in favor of the bare [sym default] scauligi proposed. We also need to support pos only arguments as well and aligning closer to python with &/ feels like a good idea. I'll play around with the complier some with some the proposed syntaxes

[Kodiologist]

but would it be possible then to only allow keywords in function calls?

In Lisp, we try to allow the use of code as data as much as possible. So, that seems counterproductive.

&optional is probably unnecessary, yeah.

[allison-casey]

makes sense.

what about this for fn args:

(defn test [a b &/ c [d 1] &* e [f 2] [&** kwargs]])

would compile to:

def test(a, b, /, c, d=1, *, e, f=2, **kwargs):

Instead of :as for args and kwargs we use the same bracket syntax we use for default args and we completely do away with the idea of &optional and &kwonly?

[Kodiologist]

Not bad. I would say we should use / and * instead of &/ and &*, and specify args and kwargs with #* args and #** kwargs (without brackets), as when calling.

[allison-casey]

then how would you identify kwonly args from the rest assignment?
(defn test [#*a b]): a could be a keyword only argument without a default or the assignment of the rest var.
[#* rest] disambiguates what is the assignment from the following kwonly args

[Kodiologist]

#*a parses as (unpack-iterable a). The symbol *, not just the character anywhere, would be what marks the start of keyword-only arguments.

[allison-casey]

That feels like it lacks clarity to me and is inconsistent with the existing bracket syntax.
I can imagine getting rid of the leading &, but the visual demarcation of the bracket feels easier to read to me
since i'm already looking for them with default args
(defn test [a #* rest b])
(defn test [a * b])
(defn test [a [* rest] b])
(defn test [a * b])
Your version would be consistent with how #* and #** are used outside of the fn form though, which could be nice.

[Kodiologist]

It makes sense to me that binding args and kwargs looks different from default arguments, because it's not really analogous.

[scauligi]

So to be clear, we're considering this for the latest syntax?

(defn foo [i j / a [b 1] [c c] [d] #* args x [y 2] z #** kwargs])
(defn akw [#* vs  #** kws])
(defn only [* x [y 2] z])

(foo "i" "j" "a" "b"  :x "ecks"  :z "zed"  :d "dee"  :other "extra thing")

def foo(a, b=1, c=c, d=None, *args, x, y=2, z, **kwargs): pass
def vkw(*vs, **kws): pass
def only(*, x, y=2, z): pass

foo("i", "j", "a", "b", x="ecks", z="zed", d="dee", other="extra thing")

[Kodiologist]

Yes, except I wouldn't implement [d] as shorthand for [d None]; it's a little odd and seems unlikely to be useful.

[allison-casey]

It makes sense to me that binding args and kwargs looks different from default arguments, because it's not really analogous.

Good point. I think I'm happy to concede at this point.
And just to be explicit:

• / demarks the end of positional only arguments
• * demarks the start of keyword only arguments
• #* <sym> collects varargs into <sym> and also marks the start keyword only arguments
• #** <sym> collects keyword varargs into <sym>
• ^ann <sym> demarks a positional argument
• ^ann [<sym> <default>] demarks a keyword argument
  and there is no more shorthand for a keyword defaulting to None

Also, I almost have a working prototype, i'll let y'all know hwo that turns out

[allison-casey]

how should annotations for varargs and kwargs be specified?
^int #* args or #* ^int args

[scauligi]

^int #* args reads better to me.

[allison-casey]

In that case I have something working (the other way would make the special form a lot messier). I'll clean it up and open a pr for y'all to mess around with tonight

=> (fn* [a ^int b / c [d 1] * ^str e ^int [f 2] g ^str #** kwargs])
def _hy_anon_var_4(a, b: int, /, c, d=1, *, e: str, f: int=2, g, **kwargs: str):
    pass