Proposal for the future of variables

[sovdeeth]

Existing Behavior

Currently, variables are either lists or not lists. Not lists have a single value and a name that ends in something that isn't ::*. Lists have multiple values and end in ::*, where each value is accessible as a 'not-list' element. All not-lists are accessible directly via hash-map, making individual element access fast. Lists are represented as tree maps, allowing good traversal and organization, but slightly lacking on the list aspect.

This means lists are, in effect, String-Object maps, ordered by their keys (0-9, a-z). It also causes ambiguity around not-lists and lists with the same name, excluding the ::*. {a::*} and {a} represent two entirely different things, despite sharing a name. It also means that Skript is rather limited in variable data structures.

Goals

This proposal aims to unshackle variables from the concepts of lists and not-lists, paving the way for more varied data types and more intuitive and scalable structures. It aims to provide a structure for lists, queues, maps, and many more such collections to be integrated into Skript variables in a uniform and minimally breaking manner. Ideally, code interfacing with the existing Variables object need not make changes.

It does not aim to change the format of variables for the user, introduce new syntaxes, or provide any methods for users to make structures.

Proposal

This idea revolves around the concept of an access operator. The most relevant one is the subscript operator in most langauges: variable[0]. The key here is turning :: from a hardcoded list separator into a more flexible access operator. array[0] becomes {array::0} in Skript. The Python slice operator, which returns all the elements in a list, list[:], would be {list::*}. This, so far, is the exact same behavior Skript has already had, just a different way of approaching it.

1. :: is now a flexible access operator that can act on things other than lists.

It has several implications that we can take advantage of, though. Since it's no longer hardcoded for a list, it means that {array::1} is actually acting on {array}. This implies that {array} itself is a list or some other collection. This is a departure from Skript's current behavior, and has a few knock-on effects.

2. Lists and other collections are now singular objects which can be stored in a single variable.
3. {list::array::*} and {list::array} are no longer completely separate objects, the latter is the collection, the former is the values within the collection.
4. {list::*}, in the above concept, no longer has to have indices without values. It now has indices and values that are the sublists themselves. This is a major source of confusion for newer users that this proposal would solve.

The addition of an access operator also means that it can act on any object that allows it. This means any sort of data type can allow access, and you can have proper types like arrays, maps, queues, and what have you all acting the same:

5. Many data structures can be used without bespoke accessing syntax.

set {map} to a map # or something similar
set {map::0} to "a"
set {map::a} to "b"
set {array} to an array of "a"# or something similar
add "b" to {array}
send {map::*} # sends "a and b"
send {map} # sends "Map: (0: a, a: b)" or something
send {array::*} # sends "a and b"
send {array} # sends "Array: (a, b)" or something

We could take advantage of this to abandon the current concept of a list and instead support maps (string -> object) and dynamic arrays of objects, removing the weird behavior of current lists to sort by their indices and generally just act like a map instead of a list.

6. Data structures can be passed in and out of functions without losing information. Since you can now pass entire structures, rather than arrays of values, maps can be returned without losing indices and copied without bespoke copy syntax.

set {map} to a map # or something similar
set {map::1} to 3
set {map::a} to "b"
return {map}
# ...
set {map} to function()
send {map::a} # sends "b"

7. The * access operation could be applied to any layer. Currently, * is only allowed at the end of a list access, but using an access operator style would allow {list::*::value} to get all matching values, or even {list::*::*} to get all 2nd-level values.

Challenges

Despite the many upsides, there are breaking changes and challenges here.

1. Not all data structures allow indiscriminate access. Some may only allow ::* or specific indices like the first and last, like queues, or only allow numbers as indices, like arrays. The operator and types will have to have communication granular and flexible enough to know these restrictions and properly error when they're broken.

2. Compatibility with existing internal logic. To avoid breaking every single syntax in Skript, we still need to put arrays of values in and get arrays of values out. This means that the ::* operator is going to have to be implemented in every collection. It also leads onto:

3. Confusion between when and where users can use {array} vs {array::*}. Since nearly all of Skript's syntax will be collection agnostic, ::* will still be used whenever a plural type input is asked for: players/displays. However, collection-specific syntax will be asking for array or arrays, and they may get confused that first element of {array::*} works, but head of {queue::*} doesn't. Above I showed add 1 to {array}. Should this add 1 to the collection, or add 1 to each value within the collection? What about add 1 to {array::*}? (I think the former should add to the collection, the latter should add to each value, though this is a major breaking change). We will have to decide on the standards around these kinds of interactions.

4. Quick variable access. Currently list elements can be accessed instantly via a single hashmap lookup. Adding in layers of access operations could feasibly slow this down significantly. On the flip side, it also means that lists can be pulled out as local variables, rather than accessing from the root every time. Performance will have to be kept in mind for this.

Future Possibilities

If such a system is implemented, it also opens the gate for typed collections, which only accept certain types, like player array. This would require some changes to SkriptPattern types, but theoretically a syntax could ask for a %type collection%, like %item/slot collection%, or %item/slot array/queue%.

It also allows addons to register their own types, and opens the door for SkriptReflect to allow custom data structures as a feature.

---

Feel free to comment your opinions, suggestions, concerns, or criticisms!

[erenkarakal]

i like the idea and would rather it was like this from the start but i think its too late to do it now and its not really a must have

[Asleeepp]

I think this would be a nice change, but its way too big of a breaking change to be implemented imo

[sovdeeth]

It'd definitely be a skript-3 sort of change, though it's not that breaking for most things. The main issues would be with changers acting on lists, like set {list::*} to 1, 2, 3 would probably have to sneakily act like set {list} to a list of 1, 2, and 3. I think it could be implemented with rather limited breaking changes, all in all.

[Fusezion]

as always sorry for bad wording

Similar to eren I like the idea, as you said would need to be a skript-3 type of addition my main concern is the handling of collection type when defining a variable using code you provided

set {map::0} to "a"
set {map::a} to "b"
set {array::1} to "a"
add "b" to {array}

in the first part you did set {map::0} to "a" so if I were to use add "b" to {map} instead of the next set would this not now be treated as an array? Especially with the fact it uses a number as the index, in that case would {map} by default have no type or would you guess if you're unsure? If I as the user set the value to a string or some other object it can safely be guessed as a map but if I set it to a number it could be a map or an array.

For myself and for others to not basically have to relearn the a core part of the design I would prefer a path similar to what was mentioned in the queues pr kind of, where we would need to first do set {foo} to [a] new %collectiontype% collection such as set {map} to a new map collection this value is then known to be a map and any value passed is expected to be matching a map.

At the same time if skript is now aiming to implement collections in a manner that breaks out of old design should skript now use 0 as a starting index instead of 1, while we can argue 1 is more understood by new people the design is not actively targeting new players.

An example of that is your usage of 1 in the above code but then -1 and 0 in your challenges section if I'm to interoperate that as -1(backwards) and 1 (forward) then 0 is an unknown value with no definition in skript

[sovdeeth]

Those are all disparate ideas about how an access operator can work, none are hard requirements. For example, in python [-1] refers to the last element in a list.
I also agree with the point about a new whatever; the examples i put above assumed the maps/arrays already existed rather than that they were being created right then. I suppose I should edit it to clarify that.

[sovdeeth]

Hopefully those edits clear it up a bit

[JakeGBLP]

In my opinion the proposed changes sound very good, but I wonder what will be of anyone using Skript on versions not supported by the update that would include them, perhaps 2.x and 3.x should keep a similar (if not equal) api and let 2.x be supported for a while to allow users to transition to the newer system or allow this new variable list/map system to be disabled from the config, I know, a hassle.

[TheLimeGlass]

It's good. {map} should always reference a map. But how are you gonna do single values now? There should be another pattern for single values. Maybe like Kotlin definition or usage of [ and ] since they're only used for filtering. Or just set variable example to or variable example = or variable example:

[sovdeeth]

Single values wouldn't change at all? I'm not sure how this changes them.