Unable to build packages assert next_token == "("

[alistair23]

When trying to build any toltec package I am seeing this error:

$  make koreader
./scripts/package_build.py  "koreader"
Traceback (most recent call last):
  File "/var/home/alistair/sources/remarkable/toltec/./scripts/package_build.py", line 44, in <module>
    repo = Repo(paths.RECIPE_DIR, paths.REPO_DIR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/repo.py", line 56, in __init__
    self.generic_recipes[name] = GenericRecipe.from_file(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 45, in from_file
    return GenericRecipe(name, path, recipe.read())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 58, in __init__
    variables, functions = bash.get_declarations(definition)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/bash.py", line 147, in get_declarations
    assert next_token == "("
AssertionError
make: *** [Makefile:60: koreader] Error 1

It doesn't matter what package I build, I see the same error.

$ python3 --version
Python 3.11.0

bash --version
GNU bash, version 5.2.2(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[Eeems]

Have you tried with python 3.8
https://github.com/toltec-dev/toltec/blob/stable/.github/workflows/pr.yml#L31

[matteodelabre]

I think this has more to do with the bash version, since the error is coming from the bash “parser”. It works on my computer running Bash 5.1.16, perhaps they changed their output format in 5.2.2? @alistair23 Could you add a print(declarations) after line 123 of scripts/toltec/bash.py and give us some of the output? I get the following output for one recipe:

declare -- LINENO
declare -- MACHTYPE="x86_64-pc-linux-gnu"
declare -x OLDPWD
declare -- OPTERR="1"
declare -i OPTIND="1"
declare -- OSTYPE="linux-gnu"
declare -x PATH="..."
declare -a PIPESTATUS=([0]="0")
declare -ir PPID="4130"
declare -- PS4="+ "
declare -x PWD=".../toltec"
declare -i RANDOM
declare -- SECONDS
declare -- SHELL="/bin/zsh"
declare -r SHELLOPTS="braceexpand:hashall:interactive-comments"
declare -x SHLVL="1"
declare -i SRANDOM
declare -- TERM="dumb"
declare -ir UID="1000"
declare -- _="-f"
declare -- arch="rmall"
declare -a flags=([0]="patch_rm2fb")
declare -- image="python:v2.1"
declare -a installdepends=([0]="display")
declare -- license="MIT"
declare -- maintainer="raisjn <[email protected]>"
declare -a makedepends=()
declare -a noextract=()
declare -- pkgdesc="Wordle clone"
declare -- pkgname="wordlet"
declare -a pkgnames=([0]="bufshot" [1]="dumbskull" [2]="genie" [3]="harmony" [4]="iago" [5]="lamp" [6]="mines" [7]="nao" [8]="remux" [9]="rpncalc" [10]="simple" [11]="wordlet")
declare -- pkgver="0.0.1-2"
declare -- section="games"
declare -a sha256sums=([0]="e7f0379d5ad9de61c32a296ab831780cfea73352fbccfe2472422165943457ef" [1]="SKIP" [2]="SKIP")
declare -a source=([0]="https://github.com/rmkit-dev/rmkit/archive/070d5df29a71e55050a5e993f047f1590d69ec1c.zip" [1]="remux.service" [2]="genie.service")
declare -- timestamp="2022-06-23T20:03:10Z"
declare -- url="https://rmkit.dev/apps/wordlet"

build ()
{
    cmake -B build -DCMAKE_TOOLCHAIN_FILE="/usr/share/cmake/$CHOST.cmake" -DCMAKE_INSTALL_PREFIX="/opt";
    cmake --build build
}
package ()
{
    install -D -m 755 -t "$pkgdir"/opt/bin "$srcdir"/build/vnsee "$srcdir"/vnsee-gui;
    install -D -m 644 -t "$pkgdir"/opt/etc/draft/icons "$srcdir"/vnsee.png;
    install -D -m 644 -t "$pkgdir"/opt/etc/draft "$srcdir"/vnsee.draft;
    install -D -m 644 -t "$pkgdir"/opt/usr/share/applications "$srcdir"/VNSee.oxide
}
prepare ()
{
    patch -d "$srcdir" -p1 < "$srcdir"/vnsee-gui-set-path.patch;
    bsdtar --strip-components 1 -xf "$srcdir"/"$_boost_pp_ver".zip -C "$srcdir"/vendor/boost-preprocessor
}

[alistair23]

This is the output

$ make koreader
./scripts/package_build.py  "koreader"
build () 
{ 
    rm .cargo/config;
    cargo build --release --bin appmarkable
}
package () 
{ 
    install -D -m 755 -t "$pkgdir"/opt/bin "$srcdir"/target/armv7-unknown-linux-gnueabihf/release/appmarkable
}
declare -- BASH="/usr/bin/bash"
declare -r BASHOPTS="checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath"
declare -i BASHPID
declare -A BASH_ALIASES=()
declare -a BASH_ARGC=()
declare -a BASH_ARGV=()
declare -- BASH_ARGV0
declare -A BASH_CMDS=()
declare -- BASH_COMMAND
declare -a BASH_LINENO=()
declare -- BASH_LOADABLES_PATH="/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:."
declare -a BASH_SOURCE=()
declare -- BASH_SUBSHELL
declare -ar BASH_VERSINFO=([0]="5" [1]="2" [2]="2" [3]="1" [4]="release" [5]="x86_64-redhat-linux-gnu")
declare -- BASH_VERSION="5.2.2(1)-release"
declare -- COMP_WORDBREAKS
declare -a DIRSTACK=()
declare -- EPOCHREALTIME
declare -- EPOCHSECONDS
declare -ir EUID="1000"
declare -a FUNCNAME
declare -a GROUPS=()
declare -i HISTCMD
declare -- HOSTNAME="als-thinkpad-x1"
declare -- HOSTTYPE="x86_64"
declare -- IFS=$' \t\n'
declare -- LINENO
declare -- MACHTYPE="x86_64-redhat-linux-gnu"
declare -x OLDPWD
declare -- OPTERR="1"
declare -i OPTIND="1"
declare -- OSTYPE="linux-gnu"
declare -x PATH="/var/home/alistair/.cargo/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/var/home/alistair/.local/bin"
declare -a PIPESTATUS=([0]="0")
declare -ir PPID="39382"
declare -- PS4="+ "
declare -x PWD="/var/home/alistair/sources/remarkable/toltec"
declare -i RANDOM
declare -- SECONDS
declare -- SHELL="/bin/bash"
declare -r SHELLOPTS="braceexpand:hashall:interactive-comments"
declare -x SHLVL="1"
declare -i SRANDOM
declare -- TERM="dumb"
declare -ir UID="1000"
declare -- _="-f"
declare -a flags=([0]="patch_rm2fb")
declare -- image="rust:v2.1"
declare -a installdepends=([0]="display")
declare -- license="MIT"
declare -- maintainer="Linus K. <[email protected]>"
declare -- pkgdesc="Front-end for apps that do not have a graphical user interface"
declare -a pkgnames=([0]="appmarkable")
declare -- pkgver="0.0.0-11"
declare -- section="devel"
declare -a sha256sums=([0]="76e151aeae0f18b206dd3c6258bf74bcb5256ee2f803e1ed2073278831158f60")
declare -a source=([0]="https://github.com/LinusCDE/appmarkable/archive/c44ee87ea2b1f1e41c9592476c076150c9a1acf4.zip")
declare -- timestamp="2021-03-10T18:36Z"
declare -- url="https://github.com/LinusCDE/appmarkable"

Traceback (most recent call last):
  File "/var/home/alistair/sources/remarkable/toltec/./scripts/package_build.py", line 44, in <module>
    repo = Repo(paths.RECIPE_DIR, paths.REPO_DIR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/repo.py", line 56, in __init__
    self.generic_recipes[name] = GenericRecipe.from_file(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 45, in from_file
    return GenericRecipe(name, path, recipe.read())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 58, in __init__
    variables, functions = bash.get_declarations(definition)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/bash.py", line 149, in get_declarations
    assert next_token == "("
AssertionError
make: *** [Makefile:60: koreader] Error 1

with this diff applied:

diff --git a/scripts/toltec/bash.py b/scripts/toltec/bash.py
index cb194e2..3617024 100644
--- a/scripts/toltec/bash.py
+++ b/scripts/toltec/bash.py
@@ -122,6 +122,8 @@ declare -p
 
     declarations = declarations_subshell.stdout.decode()
 
+    print(declarations)
+
     # Parse `declare` statements and function statements
     lexer = shlex.shlex(declarations, posix=True)
     lexer.wordchars = lexer.wordchars + "-"

[matteodelabre]

Thanks! After some investigation, the problem seems to stem from the way the IFS variable contents are printed by bash. Our primitive parser doesn’t know how to handle C-quoted strings, i.e., string literals that have a $ sign in front of them.

[matteodelabre]

shfmt’s upcoming 3.6.0 release will include both a way to build an AST for a Bash program (--to-json), and a way to generate a Bash program from an AST (--from-json). I think I’ll replace our primitive parser with that.

[alistair23]

3.6 has been released, is it possible to update to use that?

[Eeems]

@alistair23 We would happily accept a PR

[Eeems]

At some point we need to switch over to using toltecmk to build the repo instead of maintaining two versions of the build code.

[Eeems]

@alistair23 you indicated that you are still having issues with c-quoted strings when using the latest toltecmk version? Could you provide some further context?

[alistair23]

Yeah, so running make doesn't work

$ make linux-mainline
./scripts/package_build.py  "linux-mainline"
Traceback (most recent call last):
  File "/var/home/alistair/sources/remarkable/toltec/./scripts/package_build.py", line 44, in <module>
    repo = Repo(paths.RECIPE_DIR, paths.REPO_DIR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/repo.py", line 60, in __init__
    self.generic_recipes[name] = GenericRecipe.from_file(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 46, in from_file
    return GenericRecipe(name, path, recipe.read())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 59, in __init__
    variables, functions = bash.get_declarations(definition)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/bash.py", line 147, in get_declarations
    assert next_token == "("
           ^^^^^^^^^^^^^^^^^
AssertionError
make: *** [Makefile:60: linux-mainline] Error 1

Running toltecmk directly does work though

[Eeems]

Right, totlec doesn't use toltecmk yet, which was the comment I left. We still have to port the changes over to this repo. It's good to hear that toltecmk itself is working, though.