Fix the thread code inclusion problem in Emscripten builds

[ampli]

See issue #1374 .

Remove the Emscripten check from configure and instead check __EMSCRIPTEN__ in the source code.
(For some reason this didn't work for me when I previously handled this problem.)
BTW, AX_TLS already takes care not to defineTLS`.

In addition, git-ignore the file a.wasm that is left after emconfigure ./configure.

Note: This error occurs inside an autoconf macro. I didn't try to find out why.

checking if the linker (/usr/local/src/emsdk/upstream/emscripten/emcc) is GNU ld... yes
em++: error: no input files
checking whether the /usr/local/src/emsdk/upstream/emscripten/em++ linker (/usr/local/src/emsdk/upstream/emscripten/emcc) supports shared libraries... yes

[ampli]

It now fails on this:
wasm-ld: error: 'bulk-memory' feature must be used in order to emit passive segments
This doesn't happen on my computer (I'm using the latest Emscripten from GitHub.)

Which version is getting used in CI?

[linas]

I'll wait a bit before merging this...

[ampli]

I'll wait a bit before merging this...

Strangely, emmake make succeeds on my computer.
I will make changes in this PR to investigate the problem further.

[linas]

Well, the other issue #1361 is like this too: works n my machine, fails in circle-ci. ... and more: when I first spotted it, it seemed there were two different versions of emscripten in circle ci, one passed, one failed... Hm. Maye need to look at the ci config file more carefully? -- This config file: .github/workflows/bindings-js.yml ... but I don't see any issues with it.

[linas]

I looked at .github/workflows/bindings-js.yml I see nothing suspicious there. Don't know how to pick an install version

Try adding --features=bulk-memory to bindings/js/link-parser/build.sh -- that might solve the error message.

[ampli]

From the CI log:

/home/runner/work/link-grammar/link-grammar/bindings/js/emsdk/upstream/emscripten/emcc.py:985: SyntaxWarning: "is not" with a literal. Did you mean "!="?
newargs = [a for a in newargs if a is not '']

My emcc.py, as I got from giא clone https://github.com/emscripten-core/emsdk.git doesn't contain such a line.
Still investigating.

[ampli]

link-grammar/bindings/js/install_emsdk.sh

Lines 1 to 13 in 43d2392

	#!/bin/bash
	set -ex
	pushd "$(dirname "$0")"
	git clone https://github.com/emscripten-core/emsdk.git || true
	pushd emsdk
	git pull
	./emsdk install 1.38.41
	./emsdk activate 1.38.41
	popd
	popd

For some reason it installs emsdk version 1.38.41, which is one of the firs versions. The latest version, as installed by emsdk install latest, is 3.1.30. I don't know why a fixed version got used in the first place.

With 3.1.30, emmake make runs with no errors.
However, I do't know how to check its result.
It seems there is some bitrot in build.sh:

$ ./build.sh                                                                                                                                                                                                                                          LG (em-detect $%|u+6-3)
++ dirname ./build.sh
+ pushd .
/usr/local/src/link-grammar-devel/em-detect/bindings/js/link-parser /usr/local/src/link-grammar-devel/em-detect/bindings/js/link-parser
+ '[' '!' /usr/local/src/emsdk ']'
+ rm -rf dist
+ mkdir dist
+ cp ../../../link-parser/.libs/link-parser link-parser.bc
+ emcc -O3 link-parser.bc ../../../link-grammar/.libs/liblink-grammar.a --pre-js pre.js -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 -o dist/link-parser.js
wasm-ld: error: unknown file type: link-parser.bc

And indeed ../../../link-parser/.libs/link-parser is not a bytecode file.

[linas]

I merged this pull req because I actually looked to see what it did. And now that my eyes uncrossed, it was obviously "the right thing to do"

[linas]

For some reason it installs emsdk version 1.38.41, which is one of the firs versions.

The js bindings are 10-15 years old :-/

[linas]

I guess you tried this:

./emsdk install latest
./emsdk activate latest

[ampli]

./emsdk install latest
./emsdk activate latest

Yes.
However, I still don't know which fix is needed in order to generate a usable link-parser/.libs/link-parser.

[linas]

link-parser/.libs/link-parser is an ascii shell script created by autotools. In the same directory there is a link-parser.wasm. When I cp link-parser.wasm link-parser.bc I get

wasm-ld: error: link-parser.bc: not a relocatable wasm file

[ampli]

link-parser/.libs/link-parser is an ascii shell script created by autotools. In the same directory there is a link-parser.wasm. When I cp link-parser.wasm link-parser.bc I get

wasm-ld: error: link-parser.bc: not a relocatable wasm file

I get this error too. I try to trace how it is created, in order to find out how to create a relocatable result (in a hope it will work).

[linas]

how to create a relocatable result

https://groups.google.com/g/emscripten-discuss/c/cwg_EdFYOr8/m/xLpVqi9VAQAJ?pli=1

[linas]

cp ../../../link-parser/link_parser-command-line.o command-line.bc
cp ../../../link-parser/link_parser-lg_readline.o lg_readline.bc
cp ../../../link-parser/link_parser-parser-utilities.o parser-utilities.bc
cp ../../../link-parser/link_parser-link-parser.o link-parser.bc
# Originally, this had `emcc liblink-grammar.so` but the .so
# is not available on Apple Mac's (I guess it's called .shlib ??)
# So change to .a which should work for all OS'es.
emcc -O3 link-parser.bc command-line.bc lg_readline.bc parser-utilities.bc \ 
   ../../../link-grammar/.libs/liblink-grammar.a \

[linas]

That seems to build. I have no idea if the result is usable or not. However, look at the debug output here: https://github.com/linas/link-grammar/actions/runs/3952974168/jobs/6768705348 -- this is from trying to uncomment the line

emcc -O3 ../../link-grammar/.libs/liblink-grammar.so -o liblink-grammar.js

and its .. well, like the threading wasn't suppressed after all.

[linas]

Except the issue is not about threading, its about duplicate symbols. So I guess the library should not have been linked. Whatever. I'm going to comment out that line, and just punt. No one is using link-grammar js anyway, and given how things are, it seems unlikely that anyone ever will ...

[ampli]

It seems that by now Emscripten does support Pthreads:

From https://emscripten.org/docs/api_reference/wasm_workers.html:

Emscripten supports two multithreading APIs to leverage this web feature:
POSIX Threads (Pthreads) API, and

Wasm Workers API.

[linas]

Well, its still erroring out see #1377 I'm done with this for now. If you want to work on this and create a new/different pull req, please do; otherwise I will merge #1377 or a variant in a day or two.

[ampli]

Here is how I ran it using Chrome:

1. Ignore build.sh.
2. Currently, the last step of link-parser/Makefile for creating link-parser is:
   /usr/local/src/emsdk/upstream/emscripten/emcc -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Werror-implicit-function-declaration -Wall -Wextra -Wsign-compare -Wpointer-arith -Wwrite-strings -Wmissing-declarations -Wpacked -Wswitch-enum -Wmissing-format-attribute -Wstrict-aliasing -Winit-self -Wshadow -Wno-missing-field-initializers -Wno-unused-parameter -Wno-attributes -Wno-long-long -Winline -D_DEFAULT_SOURCE -std=c11 -D_BSD_SOURCE -D_SVID_SOURCE -D_GNU_SOURCE -D_ISOC11_SOURCE -fvisibility=hidden -g -O3 -o .libs/link-parser link_parser-link-parser.o link_parser-command-line.o link_parser-lg_readline.o link_parser-parser-utilities.o ../link-grammar/.libs/liblink-grammar.a -lm
   Note the -o .libs/link-parser in it. For my text, I manually changed it to -o .libs/link-parser.html.
3. Due to Cross-Origin Requests limitations, it must be loaded through a web server.

cd link-parser/.libs
python -m http.server

4. In chrome:
   localhost:8000/link-parser.html

When I open the JS console I see:

Uncaught RuntimeError: memory access out of bounds
    at dlfree (link-parser.wasm:0x435f7)
    at object_open (link-parser.wasm:0x7193)
    at dictionary_create_default_lang (link-parser.wasm:0x57f6)
    at main (link-parser.wasm:0xb50)
    at Module._main (link-parser.js:3779:72)
    at callMain (link-parser.js:3841:15)
    at doRun (link-parser.js:3884:23)
    at link-parser.js:3895:7

I also tried to run it using `node'.

1. Again ignore build.sh.
2. Replace -o .libs/link-parser by -o .libs/link-parser.js.
3. cd link-parser/.libs
   4.A. node link-parser.js # it uses node version v14.18.2 from the emsdk repository.
   It fails:

failed to asynchronously prepare wasm: CompileError: WebAssembly.instantiate(): Compiling function #316:"form_match_list" failed: Invalid opcode (enable with --experimental-wasm-threads) @+78470
...

4.B. /usr/bin/node link-parser.js # it uses node version v18.12.1.
It fails in the same point that it failed in the browsew:

/usr/local/src/link-grammar-devel/em-detect/link-parser/.libs/link-parser.js:135
      throw ex;
      ^

RuntimeError: memory access out of bounds
    at dlfree (wasm://wasm/006f930e:wasm-function[892]:0x435f7)
    at object_open (wasm://wasm/006f930e:wasm-function[153]:0x7193)
    at dictionary_create_lang (wasm://wasm/006f930e:wasm-function[128]:0x585e)
    at main (wasm://wasm/006f930e:wasm-function[25]:0xb39)
...

So link-parser actually runs but encounters a run-time error in object_open(). I will try to debug it.

[linas]

python -m http.server
localhost:8000/link-parser.html

These instructions belong in a README

[ampli]

It turned out the TLS variable got defined to _Thread_local also under the Emscripten configuration.
I addition, I found that under Emscripten, TLS variables are not initialized by default to 0 or NULL.
So free(pf) in object_open() tried to free a random address. Fixed by defining TLS to nothing by #if __EMSCRPTEN__ in utilities.h.

Also, it is possible to replace the emcc call in biuld.sh by:
(cd ../../../; emmake make EXEEXT=.html LDFLAGS="--pre-js $PWD/bindings/js/link-parser/pre.js -s WASM=1 -s ALLOW_MEMORY_GROWTH=1")

I still haven't found how the following call in build.sh is intended to work:
echo The needs of the many outweigh the needs of the few. | link-parser

Unless I miss something, there is not supposed to be link-parser through PATH at this point.
In addition, trying to invoke link-parser/link-parser gives this error:
link-parser/link-parser: line 117: exec: ... link-parser/.libs/lt-link-parser: cannot execute: Permission denied
But node ... link-parser/.libs/lt-link-parser works!
So what is the idea here?

Also note that the emsdk's bundled node.js, which is an old version, didn't work for me at all (opcode error). But it is in the PATH by default due to emsdk_env.sh. I have no idea why it is bundle, and how to cleanly overcome this problem.

[linas]

Permission denied

In the normal, non-emcc world, the link-parser/link-parser is a shell script that forces the real executable link-parser/.libs/lt-link-parser to link to the shared libraries in the build dir, instead of the shared libs in /usr/lib. So file link-parser/.libs/lt-link-parser just says its a normal ELF executable.

$ file  link-parser/link-parser
link-parser/link-parser: Bourne-Again shell script, ASCII text executable

$ file  link-parser/.libs/link-parser
link-parser/.libs/link-parser: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=a9dd561d6b5b889b138c964e5dfaa1cf99b94ece, for GNU/Linux 3.2.0, with debug_info, not stripped

For the emcc case, I assume file link-parser/.libs/lt-link-parser says its its a WebAssembly module, which would not be normally executable in Linux, and the ascii script gets tangled up.

$ file ./link-parser/dist/link-parser.wasm
./link-parser/dist/link-parser.wasm: WebAssembly (wasm) binary module version 0x1 (MVP)

For me, node is /usr/bin/node from dpkg -S /usr/bin/node says its nodejs and then dpkg -l nodejs says its version 16.17.0-deb-1nodesource1

So I assume if you install nodejs, then the emsdk-bundled version will be ignored.

[ampli]

So I assume if you install nodejs, then the emsdk-bundled version will be ignored.

link-grammar/bindings/js/link-parser/build.sh

Lines 1 to 36 in 750315d

	#!/bin/bash
	set -ex
	pushd "$(dirname "$0")"
	# Activate emscripten sdk if not done already
	[ ! $EMSDK ] && pushd ../emsdk && source ./emsdk_env.sh && popd
	# Create empty `dist` folder
	rm -rf dist
	mkdir dist
	# Compile link-parser.js & link-parser.wasm
	cp ../../../link-parser/.libs/link-parser link-parser.bc
	# Originally, this had `emcc liblink-grammar.so` but the .so
	# is not available on Apple Mac's (I guess it's called .shlib ??)
	# So change to .a which should work for all OS'es.
	emcc -O3 link-parser.bc ../../../link-grammar/.libs/liblink-grammar.a \
	--pre-js pre.js \
	-s WASM=1 \
	-s ALLOW_MEMORY_GROWTH=1 \
	-o dist/link-parser.js
	rm link-parser.bc
	# Copy contents of `data` dir
	cp -r ../../../data dist/data
	# Copy needed contents of this dir
	cp bin.js package.json README.md dist
	# Test
	pushd dist && npm link && popd
	echo The needs of the many outweigh the needs of the few. | link-parser
	pushd dist && npm unlink && popd
	popd

Line 7 sets the path the emsdk's bundled nodejs. This is the output of emsdk_env.sh:

Setting up EMSDK environment (suppress these messages with EMSDK_QUIET=1)
Adding directories to PATH:
PATH += /usr/local/src/emsdk
PATH += /usr/local/src/emsdk/upstream/emscripten
PATH += /usr/local/src/emsdk/node/14.18.2_64bit/bin

Setting environment variables:
PATH = /usr/local/src/emsdk:/usr/local/src/emsdk/upstream/emscripten:/usr/local/src/emsdk/node/14.18.2_64bit/bin:/home/amir/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/var/lib/snapd/snap/bin
EMSDK = /usr/local/src/emsdk
EMSDK_NODE = /usr/local/src/emsdk/node/14.18.2_64bit/bin/node

Note the 3rd component in PATH:
/usr/local/src/emsdk/node/14.18.2_64bit/bin
This old node version doesn't work with link-parser.wasm (aborts on invalid opcode). I needed to edit-out the provided emsdk/node PATH component so my /usr/bin/node would be used (it is version v18.12.1).

Regarding link-parser/link-parser: It exec's lt-link-parser, which is a JS file. If I start it with
#!/usr/bin/env node
and do chmod +x on it, then link-parser/link-parser works!
So my what I don't understand is why emcc doesn't add #!#!/usr/bin/env node and makes the file executable. Maybe there is a way to cause it to do it.

Regarding the test in build.sh:

link-grammar/bindings/js/link-parser/build.sh

Lines 22 to 34 in 750315d

	-o dist/link-parser.js
	rm link-parser.bc
	# Copy contents of `data` dir
	cp -r ../../../data dist/data
	# Copy needed contents of this dir
	cp bin.js package.json README.md dist
	# Test
	pushd dist && npm link && popd
	echo The needs of the many outweigh the needs of the few. | link-parser
	pushd dist && npm unlink && popd

It invokes link-parser, but which searches it in the PATH , and there is normally no . in it. So it uses an installed one (if any). In addition, line 32 performs popd, so link-parser doesn't run inside the dist directory. So how it is supposed toi find the data directory?
In addition, I don't understand the use of npm link. It tries to link to a system directory, which is not supposed to already contain link-grammar (and in any case fails for me because I don't run it as root). I think it should be tested without npm.

[ampli]

I said:

It invokes link-parser, but which searches it in the PATH , and there is normally no .

I am now reading the npm docs in order to understand how all of that is supposed to work.
If needed, I will rewrite build.sh.

[linas]

Should I merge #1377? Or should I wait?

[ampli]

Should I merge #1377? Or should I wait?

Better to wait since I'm touching the same code.
I also try to do the things in the autoconf way.