Accept batch inputs in generate_tokens and add async_generate_tokens #2423

Job	Run time
build-and-test-cpp-aarch64 (openblas)	5m 48s
build-and-push-docker-images	46m 3s
build-and-test-cpp-x86_64 (ubuntu-20.04, mkl)	6m 57s
build-and-test-cpp-x86_64 (ubuntu-20.04, dnnl)	8m 15s
build-and-test-cpp-x86_64 (macos-11, mkl)	10m 50s
build-and-test-cpp-x86_64 (macos-11, dnnl)	10m 27s
check-python-style	12s
build-python-wheels (ubuntu-20.04, auto64)	39m 19s
build-python-wheels (macos-11, auto64)	43m 0s
build-python-wheels (windows-2019, auto64)	55m 46s
build-python-wheels (ubuntu-20.04, aarch64)	1h 34m 29s
build-python-wheels (macos-11, arm64)	13m 54s
build-and-deploy-docs	37s
test-python-wheels (ubuntu-20.04)	7m 50s
test-python-wheels (macos-11)	7m 16s
test-python-wheels (windows-2019)	4m 24s
publish-python-wheels-on-pypi	0s
	5h 55m 7s

Provide feedback