Accept batch inputs in generate_tokens and add async_generate_tokens #2425

Job	Run time
build-and-push-docker-images	39m 37s
build-python-wheels (ubuntu-20.04, auto64)	56m 11s
build-python-wheels (macos-11, auto64)	50m 42s
build-python-wheels (windows-2019, auto64)	1h 20m 48s
build-python-wheels (ubuntu-20.04, aarch64)	1h 29m 1s
build-python-wheels (macos-11, arm64)	14m 24s
build-and-test-cpp-x86_64 (ubuntu-20.04, mkl)	8m 3s
build-and-test-cpp-x86_64 (ubuntu-20.04, dnnl)	6m 16s
build-and-test-cpp-x86_64 (macos-11, mkl)	11m 28s
build-and-test-cpp-x86_64 (macos-11, dnnl)	9m 53s
build-and-test-cpp-aarch64 (openblas)	8m 11s
check-python-style	10s
build-and-deploy-docs	49s
test-python-wheels (ubuntu-20.04)	3m 53s
test-python-wheels (macos-11)	3m 45s
test-python-wheels (windows-2019)	3m 25s
publish-python-wheels-on-pypi	0s
	6h 26m 36s

Provide feedback