fix: remove unused `astropy` dependency #139

Job	Run time
qa	13s
build (3.10, ubuntu-latest)	1m 3s
build_single_file_binary (linux)	2m 36s
build (3.11, ubuntu-latest)	59s
build (3.12, ubuntu-latest)	2m 30s
build_single_file_binary (windows)	5m 3s
	12m 24s

Provide feedback