End-to-end autonomous driving with vision-only is not only more cost-effective compared to LiDAR-vision fusion but also more reliable than traditional methods. To achieve a economical and robust purely visual autonomous driving system, we propose RenderWorld, a vision-only end-to-end autonomous driving framework, which generates 3D occupancy labels using a self-supervised gaussian-based Img2Occ Module, then encodes the labels by AM-VAE, and uses world model for forecasting and planning. RenderWorld employs Gaussian Splatting to represent 3D scenes and render 2D images greatly improves segmentation accuracy and reduces GPU memory consumption compared with NeRF-based methods. By applying AM-VAE to encode air and non-air separately, RenderWorld achieves more fine-grained scene element representation, leading to state-of-the-art performance in both 4D occupancy forecasting and motion planning from autoregressive world model.
基于视觉的端到端自动驾驶不仅比 LiDAR-视觉融合更具成本效益，而且比传统方法更可靠。为了实现经济且稳健的纯视觉自动驾驶系统，我们提出了 RenderWorld，这是一种基于视觉的端到端自动驾驶框架。该框架通过自监督的基于高斯的 Img2Occ 模块生成 3D 占据标签，然后由 AM-VAE 编码这些标签，并使用世界模型进行预测和规划。RenderWorld 采用高斯投影来表示 3D 场景并渲染 2D 图像，与基于 NeRF 的方法相比，大大提高了分割精度并减少了 GPU 内存消耗。通过应用 AM-VAE 分别编码空气和非空气元素，RenderWorld 实现了更精细的场景元素表示，在 4D 占据预测和基于自回归世界模型的运动规划中达到了最先进的性能。