Skip to content

Latest commit

 

History

History
177 lines (145 loc) · 6.17 KB

README.md

File metadata and controls

177 lines (145 loc) · 6.17 KB
Raw

SoundSpaces for NVAS3D

This guide provides a step-by-step process to set up and generate data using SoundSpaces for NVAS3D. You can also quickly start the installation process by running the setup.sh script included in this guide.

Prerequisites for SoundSpaces

  • Ubuntu 20.04 or a similar Linux distribution
  • CUDA
  • Conda

Installation

Here we repeat the installation steps from SoundSpaces Installation Guide.

0. Update PYTHONPATH

Ensure your PYTHONPATH is updated:

cd ml-nvas3d # the root of the repository
export PYTHONPATH=$PYTHONPATH:$(pwd)

1. Install SoundSpaces Dependencies

Install required dependencies for SoundSpaces:

apt-get update && apt-get upgrade -y && \
apt-get install -y --no-install-recommends libjpeg-dev libglm-dev libgl1-mesa-glx libegl1-mesa-dev mesa-utils xorg-dev freeglut3-dev

2. Update Conda

Ensure that your Conda installation is up to date:

conda update -n base -c defaults conda

3. Create and Activate Conda Environment

Create a new Conda environment named nvas3d with Python 3.7 and cmake 3.14.0:

conda create -n nvas3d python=3.7 cmake=3.14.0 -y && \
conda activate nvas3d

4. Install PyTorch

Install PyTorch, torchvision, torchaudio, and the CUDA toolkit. Replace the version accordingly with your CUDA version:

conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia

5. Install Habitat-Sim

Install Habitat-Sim:

# step outside the repo root and git clone Habitat-Sim
cd .. && \ 
git clone https://github.com/facebookresearch/habitat-sim.git && \
cd habitat-sim && \
pip install -r requirements.txt && \
git checkout RLRAudioPropagationUpdate && \
python setup.py install --headless --audio --with-cuda

This guide is based on commit 30f4cc7.

6. Install Habitat-Lab

Install Habitat-Lab:

# step outside the repo root and git clone Habitat-Lab
cd .. &&\
git clone https://github.com/facebookresearch/habitat-lab.git && \
cd habitat-lab && \
git checkout v0.2.2 && \
pip install -e . && \
sed -i '36 s/^/#/' habitat/tasks/rearrange/rearrange_sim.py # remove FetchRobot

This guide is based on commit 49f7c15.

7. Install SoundSpaces

Install SoundSpaces:

# step outside the repo root and git clone SoundSpaces
cd .. &&\ 
git clone https://github.com/facebookresearch/sound-spaces.git && \
cd sound-spaces && \
pip install -e .

This guide is based on commit 3768a50.

8. Install Additional Packages

Install additional Python packages needed:

pip install scipy torchmetrics pyroomacoustics

Quick Installation

To streamline the installation process, run the setup.sh script, which encapsulates all the steps listed above:

bash setup.sh

Preparation of Matterport3D Room and Material

1. Download Example MP3D Room

Follow these steps to download the MP3D data in the correct directory:

(1) Switch to habitat-sim directory:

cd /path/to/habitat-sim

(2) Run the dataset download script:

python src_python/habitat_sim/utils/datasets_download.py --uids mp3d_example_scene

(3) Copy the downloaded data to this repository:

mkdir -p /path/to/ml-nvas3d/data/scene_datasets/ 
cp -r data/scene_datasets/mp3d_example /path/to/ml-nvas3d/data/scene_datasets/mp3d

After executing the above steps, ensure the existence of the data/scene_datasets/mp3d/17DRP5sb8fy directory. For additional rooms, you might want to consider downloading from Matterport3D.

2. Download Material Configuration File

Download the material configuration file in the correct directory:

cd /path/to/ml-nvas3d && \
mkdir data/material && \
cd data/material && wget https://raw.githubusercontent.com/facebookresearch/rlr-audio-propagation/main/RLRAudioPropagationPkg/data/mp3d_material_config.json

Note

Now, you are ready to run Demo using our pretrained model.

To explore with more data, you can consider running Training Data Generation.

Extended Usage: SoundSpaces for NVAS3D

For users interested in exploring with more data, follow the steps outlined below.

Generate grid points in room:

To create grid points within the room, execute the following command:

python soundspaces_nvas3d/rir_generation/generate_grid.py --grid_distance 1.0
  • Input directory: data/scene_datasets/mp3d/
  • Output directory: data/scene_datasets/metadata/mp3d/grid_{grid_distance}

Generate RIRs:

To generate room impulse responses for all grid point pairs, execute the following command:

python soundspaces_nvas3d/rir_generation/generate_rir.py --room 17DRP5sb8fy
  • Input directory: data/scene_datasets/mp3d/{room}
  • Output directory:data/examples/rir_mp3d/grid_{grid_distance}/{room}

Minimal Example of RIR Generation Using SoundSpaces

We provide an mimimal example code for generating RIRs using our codebase. To generate sample RIRs, execute the following command:

python demo/sound_spaces/nvas3d/example_render_ir.py

Optionally, for those interested in training an audio-visual network, images can be rendered using the following methods:

Render target images:

To render images centered at individual grid points, execute the following command:

python soundspaces_nvas3d/image_rendering/run_generate_target_image.py
  • Input directory: data/scene_datasets/mp3d/{room}
  • Output directory: data/examples/target_image_mp3d/grid_{grid_distance}/{room}

Render environment maps:

To render environment maps from all grid points, execute the following command:

python soundspaces_nvas3d/image_rendering/run_generate_envmap.py
  • Input directory: data/scene_datasets/mp3d/{room}
  • Output directory: data/examples/envmap_3d/grid_{grid_distance}/{room}