Mesh Neural Cellular Automata (SIGGRAPH 2024) [Project Page]

This is the official implementation of the paper titled "Mesh Neural Cellular Automata".

Getting Started

For a quick and hands-on experimentation with MeshNCA, we suggest trying our Google Colab notebook (click on Open in Colab button at the top of the readme). You can follow the steps below to run our code locally for in-depth experimentation.

Installing the packages

1. Clone the repository

git clone https://github.com/IVRL/MeshNCA.git
cd MeshNCA

2. Create a new virtual environment and install the required python packages.
   • We Use Kaolin for differentiable rendering. Currently, Kaolin only supports PyTorch >= 1.8, <= 2.1.1
   • We use PyG (PyTorch Geometric) to implement the message passing in MeshNCA architecture.

python3 -m venv env
source env/bin/activate
chmod +x install_requirements.sh
./install_requirements.sh

Downloading the data

Run the download_data.py script to download the data.

python3 download_data.py

The data will be downloaded to the data directory and has three parts:

• The mesh dataset consisting of 48 meshes with valence close to 6 and uniformly distributed vertices.
• 72 Physically-based-rendering (PBR) textures, each containing 5 textures (albedo, normal, roughness, height, and ambient occlusion) All textures have a resolution of 1024x1024.
• 45 RGB textures adopted from DyNCA.

How to run

Here, we outline the steps to train and visualize the MeshNCA model.

To run the code you will need to have a yaml config file that specifies the training settings. You can find example config files in the configs directory.

For example, to train the MeshNCA model with PBR textures config, you can use the following command:

python3 train_image.py --config configs/pbr_texture.yaml

Config files

Each setting in the config file is explained in the comments. Currently, we only support the image-guided training scheme and provide two sample config files for training with PBR textures and single RGB textures. The configuration and the training code for text-guided and motion-guided training schemes will be released soon.

• configs/pbr_texture.yaml: Contains the training settings for PBR textures. You can use this config file when you want MeshNCA to simultaneously learn and synthesize multiple related textures such as the albedo, normal, roughness, height, and ambient occlusion maps.
• configs/single_texture.yaml: Contains the training settings for RGB textures.

Example:

The target images are specified in the appearance_loss_kwargs section of the config file. The target_images_path dictionary contains the paths to the target images, and the num_channels dictionary specifies the number of channels in each target image.

loss:
  appearance_loss_kwargs:
    # The single channel images will be expanded to 3 channels for evaluating the VGG-based style loss
    target_images_path: {
      "albedo": "data/pbr_textures/Abstract_008/albedo.jpg",
      "height": "data/pbr_textures/Abstract_008/height.jpg",
      "normal": "data/pbr_textures/Abstract_008/normal.jpg",
      "roughness": "data/pbr_textures/Abstract_008/roughness.jpg",
      "ambient_occlusion": "data/pbr_textures/Abstract_008/ambient_occlusion.jpg",
    }
    # Number of channels in the target images
    num_channels: {
      "albedo": 3,
      "height": 1,
      "normal": 3,
      "roughness": 1,
      "ambient_occlusion": 1,
    }

If you face out of memory issues, you can reduce the num_views or the batch_size in the config file.

train:
  batch_size: 1
  camera:
    num_views: 6  

To record the training logs in wandb (Weights and Biases), you can set the wandb key in the config file.

wandb:
  project: "Project Name"
  key: "Your API Key"

Citation

If you make use of our work, please cite our paper:

@article{pajouheshgar2024mesh,
  title={Mesh neural cellular automata},
  author={Pajouheshgar, Ehsan and Xu, Yitao and Mordvintsev, Alexander and Niklasson, Eyvind and Zhang, Tong and S{\"u}sstrunk, Sabine},
  journal={ACM Transactions on Graphics (TOG)},
  volume={43},
  number={4},
  pages={1--16},
  year={2024},
  publisher={ACM New York, NY, USA}
}

License

Shield:

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.