Skip to content

Latest commit

 

History

History
266 lines (189 loc) · 8.65 KB

CONTRIBUTING.md

File metadata and controls

266 lines (189 loc) · 8.65 KB
Raw

Contributing to Determined

Reporting Issues and Feature Requests

If you encounter an issue or would like to request a new feature, please create an issue on GitHub. You can also join the Slack to get support and talk with other users and developers in real-time.

Project Roadmap

https://github.com/determined-ai/determined/wiki/Project-Roadmap

Contributing Changes

We welcome outside contributions. If you'd like to make a contribution, please:

  1. Tell us about what you'd like to contribute on our Slack or community mailing list. We'd hate for you to duplicate effort is already in-flight.

  2. Apply the linter with make fmt and test locally with make test before submitting your code. Make sure that your code doesn't accidentally include cloud credentials. We recommend using git-secrets to automatically prevent this.

  3. The first time you submit code, you'll need to sign a CLA.

  4. Submit a pull request. Someone from the Determined team will review the request and provide feedback. Once we agree that the code is in good shape, it will be merged into the master branch.

Installation from Source

Setting up

Determined can be developed and run on both Linux and macOS (Linux is strongly recommended for production deployments). Determined has been tested with Ubuntu 16.04 LTS, Ubuntu 18.04 LTS, Arch Linux, CentOS 7, and macOS. Ubuntu is recommended; on AWS, a good AMI to use is a recent version of "Deep Learning Base AMI (Ubuntu)".

Start by cloning the Determined repository:

git clone --recurse-submodules https://github.com/determined-ai/determined.git

Prerequisites

  • Go (>= 1.18)
  • Python (>= 3.6, <= 3.9), including:
    • python3-venv
    • python3-wheel
    • python3-dev
  • Node (>= 16.13, < 17)
  • NPM (>= 8)
  • Docker (>= 19.03)
  • Helm (>= 3.0.0)
  • protobuf-compiler (>= 3.0)
  • Java (>= 7)
  • cURL (>= 7)
  • jq (>= 1.6)
  • socat (>= 1.7)

If you are installing prerequisites from your Linux distribution's package repository, ensure that they meet the version requirements above, particularly Python and Node.

Install Prerequisites with Homebrew for Linux and macOS

Because the versions of prerequisites from Linux distribution package repositories can vary widely, we recommend installing the Determined build prerequisites with Homebrew.

The following instructions are also applicable for building Determined on macOS.

Install a compiler and build tools:

  • macOS: Install the Command Line Tools for Xcode from Apple
  • Debian and Ubuntu: sudo apt install build-essential
  • Red Hat, CentOS, and Fedora: sudo yum install gcc make perl-devel
  • SUSE and openSUSE: sudo zypper install -t pattern devel_basis

Install Homebrew:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Add Homebrew to your PATH:

echo 'eval "$($HOME/.linuxbrew/bin/brew shellenv)"' >> $HOME/.profile
eval "$($HOME/.linuxbrew/bin/brew shellenv)"

Install the Determined prerequisites:

brew install [email protected] [email protected] node@16 openjdk@11 protobuf docker helm curl jq socat

Add Python, Node, and openJDK to your PATH:

echo 'export PATH="$HOME/.linuxbrew/opt/[email protected]/bin:$HOME/.linuxbrew/opt/node@16/bin:$HOME/.linuxbrew/opt/openjdk@11/bin:$PATH"' >> $HOME/.profile
source $HOME/.profile

On Red Hat and CentOS 8 and Ubuntu 16.04, add the compiled GCC 11 libraries to your LD_LIBRARY_PATH:

echo 'export LD_LIBRARY_PATH=$HOME/.linuxbrew/Cellar/gcc/$(ls $HOME/.linuxbrew/Cellar/gcc/)/lib/gcc/lib64/' >> $HOME/.profile
source $HOME/.profile

Building Determined

cd determined
python3 -m venv $HOME/.virtualenvs/determined
. $HOME/.virtualenvs/determined/bin/activate
$HOME/.virtualenvs/determined/bin/python3.7 -m pip install --upgrade pip
export PATH=$PATH:$HOME/go/bin
make all

In the future, ensure that you activate the virtualenv (by running the activate command above) whenever you want to interact with Determined. Tools such as virtualenvwrapper or direnv may help streamline the process.

Building Errata

If you are building Determined using Python 3.9, you must manually change the version of TensorFlow to 2.8.0 in the end-to-end testing requirements file because the version of TensorFlow currently set in the end-to-end requirements file is not compatible with Python 3.9:

tensorflow==2.8.0; sys_platform != 'darwin' or platform_machine != 'arm64'

Running Determined

A minimal Determined cluster consists of three services: a PostgreSQL database, a Determined master, and a Determined agent.

For running these services in development, please use devcluster. It offers an intuitive UI as well as easy rebuild, restart, and configuration of master and one or more local agents.

Accessing Determined

After following either set of instructions above, the WebUI will be available at http://localhost:8080. You can also use our command-line tool, det, to interact with Determined. For example, det slot list should print out a line for each GPU on your machine, if you have any, or a line for your CPU, if not. For more information, see the reference documentation.

Training a Sample Model

The tutorials/mnist_pytorch directory contains code to train a convnet on MNIST using PyTorch. To train a model, run

det experiment create <config> tutorials/mnist_pytorch/

where <config> can be

  • tutorials/mnist_pytorch/const.yaml to train a single model with fixed hyperparameters
  • tutorials/mnist_pytorch/adaptive.yaml to train multiple models using an adaptive hyperparameter search algorithm

Determined also supports several other hyperparameter search methods.

After starting a model, you can check on its progress using the WebUI or the CLI command det experiment list.

Development

Linting and typechecking

Run make check.

Unit tests

Run make test.

Integration tests

# Run a Determined cluster
make -C tools run

# Run integration tests locally.
pytest -m "e2e_cpu" e2e_tests/tests

Debugging

Connecting to PostgreSQL

To connect directly to the Determined metadata database, run this command from the Determined master host:

docker run -it --rm \
  --network determined \
  -e PGPASSWORD=my-postgres-password \
  postgres:10 psql -h determined-db -U postgres -d determined

Get profiling information

go tool pprof http://master-ip:port  # for CPU samples
go tool pprof http://master-ip:port/debug/pprof/heap  # for heap samples
go tool pprof -http :8081 ~/pprof/sample-file

GPU support

To use Determined with GPUs, the Nvidia drivers (>= 384.81) and nvidia-container-toolkit must be installed.

To verify that your system can run containers that use GPUs and CUDA, run:

docker run --gpus all --rm nvidia/cuda:10.0-cudnn7-runtime-ubuntu16.04 nvidia-smi

If this command displays one or more GPUs, the Determined agent should automatically detect the system's GPUs and make them available for running experiments.

Secrets

To prevent cloud credentials from accidentally being exposed on GitHub, install and configure the git-secrets tool. This sets up git hooks to prevent pushing code that contains secrets (based on regex).

For Mac, the tool can be installed via brew install git-secrets. For other OSes see installation instructions here.

Then navigate to the repository, set up the git hooks, and define the regexes:

cd /path/to/my/repository

# Set up the git hooks for this repo
git secrets --install

# Add AWS regexes
git secrets --register-aws
# Add GCP regex
git secrets --add '"private_key":\s"-----BEGIN\sPRIVATE\sKEY-----'