Getting started

Devspace is a Continuous Integration tool managed by Jenkins CI providing an automation framework that runs repeated jobs. The default deployment initializes a Jenkins CI master with a predefined set of jobs.

Running and maintaining Devspace requires brief understanding of:

Running and maintaining Devspace in OpenStack requires, in addition, brief understanding of:

Ansible
- playbook
access to openstack tenancy

Running Devspace requires access to SSH and Git configuration files used for fetching and pushing the Git repositories.

Devspace code depends on the following repositories:

and for OpenStack (optional)

ansible-role-devspace

Installation

You can either deploy manually a devspace on a Docker host or you can use the Ansible playbooks to deploy a devspace on OpenStack.

Deploy on a Docker host

The following instructions explain how to deploy a devspace on a Docker host.

Log into the Docker host using ssh
Install the prerequisites Docker engine and Docker compose either globally or in a virtual environment:
```
$ pip install docker-compose
```

Create a directory /data/username and change ownership:

$ sudo mkdir /data/username
$ sudo chown username /data/username

Clone the devspace Git repository:

$ git clone https://github.com/openmicroscopy/devspace.git
$ cd devspace

Generated self-signed SSL certificates for the Jenkins and NGINX containers:
```
$ ./sslcert jenkins/sslcert HOST_IP
$ ./sslcert nginx/sslcert HOST_IP
```
alternatively put your own certificate .crt and .key in the above locations.
Copy the SSH and Git configuration files used for fetching and pushing the Git repositories under slave/.ssh and slave/.gitconfig. This is usually your own SSH and Git configuration files
Run rename.py to match your topic name. Specify the Git user corresponding to the confguration files used above. If you do not yet have topic branches available on origin, use "develop" or one of the main branches:
```
$ ./rename.py MYTOPIC --user git_user
```
This will also replace the USER_ID of the various Dockerfile with the ID of the user who will run the devspace, assumed to be: id -u, i.e. the current user.

Set environment variables in .env, especially:

JENKINS_USERNAME=devspace
JENKINS_PASSWORD=<password>

Optionally, commit all the deployment changes above on the local clone of the devspace repository.

Start and configure:

Start devspace using docker-compose:
```
$ docker-compose up -d
```
By default, this will use the name of the directory as the project name. In the case of a shared Docker host, it is possible to override the project name using
```
$ docker-compose up -p my_project -d
```
Retrieve the dynamic port of the Jenkins NGINX container. You can access the Jenkins UI from https://HOST_IP:PORT after accepting the self-signed certificate:
```
$ docker-compose -p my_project port nginxjenkins 443
```

[Optional] Turn on Basic HTTP authentication for Jenkins

sudo htpasswd -c jenkins/conf.d/passwdfile nginx

and update jenkins/conf.d/jenkins.conf:

auth_basic "Restricted";
auth_basic_user_file /etc/nginx/conf.d/passwdfile;

Deploy on OpenStack

The following instructions explain how to deploy a devspace on OpenStack. First, you will need to have an account on OME OpenStack.

The SSH and Git configuration files are used for fetching from and pushing to the Git repositories. They will be copied to the devspace.

The OpenStack and SSH and Git following steps are only need to be done the first time you generate instances.

OpenStack configuration

Log into OpenStack
Register a key:
- Go to the Access & Security tab
- Click on Key Pairs and then on Import Key Pair
- Copy the content of the public key you use to access our resources e.g. id_rsa.pub
- The name you used is referred below as your_openstack_key
Download your configuration:
- Go to the Access & Security tab
- Then API Access
- Click on Download OpenStack RC File v2.0
- The file will be named by default omedev-openrc.sh. It will be used to set environment variables needed to connect to OpenStack via the command line.

SSH and Git configuration files

In order to be able to push result of the build job to your GitHub account, you will need a SSH key without passphrase. The key must be named id_gh_rsa. The key and the configuration files will be copied to the devspace.

Create a directory devspace_config where you wish and a directory devspace_config/.ssh

Generate a SSH key without passphrase in devspace_config/.ssh directory:

  $ ssh-keygen -t rsa -b 4096 -C "your_email_address" -f path/to/devspace_config/.ssh/id_gh_rsa -q -P ""

Upload the corresponding public key i.e. id_gh_rsa.pub to your GitHub account

Create a file devspace_config/.ssh/config and add the following:

  Host github.com
      User git
      IdentityFile ~/.ssh/id_gh_rsa

Create a devspace_config/.gitconfig file

Generate a GitHub token and add it to devspace_config/.gitconfig. Minimally the file should contain:

  [github]
          token = your_token
          user = your_github_username
  [user]
          email = your_email_address
          name = your_real_name

The devspace_config directory should look like:

/path/to/devspace_config/
   .gitconfig
   .ssh/
        config
        id_gh_rsa
        id_gh_rsa.pub

Create and provision the devspace

Clone the devspace Git repository:

  $ git clone https://github.com/openmicroscopy/devspace.git

Create a virtual environment and from the devspace directory, install shade to access OpenStack via the command line and Ansible:
```
  $ virtualenv ~/dev
  $ . ~/dev/bin/activate
  $ cd devspace
  (dev) $ pip install -r requirements.txt
```
Source the OpenStack configuration file to set the environments variables allowing connection to OpenStack via the command line, adjust to your local configuration:
```
  (dev) $ . path/to/omedev-openrc.sh
  Enter your password
```

The following commands need to be executed from the ansible subdirectory.

Install the various ansible roles from the Galaxy website:

  (dev) $ cd ansible
  (dev) $ ansible-galaxy install -r requirements.yml

To "upgrade" roles, you may want to specify --force when installing the roles.

Create an instance on OpenStack using the playbook create-devspace.yml. It is recommended to prefix the name of the devspace by your name or your initals:
```
  (dev) $ ansible-playbook create-devspace.yml -e vm_name=your_name-devspace-name -e vm_key_name=your_openstack_key
```

By default the size of the volume is 50GiB, if you required a larger size, it can be set by passing for example -e vm_size=100. The Floating IP of the generated instance is referred as devspace_openstack_ip below.

To provision the devpace, use the playbook provision-devspace.yml. Before running the playbook you will minimally need to edit the value of the parameter:
- configuration_dir_path: set it to path/to/devspace_config

See ansible-role-devspace for a full list of supported parameters. Provision the devspace by running:

    (dev) $ ansible-playbook -u centos -i devspace_openstack_ip, provision-devspace.yml

Access the devspace and determine ports

Ports to access the various services are dynamically assigned. You will have to log in to the devspace as the omero user to determine the port used by a given service using your usual ssh key and not the id_gh_rsa key:

    ssh omero@devspace_openstack_ip
    cd devspace

The port to access the Jenkins UI is obtained by running:

  docker-compose port nginxjenkins 443

The output of the command looks like:

  WARNING: The JENKINS_PASSWORD variable is not set. Defaulting to a blank string.
  WARNING: The USER_ID variable is not set. Defaulting to a blank string.
  0.0.0.0:xxxx

where xxxx is the assigned port $JENKINS_PORT

The Jenkins UI will be available at:

  https://devspace_openstack_ip:$JENKINS_PORT

The port to access OMERO.web is obtained by running:

 docker-compose port nginx 80

The command will generate a similar output that the one above

  WARNING: The JENKINS_PASSWORD variable is not set. Defaulting to a blank string.
  WARNING: The USER_ID variable is not set. Defaulting to a blank string.
  0.0.0.0:xxxx

where xxxx is the assigned port $WEB_PORT

OMERO.web will be available at

  http://devspace_openstack_ip:$WEB_PORT/web

The port to access OMERO.insight or OMERO.cli is obtained by running:
```
 docker-compose port omero 4064 
```
- The command will generate a similar output that the one above
```
  WARNING: The JENKINS_PASSWORD variable is not set. Defaulting to a blank string.
  WARNING: The USER_ID variable is not set. Defaulting to a blank string.
  0.0.0.0:xxxx
```
  where xxxx is the assigned port $SERVER_PORT
- To login either via OMERO.insight or OMERO.cli use $SERVER_PORT as the port value and devspace_openstack_ip as the server value. You must use the secure connection.

Job configurations

Job workflow

The default deployment initializes a Jenkins server with a predefined set of jobs.

The table below lists the job names, the Jenkins node labels and the associated docker they are associated with and a short description of the jobs.

Job name	Name	Description	docker name
Trigger		Runs all the following jobs in order
BIOFORMATS-push	testintegration	Merges all Bio-Formats PRs	devspace_testintegration_1
BIOFORMATS-build	testintegration	Builds Bio-Formats components	devspace_testintegration_1
BIOFORMATS-image	testintegration	Builds a Docker image of Bio-Formats	devspace_docker_1
OMERO-push	testintegration	Merges all OMERO PRs	devspace_testintegration_1
OMERO-build	testintegration	Builds OMERO artifacts (server, clients)	devspace_testintegration_1
OMERO-server	omero	Deploys an OMERO.server	devspace_omero_1
OMERO-web	web	Deploys an OMERO.web client	devspace_web_1
OMERO-test-integration	testintegration	Runs the OMERO integration tests	devspace_testintegration_1
OMERO-robot	testintegration	Runs the Robot tests	devspace_testintegration_1
nginx	nginx	Reloads the nginx server	devspace_nginx_1
OMERO-docs	testintegration	Builds the OMERO documentation	devspace_testintegration_1

This means that by default the following repositories need to be forked to your GitHub account:

If you do not have some of the repositories forked, you will need to remove the jobs from the list of jobs to run either from the Trigger job configuration or directly from the Jenkins UI i.e. Trigger > Configure.

Default packages used

Name	Version	Optional
Java	openJDK 1.8	openJDK 1.8 devel, oracleJDK 1.8
Python	2.7	-
Ice	3.6	3.5
PostgreSQL	9.4	https://hub.docker.com/_/postgres/
Nginx	1.8	-
Redis	latest	https://hub.docker.com/_/redis/

Troubleshooting

See Troubleshooting

ADVANCE: extend omero-install

In order to install additional components or new version of packages e.g. PostgreSQL 10, it is required to:

Modify the files in omero-install
Create a new image of devslave-c7-docker using the updated omero-install files
Push the new image to Docker Hub. You will need to your own account
Modify each Dockerfile of this repository to use the new image

Upgrade

See Changelog

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Getting started

Installation

Deploy on a Docker host

Deploy on OpenStack

OpenStack configuration

SSH and Git configuration files

Create and provision the devspace

Access the devspace and determine ports

Job configurations

Job workflow

Default packages used

Troubleshooting

ADVANCE: extend omero-install

Upgrade

About

Releases

Packages

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 684 Commits
ansible		ansible
git		git
home		home
jenkins		jenkins
nginx		nginx
server		server
slave		slave
web		web
.env		.env
.gitignore		.gitignore
.travis.yml		.travis.yml
CHANGELOG.md		CHANGELOG.md
README.md		README.md
Troubleshooting.md		Troubleshooting.md
backup.sh		backup.sh
bak		bak
docker-compose.yml		docker-compose.yml
ex		ex
ip		ip
rename.py		rename.py
requirements.txt		requirements.txt
runtest.sh		runtest.sh
sslcert		sslcert

atarkowska/devspace

Folders and files

Latest commit

History

Repository files navigation

Getting started

Installation

Deploy on a Docker host

Deploy on OpenStack

OpenStack configuration

SSH and Git configuration files

Create and provision the devspace

Access the devspace and determine ports

Job configurations

Job workflow

Default packages used

Troubleshooting

ADVANCE: extend omero-install

Upgrade

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages