From Travis CI documentation
Continuous Integration is the practice of merging in small code changes frequently - rather than merging in a large change at the end of a development cycle. The goal is to build healthier software by developing and testing in smaller increments.
In NYPL Digital's CI/CD stack, when a pull request is merging with, e.g., development
branch, a series of steps is taken to build, test, and deploy to the branch.
Travis CI is a tool with options to deploy to our services of choice, e.g. AWS Elastic Beanstalk.
Travis CI looks for a configuration file named .travis.yml
to receive instructions on how to build, test, and deploy an application. This means the following:
- The file MUST be located at the root level of the code repository, such as
NYPL/staff-picks/.travis.yml
. - The file MUST be named
.travis.yml
, with a leading period in the file name.
A basic build on Travis CI consists of the following steps
install
: installs required dependencies for the applicationscript
: runs the build script(s) for the application
There are many steps in between the two configuration stages mentioned above that can be added for further customization. Here's a list of frequently used Travis CI configuration stages on NYPL applications, from top to bottom of .travis.yml
file:
language
: Programming language of the code repository- Runtime environment version managers such as
rvm
forruby
,node_js
forjavascript
withnode_js
, etc. sudo
to indicate whethersudo
privileges are required.before_install
to include any applications that are needed to build and/or test an application, but are not included as part of the standard virtualization machine generated by Travis.addons
is similar tobefore_install
, for quick shortcuts if you already know which APT package you want to installcache
to cache content that does not often change, to speed up your build process, e.g.node_modules
directory from node apps.install
: Same description as mentioned above.before_script
: Install dependencies required to build the applicationscript
: Same description as mentioned above.before_deploy
: Used for user-friendly messages to notify user of deployment.deploy
: Sets up watched branches and AWS credentials needed to deploy to, e.g. Elastic Beanstalkafter_deploy
: Used for user-friendly messages to notify user deployment is successfully triggered.
Testing is often run during the script
configuration stage as part of the build.
Travis has a deploy
configuration stage, with a list of preconfigured providers you can use for deployments.
For new Travis users:
- Sign in to travis-ci.com using your Github account
- Create a personal access token on Github and give it the scope listed here (make sure you use the travis-ci.com scope, NOT the travis-ci.org scope)
- From the commandline run
travis login --pro --github-token <your_token>
To configure Travis to deploy:
- From the root directory of your app
- Find the encrypted command stored [in nypl-digital-dev parameter store](https://us-east-1.console.aws.amazon.com/systems-manager/parameters/production/travis/add_aws/description?region=us-east-1&tab=Table#list_parameter_filters=Name:Contains:travis) which will automatically add encrypted credentials for AWS_ACCESS_KEY_ID_DEVELOPMENT, AWS_SECRET_ACCESS_KEY_DEVELOPMENT, AWS_SECRET_ACCESS_KEY_QA, AWS_SECRET_ACCESS_KEY_PRODUCTION, AWS_ACCESS_KEY_ID_QA, and AWS_ACCESS_KEY_ID_PRODUCTION to `.travis.yml`. (If this is a new Travis integration - or new to you - [ensure the repo is locally associated with the right Travis endpoint](#failure-to-decrypt-environmental-variables).)
- Run the command in the root directory of your app
- Add
deploy
entries to.travis.yml
for each deploy hook. Beanstalk example:
deploy:
- provider: elasticbeanstalk
skip_cleanup: true
access_key_id: "$AWS_ACCESS_KEY_ID_DEVELOPMENT"
secret_access_key: "$AWS_SECRET_ACCESS_KEY_DEVELOPMENT"
region: us-east-1
app: discovery-api # This is the name of the app
env: discovery-api-dev # This is the specific deployment
bucket_name: elasticbeanstalk-us-east-1-224280085904 # This is common across EB deployments in a given account
bucket_path: discovery-api-dev # Conventionally this should equal the `env` name
on:
repo: NYPL-discovery/discovery-api
branch: development # specify the branch for travis to watch
Lambda example:
deploy:
- provider: script
skip_cleanup: true
script: npm run deploy-development
on:
branch: development
The Travis "Settings" page for a given repo allows you to enable two kinds of builds:
- "Build pushed branches": Create a build for all pushed branches and commits (i.e. build when "my-feature" branch is pushed, build when "development" is updated in any way)
- "Build pushed pull requests": Create a build any time a PR is created by checking out the target branch, locally merging the PR branch into it, and running the test suite against the result.
Both options are enabled by default. The former serves as an essential baseline check. The latter adds assurance that the result of the merge will continue to pass.
Note that the deploy
section is not evaluated in the context of a pushed pull request.
When using Chrome to conduct tests such as with karma
, Chrome is not installed by default on the virtualization environment.
The idea is to install Chrome as an apt addon and enabling display within the Travis virtualization machine so Chrome can run with a monitor on the background.
addons:
chrome: stable # have Travis install chrome stable.
...
before_script:
- export DISPLAY=:99.0 # Export display to Travis
- sh -e /etc/init.d/xvfb start # Start graphical operations
The installation of Chrome would still be required even if chrome-headless is used by karma
or other test suites.
If it seems your encrypted environmental variables aren't being set in builds and/or you see something like the following in your build log, this section is for you:
Setting environment variables from .travis.yml
$ export CpxjaQ=[secure]
...
One reason Travis will fail to properly decrypt environmental variables is if there's a mismatch between the Travis endpoint used to encrypt the values and the Travis endpoint used to run builds. Travis is in the process of moving from travis-ci.org to travis-ci.com. All builds will eventually migrate to the latter. Our repos are currently split between them, with most of the newer Travis integrations existing on .com
.
To determine whether .org
or .com
is correct for a given repo, simply check the Travis URL of any attempted build.
To determine which endpoint is actually used to encrypt on your local machine, check ~/.travis/config.yml
. For example:
endpoints:
https://api.travis-ci.org/:
access_token: [snip]
https://api.travis-ci.com/:
access_token: [snip]
repos:
NYPL-discovery/discovery-front-end:
endpoint: https://api.travis-ci.org/
NYPL-discovery/schemaservice:
endpoint: https://api.travis-ci.com/
The endpoints
section indicates I've authenticated against both .org
and .com
at some point. Under repos
, we see that builds for discovery-front-end are expected to be run on .org
whereas builds for schemaservice are expected to be run on .com
. Consequently any time I run travis encrypt ...
locally for either of those repos, the encryption will be tied to those endpoints.
When you run travis encrypt ...
, the command makes a guess about whether the integration will be run on .org
or .com
. It makes that guess based on the contents of the repos
section described above and, failing a match there, falls back on your default endpoint (see notes on travis endpoint ...
below). When travis encrypt
defaults to the wrong endpoint, correct that by re-running your travis encrypt ...
command with --com
or --org
at the end. (This will add the missing entry to your ~/.travis/config.yml
for subsequent calls.)
To ensure all new Travis integrations default to a .com
association for you going forward:
travis endpoint --com --set-default
Many apps use $TRAVIS_BRANCH
environmental variable in .travis.yml
to inspect the current branch. This allows us to DRY our deployment code by using only a single deploy
provider that uses $TRAVIS_BRANCH
to control the place deployed to. Example
Note that Travis will not attempt to deploy anything in the context of a PR build. For example, when Travis checks out "development" to locally merge "my-feature" and run tests, Travis will not thereafter attempt to deploy to "development" even though $TRAVIS_BRANCH
is "development"; It skips deploy
altogether for PR builds.
When creating a PR in a repo with "Build pushed pull requests" enabled, Travis will show two builds: One for the "push" and one for the PR. Typically one creates a PR to merge a recently pushed branch, so both builds will appear "pending". If the source branch was updated a while ago, the build status of the "push" may already be visible when the PR is created. You'll therefore see one pending "PR" build and one "push" build showing the status of the last push to that branch. If the last push to the source branch was a failure, that build will appear as a failure in your PR. This is sensible and good, but may be distressing because it suggests your PR triggered the failure, but in fact the failure may be old and/or unrelated.
A concrete example: When creating a PR to merge "development" into "qa", we immediately saw that the "push" build was a failure. On clicking through, the failure was found to be in the deployment stage. This was distressing because it initially appeared a deployment was being executed in response to a PR, but in fact the deployment was triggered by the original push to "development" (i.e. we simply hadn't noticed the original failure.)