Find the latest release on github and download the installer jar, double-click to launch, and when complete, consult README.html
in the install directory.
- Tl;dr
- Quickstart
- Get YADA
- I downloaded the war file
- I cloned the github repo
- Sanity check
- Additional Filesystem Configuration
- Caveats
- Advanced Topics
- YADA Index
- Automation
- Server Configuration 6. Linux Environment 7. Apache HTTP 8. Tomcat
- Maven 10. Environment Variables 11. Build Properties 12. Profiles 13. Examples
First, acquire the YADA-Quickstart application in one of the following ways:
- Find the latest release on github and download the installer jar
or
- Clone the YADA-Quickstart github repo
Refer to the appropriate section below, either I downloaded the jar file or I cloned the github repo:
If you cloned from github, skip to the next section.
The downloadable jar is a java installer built using the popular open source IzPack solution. In the current release, the installer will drop the Tomcat 8.x servlet container into a default or custom location, drop the warfile into webapps
, inflate it, reconfigure it, along with a HyperSQL®-based YADA index, for your environment, and recompress it. The installer will then drop README.html
into the install directory which contains instructions for install validation and login.
More specifically, the installer deploys the following:
README.html
ant.log
apache-tomcat-8.0.24\
bin\
cruft\
files\
util\
util
,files
, andbin
directories will be empty.cruft
contains install artifactsant.log
is an install artifactapache-tomcat-<version>
is obviously the$CATALINA_HOME
loctaion
Inside apache-tomcat-<version>
you'll find YADA-Quickstart-<version>.war
and a like-name directory.
The installer also puts the HyperSQL® and SQLite® JDBC jars in apache-tomcat-<version>/lib
.
The Quickstart war uses a local HyperSQL® file for the YADA index. This is not likely to be your final implementation–nor should it–however, it is possible to use it to perform sanity checks and more extensive automated testing. This file is located in the webapps
directory.
In addition to conforming the YADA index to your environment, the YADA.properties
file, which contains the YADA index configuration is also conformed, and located in webapps/YADA-Quickstart-<version>/WEB-INF/classes
.
Note: If you choose to redeploy the war file to another tomcat instance, you may need to reset some of the configuration in
YADA.properties
and in the YADA index.
Refer to the Sanity Check to validate your install. If you have issues, check Additional Filesystem Configuration and Caveats.
You will soon run the maven package
goal.
First, however, you have an opportunity to ensure maven builds the war file with the correct contextual values for your environment. Find the build.properties
file in the appropriate YADA-Quickstart/src/main/resources/
subdirectory (i.e., local
or dev
,) and modify per your environment.
If the local
properties file is left unchanged, the default local
build will create a warfile identical to the downloadable version referenced in the previous section. It is preferable to modify build.properties
before building, rather than to modify YADA.properties
and context.xml
afterward. This is because in the future, you can automate deployment of the warfile using a variety of methods, including maven-cargo-plugin, custom scripts, or continuous integration (CI) services, and you won't want to, or won't be enabled to modify YADA.properties
or context.xml
between build and deployment. Information about the settings in build.properties
can be found inline, in the file.
After making your desired changes, simply run mvn package
in the YADA-Quickstart
directory. This will result in a YADA-Quickstart-<version>.war
file in the target
subdirectory. This warfile can now be copied to your tomcat container.
To confirm YADA is running, access the following url:
http://host.domain:port/yada.jsp?q=YADA+default&py=true
Obviously, substitute host.domain:port
with your environment's values. q
is the alias for qname
which points to the named YADA query you will execute. py
is the alias for pretty
which "pretty-prints" the JSON string. If you're using a browser, and a browser plugin to format JSON, you can omit py=true
.
Your result should be the a JSON object string containing YADA is alive!
in your browser.
{
"RESULTSET": {
"total": 1,
"ROWS": [{"'YADAISALIVE!'": "YADA is alive!"}],
"qname": "YADA default",
"page": "1",
"records": 1
},
"version": "5.1.2_SNAPSHOT"
}
As alluded to above, there are a few filesystem touchpoints that need to be configured. If you've read the comments embedded in build.properties
, YADA.properties
, and context.xml
you already know what to expect. The easiest (and default) configuration is to create a filesystem path /apps/yada
and deploy, create, or link all other relevent directories beneath.
If you are deploying multiple instances on the same server, e.g., for dev and test, you can append the "env" value to the path, e.g., /apps/yada/local
or /apps/yada/dev
. This can also be a symlink to it's own directory.
files/in
: this is the base directory for uploadsfiles/out
: this is the base directory for i/o outpututils
: this is where YADA service scripts are deployed, e.g.,curlerWS.sh
bin
: this is where script plugins should be deployedtomcat
: this is usually a softlink to$TOMCAT_HOME
web
: if using apache, this can be a softlnk to$APACHE_HOME
, or a separate config altogether which shares the binary
Your filesystem should could like this:
/
|
+- apps
|
+- yada
|
+- (optional env subdir, e.g., local or dev, or symlink to .)
|
+- bin
|
+- utils
|
+- files
| |
| +- in
| |
| +- out
|
+- tomcat (probably symlink to $TOMCAT_HOME)
|
+- web (probably symlink to $APACHE_HOME)
YADA-Quickstart contains all the necessary jars in WEB-INF/lib
. However, some versions of Tomcat require JDBC-driver jars to be installed in $TOMCAT_HOME/lib
instead, probably so they are loaded by a different class loader (not sure why this matters.) In any event, if you're having driver issues, this could be why.
Some networks use proxy servers and others require the passing of credentials. It is highly unlikely in the initial setup and testing of JDBC queries, that you've encountered any proxy or authentication issues. Nevertheless, YADA has features to enable the use of proxies to access third party resources via REST requests on a per-request basis. YADA also enables the inclusion (pass-thru) of named cookies to internally authenticated resources on a per-request basis. If you experience proxy or authentication related issues, consult the YADA Parameters Reference, in particular the sections on the proxy and cookies params.
The YADA-Quickstart default settings include a HyperSQL® pre-populated implementation of the YADA Index. If you want to use a different database engine, namely MySQL®, PostgreSQL®, or Oracle®, you must have an instance of your desired engine at the ready, and run the appropriate database script.
Whether you cloned the YADA-Quickstart on github or downloaded the war file, you may want to view the database configuration scripts for Oracle®, MySQL®, and PostgreSQL®, HyperSQL® and SQLite®, as well as scripts for inserting the essential queries to enable YADA to work, and queries to run the TestNG tests. These scripts are in the main YADA repo in the following directory:
YADA
|
+-- yada-war
|
+-- src
|
+-- main
|
+-- resources
|
+-- scripts
|
+-- YADA_db_HSQLdb.sql
+-- YADA_db_MySQL.sql
+-- YADA_db_Oracle.sql
+-- YADA_db_PostgreSQL.sql
+-- YADA_db_SQLite.sql
+-- YADA_query_essentials.sql
+-- YADA_query_tests.sql
+-- YADA_db_uni_<old>_to_<new>.sql
The default HyperSQL® db is found in YADA-Quickstart here:
YADA-Quickstart
|
+-- src
|
+-- main
|
+-- resources
|
+-- db
|
+--YADADB
In addition to the vendor-specific configuration script, one must run YADA_query_essentials.sql
as well. YADA_query_tests.sql
need only be executed if it is intended to run the TestNG tests.
The ...db_uni_<old>_to_<new>.sql
scripts are for upgrades to existing dbs only.
YADA is a versatile platform which can be installed in a variety of environments with a range of options and dependency configurations. For example, an admin may wish to deploy YADA
- in different Tomcat versions
- in multiple instances on one machine (e.g., dev and test environments)
- remotely, or locally to the build environment
- via scp, or psi-probe, or tomcat-manager
- using a continuous integration (CI) service
- using tomcat's provided shell scripts to restart the container, or instead, the linux
service
command - with or without
sudo
protection - with or without interactive password prompting
- with or without an apache front-end to Tomcat
Just these eight options elicit hundreds of different possibilities.
The deployment script included in the YADA-Quickstart provides facilities for a variety of configuration scenarios. This script can be extended to support many other scenarios. Pull-requests for mods are encouraged.
<tr>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
<th></th>
<th></th>
<th></th>
<th>√</th>
<th></th>
</tr>
<tr>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
<th></th>
<th></th>
<th></th>
<th></th>
<th>√</th>
</tr>
<tr>
<th></th>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
<th>√</th>
<th>√</th>
<th>√</th>
<th></th>
</tr>
<tr>
<th></th>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
<th>√</th>
<th>√</th>
<th></th>
<th>√</th>
</tr>
<tr>
<th></th>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
<th>√</th>
<th></th>
<th>√</th>
<th></th>
</tr>
<tr>
<th></th>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
</tr>
<tr>
<th></th>
<th>√</th>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
<th>√</th>
<th>√</th>
<th></th>
</tr>
<tr>
<th></th>
<th>√</th>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
<th>√</th>
<th></th>
<th>√</th>
</tr>
<tr>
<th></th>
<th>√</th>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
<th></th>
<th>√</th>
<th></th>
</tr>
<tr>
<th></th>
<th>√</th>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
<th></th>
<th></th>
<th>√</th>
</tr>
env | warfile deployment | container restart | ||||||
---|---|---|---|---|---|---|---|---|
Local | Remote | scp | cp | psi-probe | ssh | sudo | script | service |
√ | √ | √ | √ | |||||
√ | √ | √ | √ |
The deployment script is typically executed by the exec-maven-plugin
. You can change the arguments passed to it by modifying the pom in the exec-maven-plugin
configuration section:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.3.2</version>
<configuration>
<skip>${skip.war.deploy}</skip>
<executable>/bin/bash</executable>
<arguments>
<!-- default arguments are equivalent to '-d psiprobe -r script'-->
<argument>${project.build.directory}/deploy.sh</argument>
<!-- uncomment the following to use scp to deploy the war file remotely -->
<!-- <argument>-d</argument>
<argument>scp</argument> -->
<!-- uncomment the following to use cp to deploy the war file locally -->
<!-- <argument>-d</argument>
<argument>cp</argument> -->
<!-- uncomment the following to use linux 'service' command instead of catalina script -->
<!-- <argument>-r</argument>
<argument>service</argument> -->
<!-- uncomment the following 'sudo' arg to use sudo locally or over ssh -->
<!-- <argument>-s</argument> -->
</arguments>
</configuration>
<executions>
<execution>
<id>restart-tomcat</id>
<phase>pre-integration-test</phase>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
</plugin>
Your environment may need to be conformed or prepared, even with the YADA-Quickstart. Here is a typical sequence of steps:
Note: You may need to
sudo
all these commands.
# Another reminder to 'sudo'
# if you're using amazon EC2, you may need to install tomcat and maven
yum info tomcat7
yum install tomcat7
yum install tomcat7
yum info maven
wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
yum install -y apache-maven
mvn --version
# create the yada *nix group
groupadd yada
# add the yada user
useradd -g yada -d /home/yada yada
# set the yada user password
passwd yada # yada-adm1n by default
# add the current user, yada, apache, and tomcat
# to eachothers' groups
usermod -G tomcat,yada apache
usermod -G apache,yada tomcat
usermod -G apache,tomcat yada
usermod -G apache,tomcat,yada <current_user>
# create the app dir and adjust privs
mkdir -p /apps/yada
chgrp yada /apps/yada
chmod 775 /apps/yada
# switch to the 'yada' user
su - yada
# create the subdirs
cd /apps/yada
mkdir -p bin files/in files/out lib log util
# create symlinks to your env, apache, and tomcat (your tomcat path may differ)
ln -s . local
ln -s /path/to/tomcat7 tomcat
ln -s /path/to/httpd web
# Now it's a good idea to provide sudo NOPASSWD access
# to the 'yada' group to enable automated execution of
# tomcat and apache restart. If you are unfamiliar with
# editing sudoers, please ask for help or consult the
# google.
As stated in the yada.httpd.conf
file itself:
This a virtual host httpd.conf file for YADA, showing some of the options that one can use when fielding yada requests with apache.
The main advantage here, other than the standard advantages of fronting tomcat with apache httpd, is support for yada i/o, mapping directories and headers for file transfers without having to rely on symlinks inside the webapp.
You'll notice there are also configs for CORS support, which tomcat could support on its own with the CORS filter, and support for path-style parameters, which tomcat supports with the URLRewrite filter.
Edit the values for your environment and drop it into /path/to/httpd/conf.d and restart apache.
Remember depending on the httpd version, config, etc. the env vars may be ignored during startup. You may need to replace them with absolute values. You also might need to build and install mod_jk.
# copy yada.httpd.conf to deployment directory
cd /apps/yada/web/conf.d
cp ~/YADA-Quickstart-<version>/src/main/resources/dev/yada.httpd.conf .
# Edit yada.httpd.conf for your environment.
# Look for 'SET THIS' comments, in particular.
# In some apache instances, 'SetEnv' directives are honored during
# configuration time. In others they are not. In some instances,
# variables are ignored only for certain settings. See the
# comments in the file to figure out what needs setting, and check
# your apache error_log as well. Also, apache might not startup if
# the path to DocumentRoot doesn't exist yet.
vi /apps/yada/web/conf.d/yada.httpd.conf
psi-probe is a great Tomcat manager. YADA's deploy.sh
script supports it–by default, in fact. It is not required for YADA warfile deployment.
Get psi-probe at https://github.com/psi-probe/psi-probe and follow the directions there. psi-probe requires modifications to tomcat-users.xml
. This file can be found in the YADA-Quickstart github repo at src/main/resources/dev
Assuming you've got maven
installed and working, here are some commands for getting psi-probe
working on an Amazon EC2 instance:
# install git
yum install git
# clone the psi-probe repo
git clone https://github.com/psi-probe/psi-probe
# run the maven package goal
cd psi-probe
mvn package
# copy the war artifact to the webapps dir
cp psi-probe/web/target/probe.war /apps/yada/tomcat/webapps
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<tomcat-users>
<!--
NOTE: By default, no user is included in the "manager-gui" role required
to operate the "/manager/html" web application. If you wish to use this app,
you must define such a user - the username and password are arbitrary.
-->
<!--
NOTE: The sample user and role entries below are wrapped in a comment
and thus are ignored when reading this file. Do not forget to remove
<!.. ..> that surrounds them.
-->
<!--
<role rolename="tomcat"/>
<role rolename="role1"/>
<user username="tomcat" password="tomcat" roles="tomcat"/>
<user username="both" password="tomcat" roles="tomcat,role1"/>
<user username="role1" password="tomcat" roles="role1"/>
-->
<role rolename="admin" />
<role rolename="manager" />
<role rolename="manager-gui"/>
<role rolename="probeuser" />
<role rolename="poweruser" />
<role rolename="poweruserplus" />
<user username="admin" password="yadapr0be" roles="admin,manager,manager-gui" />
</tomcat-users>
You can effectively just download the YADA-Quickstart and drop it into your running Tomcat instance to get started. Though you will likely want to set some custom values for properties, the defaults may work for you.
If instead, you cloned YADA-Quickstart from the github repo, you likely know it is a maven project which relies on the maven war plugin, and specifically, it's overlay feature, to create a YADA installation that is specific to your environment.
Once the repo is cloned to local filesystem, simply update the properties with your environment-specific values and build the project.
Detailed instructions follow.
As elucidated in the build.properties
, context.xml
, and YADA.properties
files, there are a few environment variables which can (and should) be used to more securely pass values to the build. These include usernames, password, et al.
In build.properties
, referenced by deployment and test scripts
user=${env.YADA_USER} # the user associated to the deployment
YADA.user=${env.YADA_USER} # the proxy user if proxy authentication is required (usu only for testing)
YADA.pass=${env.YADA_PASS} # the proxy password if proxy authentication is required (usu only for testing)
YADA.proxy=${env.YADA_PROXY} # the proxy host:port, if necessary (usu only for testing)
In src/main/resources/local/build.properties
, the default values specific to the local HyperSQL® implementation of YADA Index. This is not a secured system. When a more robust database solution is implemented that requires authentication, the property values below should replace the defaults.
The environment variable substitutions are provided already in src/main/resources/dev/build.properties
(see below.)
If you look at the test configuration in the
yada-war
submodule of theYADA
project, you'll see multiple configurations for different database engines. That may add clarity to this config, or it may confuse you more.
The properties values are substituted in src/main/webapp/WEB-INF/classes/YADA.properties
and src/main/webapp/META-INF/context.xml
during the resources build phase.
YADA.index.adaptor=${env.YADA_INDEX_ADAPTOR}
YADA.index.url=${env.YADA_INDEX_URL}
YADA.index.validationQuery=${env.YADA_INDEX_VALIDATION_QUERY}
YADA.index.driverClassName=${env.YADA_INDEX_DRIVER_CLASSNAME}
YADA.index.username=${env.YADA_INDEX_USERNAME}
YADA.index.password=${env.YADA_INDEX_PASSWORD}
The build.properties
file is well-documented. Please refer to it for required and optional property settings, necessary for building.
The pom.xml is liberally documented. Below is the profiles
section to illuminate the build logic. The pom.xml has a lot more documentation than what is shown below. A key takeaway here is that regarding the war-deploy
profile:
- When included, in concert with any profile, will execute the
deploy.sh
script with options as configured in theexec-maven-plugin
section - When omitted, the build can be configured to use the
maven-cargo-plugin
instead (quite easily in thelocal
profile–good for testing.)
<profiles>
<!-- 'local' -->
<!-- The 'local' profile is active by default. -->
<!-- Remember that passing any other profile name to maven will override
activeByDefault settings, so you will need to include the 'local' profile
name if you want to also use 'war-deploy' -->
<!-- The 'local' profile sets the ${env} property to 'local', which ultimately
is translated into the path '/resources/conf/local' in order to use the environment-specific
'build.properties' file. 'local' can also turn on 'cargo-deploy' (off by default) and turns off
auth testing, assuming that you want to deploy to a local tomcat instance,
and that it won't be behind your authentication gateway, so you'll just do
noauth, if any, testing. -->
<profile>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<id>local</id>
<properties>
<env>local</env>
<skip.cargo.deploy>true</skip.cargo.deploy>
<skip.auth.test>true</skip.auth.test>
</properties>
</profile>
<!-- 'dev' -->
<!-- The 'dev' profile sets the ${env} property to 'dev', which ultimately
is translated into the path '/resources/conf/dev' in order to use the environment-specific
'build.properties' file and 'deploy.sh' script. 'dev' does not automatically
deploy, in case you want to just run tests without wasting time deploying
the same version of the war file -->
<profile>
<id>dev</id>
<properties>
<env>dev</env>
</properties>
</profile>
<!-- 'test' -->
<!-- The 'test' profile sets the ${env} property to 'test', which ultimately
is translated into the path '/resources/conf/test' in order to use the environment-specific
'build.properties' file and 'deploy.sh' script. 'test' does not automatically
deploy, in case you want to just run tests without wasting time deploying
the same version of the war file -->
<profile>
<id>test</id>
<properties>
<env>test</env>
</properties>
</profile>
<!-- 'load' -->
<!-- Maybe you need a second internal implementation pointing to your
production environment just for ETL? This is here to illustrate that "sure,
you can do that!" -->
<profile>
<id>load</id>
<properties>
<env>load</env>
<skip.auth.test>true</skip.auth.test>
<skip.noauth.test>true</skip.noauth.test>
</properties>
</profile>
<!-- 'prod' -->
<!-- The 'prod' profile sets the ${env} property to 'prod', which ultimately
is translated into the path '/resources/conf/prod' in order to use the environment-specific
'build.properties' file for filtering and 'deploy.sh' script for deployment.
Unlike other environment settings, no tests are run by default. -->
<profile>
<id>prod</id>
<properties>
<env>prod</env>
<skip.auth.test>true</skip.auth.test>
<skip.noauth.test>true</skip.noauth.test>
</properties>
</profile>
<!-- When war-deploy is active, the exec plugin will execute with designated
environment settings. For example the command 'mvn -pdev,war-deploy' will
build your warfile, run your /resources/conf/dev/deploy.sh script, and then
run both auth and no-auth testing -->
<profile>
<id>war-deploy</id>
<properties>
<skip.war.deploy>false</skip.war.deploy>
</properties>
</profile>
</profiles>
This is the section of the pom concerning automated deployment via deploy.sh
<!-- If you choose to use the included deployment script for non-local
environments, this is where it becomes relevant, to redeploy the war file
via curl/psiprobe and restart Tomcat. -->
<!-- To deploy without testing, use 'pre-integration-test' as the goal -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.3.2</version>
<configuration>
<skip>${skip.war.deploy}</skip>
<executable>/bin/bash</executable>
<arguments>
<!-- default arguments are equivalent to '-d psiprobe -r script'-->
<argument>${project.build.directory}/deploy.sh</argument>
<!-- uncomment the following to use scp to deploy the war file remotely via scp -->
<!-- <argument>-d scp</argument> -->
<!-- uncomment the following to use cp to deploy the war file locally -->
<!-- <argument>-d cp</argument> -->
<!-- uncomment the following to use linux 'service' command instead of catalina script -->
<!-- <argument>-r service</argument> -->
<!-- uncomment the following 'sudo' arg to use sudo locally or over ssh -->
<!-- <argument>s</argument> -->
</arguments>
</configuration>
<executions>
<execution>
<id>restart-tomcat</id>
<phase>pre-integration-test</phase>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
</plugin>
# build the default 'local' package. This will not deploy anywhere.
mvn package
# build the 'dev' package.
mvn package -P dev
# build the 'local' package and deploy
# This uses 'deploy.sh' via 'exec-maven-plugin' (requires configuration–see the pom)
mvn pre-integration-test -P local,war-deploy
# build and deploy to the 'dev' server (or at least, using the 'dev'
# configuration). Again, it uses 'deploy.sh' via 'exec-maven-plugin' (requires configuration–see the pom)
mvn pre-integration-test -P dev,war-deploy
# build, deploy to the 'dev' server and run custom integration tests
mvn verify -P dev,war-deploy