==NOTE: The Plugin API has been revised for release 7.1.0. Only the pl
parameter is required now, and arguments formerly passed with now deprecated a
, pr
, pa
, or b
parameters will instead be appended to the pl
parameter, after the plugin class-name and a comma. Read on for details.==
YADA plugins are intended, like most plugin implementations, to extend functionality in unexpected, unpredicted (to api authors), and incredibly useful ways.
One of the first plugin implementations for YADA included a "Loader" or ETL class which calls the Oracle® sqlldr
to move millions of rows of RNASeq gene expression data into an Oracle® database, first transforming the data from output generated by Cufflinks.
Another plugin exported pharmacology data, again from Oracle®, into a opendoc-compatible (.docx) Word document using a template stored as a resource.
There are two plugin API at your disposal:
A Java® API which enables a developer to leverage the internal YADA methods without having to use http. Typically these are installed locally to the server as additions to the com.novartis.opensource.yada.plugin
package. Java® plugins are introspected using the Reflection API, and thus can be evaluated at the request level or at the query level, meaning a multi-query request can include a query that requires plugin execution, without adversely affecting it's co-requested queries.
Enables any os-compatible executable stored on the server in the correct location (yada.bin) to be executed by the YADA service. The ScriptPlugin API actually is an abstraction of the Java® Plugin API, using special Java® classes to make system calls to desired scripts. For the moment, Script Plugins can only be evaluated at the request level.
Request-level plugins execute in red boxes
To call a YADA Plugin, simply use the plugin
or pl
parameter, with the class name or FQCN of the plugin class as the value:
# Long parameter name
hostname:port/yada.jsp?q=YADA test SELECT&plugin=TestPreprocessor
# Short parameter name
hostname:port/yada.jsp?q=YADA test SELECT&pl=TestPreprocessor
# Fully Qualified Class Name (FQCN)
hostname:port/yada.jsp?q=YADA test SELECT&pl=com.novartis.opensource.yada.plugin.TestPreprocessor
To pass arguments to a plugin class or script, simply append the a comma-separated list to the pl
parameter, after the class name, and a comma:
hostname:port/yada.jsp?q=YADA test SELECT&pl=TestPreprocessor,abc,def
The following information is @deprecated as of 7.1.0. It is here for context, reference, and backward-compatibility. For new installations, it can be ignored.
To pass arguments to the plugin class, use the appropriate arg parameter:
args
: for any Java® plugin–the plugin type, pre, post, bypass is autodetected, so the generic args param is usefulpreargs
,pr
: for any preprocessor pluginpostargs
,pa
: for any postprocessor pluginbypassargs
,b
: for any bypass plugin
The following examples would resolve identically:
# args parameter hostname:port/yada.jsp?q=YADA test SELECT&pl=TestPreprocessor&args=abc,def
hostname:port/yada.jsp?q=YADA test SELECT&pl=TestPreprocessor&preargs=abc,def
As described earlier, plugins are separated into three categories, each represented by a java interface:
for transformation of configuration parameters before queries are executed
for transformation of YADA results before they are delivered to the requesting client
to circumvent the conventional YADA process flow and effectively, do anything
Each interface has a single overloaded method: engage
.
While the Java® interface determines the type of plugin to be executed (pre, post, bypass), it is the engage
method signature that determines whether to engage the plugin at the request or query level.
There also exists an abstract implementation of each interface:
to serve as a superclass for either request-level or query-level preprocessor plugin subclasses
to serve as a superclass for either request-level or query-level postprocessor plugin subclasses
to serve as a superclass for either request-level or query-level bypass plugin subclasses
As the engage
method is overloaded in each interface, it is better to create a plugin by extension of an abstract superclass, to override only the method relevant to the use case, i.e., request-level or query-level method signature, rather than implementing both overloaded versions of engage
, only to use one.
// com.novartis.opensource.yada.plugin.Preprocess request-level method
public YADARequest engage(YADARequest yadaReq) throws YADAPluginException;
// com.novartis.opensource.yada.plugin.Preprocess query-level method
public YADARequest engage(YADARequest yadaReq, YADAQuery yq) throws YADAPluginException;
public abstract class AbstractPreprocessor implements Preprocess
{
/**
* Null implementation
* @see com.novartis.opensource.yada.plugin.Preprocess#engage(com.novartis.opensource.yada.YADARequest)
*/
@Override
public YADARequest engage(YADARequest yadaReq)
throws YADAPluginException
{
return null;
}
/**
* Null implementation
* @see com.novartis.opensource.yada.plugin.Preprocess#engage(com.novartis.opensource.yada.YADARequest, com.novartis.opensource.yada.YADAQuery)
*/
@Override
public void engage(YADARequest yadaReq, YADAQuery yq)
throws YADAPluginException
{
/* Nothing to do */
}
}
// com.novartis.opensource.yada.plugin.Postprocess request-level method
public String engage(YADARequest yadaReq, String result) throws YADAPluginException
// com.novartis.opensource.yada.plugin.Postprocess query-level method
public String engage(YADAQuery yq) throws YADAPluginException
public abstract class AbstractPostprocessor implements Postprocess
{
/**
* Null implementation
* @see com.novartis.opensource.yada.plugin.Postprocess#engage(com.novartis.opensource.yada.YADAQuery)
*/
@Override
public void engage(YADAQuery yq)
throws YADAPluginException
{
/* nothing to do */
}
/**
* Null implementation
* @throws YADAPluginException when there is a processing error
* @see com.novartis.opensource.yada.plugin.Postprocess#engage(com.novartis.opensource.yada.YADARequest, java.lang.String)
*/
@Override
public String engage(YADARequest yadaReq, String result)
throws YADAPluginException
{
return null;
}
}
// com.novartis.opensource.yada.plugin.Bypass request-level method
public String engage(YADARequest yadaReq) throws YADAPluginException
// com.novartis.opensource.yada.plugin.Bypass query-level method
public YADARequest engage(YADARequest yadaReq, YADAQuery yq) throws YADAPluginException;
public abstract class AbstractBypass implements Bypass
{
/**
* Null implementation
* @see com.novartis.opensource.yada.plugin.Bypass#engage(com.novartis.opensource.yada.YADARequest)
*/
@Override
public String engage(YADARequest yadaReq)
throws YADAPluginException
{
return null;
}
/**
* Null implementation
* @see com.novartis.opensource.yada.plugin.Bypass#engage(com.novartis.opensource.yada.YADARequest, com.novartis.opensource.yada.YADAQuery)
*/
@Override
public YADAQueryResult engage(YADARequest yadaReq, YADAQuery yq)
throws YADAPluginException
{
return null;
}
}
The Preprocess
request-level implementation takes the original YADARequest
as an argument, passed from the Service._execute
method, and returns the modified version of the request, or an equivalent.
// com.novartis.opensource.yada.plugin.Preprocess request-level method
public YADARequest engage(YADARequest yadaReq) throws YADAPluginException;
Here's an example of a request-level preprocessor. This is the implementation of the engage
method in com.novartis.opensource.yada.plugin.TestPreprocessor
/**
* Executes the query defined in {@code yadaReq} and analyzes the results. If the
* result count is 0, a new request with an {@code INSERT} is query is created. If
* the result count is >0, a new request with a {@code DELETE} query is created.
* The new reqeust is then returned from the method.
* @see com.novartis.opensource.yada.plugin.Preprocess#engage(com.novartis.opensource.yada.YADARequest)
*/
@Override
public YADARequest engage(YADARequest yadaReq) throws YADAPluginException
{
// store the original qname
String qname = yadaReq.getQname();
// create a new request object
YADARequest lyadaReq = new YADARequest();
// set the qname
lyadaReq.setQname(new String[] { qname });
// execute the new request with the original qname
Service svc = new Service(lyadaReq);
// store the result
String result = svc.execute();
l.debug(result);
JSONObject res;
try
{
// eval the result as json
res = new JSONObject(result);
// if the result doesn't contain data, create another new request to insert it
if(res.has("RESULTSET") && res.getJSONObject("RESULTSET").getInt("total") == 0)
{
lyadaReq = new YADARequest();
lyadaReq.setQname(new String[] { "YADA test INSERT" });
lyadaReq.setParams(new String[] { "A,10,7.5,26-SEP-2014" });
}
// if the result contains data, create a new request to delete it
else if(res.has("RESULTSET") && res.getJSONObject("RESULTSET").getInt("total") > 0)
{
lyadaReq = new YADARequest();
lyadaReq.setQname(new String[] { "YADA test DELETE" });
}
else
// if the result is non-conforming, throw an exception
{
throw new YADAPluginException("Plugin failed.");
}
}
catch (JSONException e)
{
throw new YADAPluginException("Unable to parse result.");
}
// plugins shouldn't stomp on other plugins
lyadaReq.setPlugin(yadaReq.getPluginConfig());
// return the new locally created request object
return lyadaReq;
}
}
The Preprocess
query-level implementation takes both the YADARequest
and currently processing YADAQuery
object as arguments, and again, returns a modified version of the request, or an equivalent. A Preprocess
query-level implementation is ideal for use as a security gatekeeper, to limit execution of specified queries to authorized users. See the Security Guide for details and examples.
// com.novartis.opensource.yada.plugin.Preprocess query-level method
public YADARequest engage(YADARequest yadaReq, YADAQuery yq) throws YADAPluginException;
The Postprocess
request-level implementation takes the existing YADARequest
and the latest "return-ready" String
result arguments, passed from the Service._execute
method, and returns, typically, a transformed version of result. However, it could return any String
.
// com.novartis.opensource.yada.plugin.Postprocess request-level method
public String engage(YADARequest yadaReq, String result) throws YADAPluginException;
Here is an example of a postprocessor. This is the implementation of the engage
method in com.novartis.opensource.yada.plugin.TestPostprocessor
:
/**
* Simply returns the string {@code It worked.}
* @see com.novartis.opensource.yada.plugin.Postprocess#engage(com.novartis.opensource.yada.YADARequest, java.lang.String)
*/
@Override
public String engage(YADARequest yadaReq, String result) throws YADAPluginException
{
// return an arbitrary string
return "It worked.";
}
Obviously, a postprocessor can be much more complex. The YADARequest
object can contain a pa
or postargs
array containing arguments pertinent to the plugin, and the plugin can also leverage any other parameter contained in the request. Here's a more complex example. This is the engage
method of the class com.novartis.opensource.yada.XSLPostprocessor
included in the library, which relies on the postargs
array.
/**
* Instantiates a {@link TransformerFactory}, gets a new {@link Transformer} using the stylesheet
* passed in the request's {@link YADARequest#postargs}, transforms {@code result} and returns the output.
* @see com.novartis.opensource.yada.plugin.Postprocess#engage(com.novartis.opensource.yada.YADARequest, java.lang.String)
*/
@Override
public String engage(YADARequest yadaReq, String result) throws YADAPluginException {
String xmlInput = result;
String xslResult = "";
String xsl = "";
StringWriter output = new StringWriter();
// 1. Instantiate a TransformerFactory.
TransformerFactory tFactory = TransformerFactory.newInstance();
// 2. Use the TransformerFactory to process the stylesheet Source and generate a Transformer.
Transformer transformer;
try
{
// this is where 'postargs' value in YADARequest is used
List<String> args = yadaReq.getPostArgs();
for(String arg:args)
{
if (arg.endsWith(".xsl"))
xsl = arg;
}
transformer = tFactory.newTransformer(new javax.xml.transform.stream.StreamSource(new Finder().getEnv("yada.util")+xsl));
// 3. Use the Transformer to transform an XML Source and send the output to a Result object.
setTransformerParameters(args, transformer);
transformer.transform
(new StreamSource(new StringReader(xmlInput)),
new StreamResult(output));
xslResult = output.getBuffer().toString();
changeResultFormat(args, yadaReq);
}
catch (TransformerConfigurationException e)
{
e.printStackTrace();
throw new YADAPluginException(e.getMessage());
}
catch (TransformerException e)
{
e.printStackTrace();
throw new YADAPluginException(e.getMessage());
}
catch (YADAResourceException e)
{
e.printStackTrace();
throw new YADAPluginException(e.getMessage());
}
return xslResult;
}
The Postprocess
query-level implementation performs the same role as it's request-level sibling, but obviously with a single query result, extracted from the YADAQuery
object, rather than with the entire post-format, request result.
// com.novartis.opensource.yada.plugin.Postprocess query-level method
public String engage(YADAQuery yq) throws YADAPluginException
TODO query-level postprocessor example
The Bypass
request-level implementation takes the original YADARequest
as an argument, passed from the Service._execute
method, and returns any String
suitable as a the payload of an HTTP response. Due to the dynamic discovery and execution of plugins, if a Bypass
implementation returns anything other than null, Service._execute
will halt further execution and immediately return the result.
// Bypass method signature
public String engage(YADARequest yadaReq) throws YADAPluginException;
Here is an example of a postprocessor. This is the implementation of the engage method in com.novartis.opensource.yada.plugin.TestBypass
:
/**
* Simply returns the string {@code Bypass worked."}
* @see com.novartis.opensource.yada.plugin.Bypass#engage(com.novartis.opensource.yada.YADARequest)
*/
@Override
public String engage(YADARequest yadaReq) throws YADAPluginException
{
return "Bypass worked.";
}
The Bypass
query-level implementation works in similar fashion but rather than returning from Service._execute
immediately, the Service
object will simply iterate immediately to the next query.
// com.novartis.opensource.yada.plugin.Bypass query-level method
public YADARequest engage(YADARequest yadaReq, YADAQuery yq) throws YADAPluginException;
TODO Bypass query-level example
Script plugins adhere to the same contracts as Java® plugins. In fact, they are facilitated by built-in Java® plugins which utilize java.lang.Process
objects to make system calls to registered scripts operating in supported scripting environments. Script plugins differ (on the Java® side) by the extra step of marshaling arguments and return values to and from the runtime and system environments. Their implementations, however, are simple.
Note: Script plugins, for the moment, may only be used at the request level, not at the query level.
To call a script plugin, simply use the included script plugin classes as the first argument to the pl
or plugin
parameter, the script name as the 2nd argument, and any other arguments subsequently.
- The
pl
orplugin
parameter is now required. - The first element of the comma-delimited value of this parameter must be one of the following:
- ScriptPreprocessor
- ScriptPostprocessor
- ScriptBypass
- The second element of the comma-separated value of this parameter must be the path to the executable script, relative to the
yada.bin
directory. - Any proceeding arguments should adhere to the order expected by the script denoted in the first argument
- The arguments passed to the script will be the "stringified" versions of the values passed to the built-in Java® handler plugins' engage methods
- For both
ScriptPreprocessor
andScriptBypass
this will be a JSON representation of theYADARequest
object - For
ScriptPostprocessor
this will be both a JSON representation of theYADARequest
object and theString
result of the query - The return values of scripts must adhere to the proper syntax
- For
ScriptPreprocessor
the return value must be a JSON string convertible into aYADARequest
object. See the script preprocessor section for details. - For both
ScriptPostprocessor
andScriptBypass
, the return value must be a string result effectively intended to be returned to the client
# the following calls are equivalent
# with both plugin, script, args
http://example.com/yada.jsp?qname=myquery&plugin=ScriptPostprocessor,myscript.pl,arg1,arg2
# with alias
http://example.com/yada.jsp?qname=myquery&pl=ScriptPostprocessor,myscript.pl,arg1,arg2
The following information is @deprecated as of 7.1.0. It is here for context, reference, and backward-compatibility. For new installations, it can be ignored.
To call a script plugin, simply indicate the script name, and it's arguments, in the appropriate "args" parameter in the request:
http://example.com/yada.jsp?qname=myquery&postargs=myscript.pl,arg1,arg2
Note the following rules for calling and processing a Script Plugin:
- The
plugin
orpl
parameter is optional. If omitted, one of the following specific argument parameters must be used:preargs
,pr
,postargs
,pa
,bypassargs
,b
- If the
plugin
orpl
parameter is included, the genericargs
ora
parameter may be used- The first value in the argument parameter must be the path to the executable script, relative to the
yada.bin
directory- Any proceeding arguments should adhere to the order expected by the script denoted in the first argument
- The arguments passed to the script will be the "stringified" versions of the values passed to the built-in Java® handler plugins' engage methods
- For both
ScriptPreprocessor
andScriptBypass
this will be a JSON representation of theYADARequest
object- For
ScriptPostprocessor
this will be both a JSON representation of theYADARequest
object and theString
result of the query- The return values of scripts must adhere to the proper syntax
- For
ScriptPreprocessor
the return value must be a JSON string convertible into aYADARequest
object. See the script preprocessor section for details.- For both
ScriptPostprocessor
andScriptBypass
, the return value must be a string result effectively intended to be returned to the client
As shown in the code samples below, dynamic type detection enables the exclusion of the
plugin
, orpl
request parameter when calling script plugins. If the parameter is excluded, however, it is critical to use the correct type-specific "args" parameter.
# the following calls are equivalent # with both plugin and type-specific args http://example.com/yada.jsp?qname=myquery&plugin=ScriptPostprocessor&postargs=myscript.pl,arg1,arg2
http://example.com/yada.jsp?qname=myquery&plugin=ScriptPostprocessor&args=myscript.pl,arg1,arg2
http://example.com/yada.jsp?qname=myquery&postargs=myscript.pl,arg1,arg2
http://example.com/yada.jsp?q=myquery&pl=ScriptPostprocessor&pa=myscript.pl,arg1,arg2
http://example.com/yada.jsp?q=myquery&pl=ScriptPostprocessor&a=myscript.pl,arg1,arg2
http://example.com/yada.jsp?q=myquery&pa=myscript.pl,arg1,arg2
Internally, the engage
method will accept the YADARequest
object from the Service
object. It will then pass it to the designated script as a string representation of parameter map obtained from the HttpServletRequest
. In fact, the string version of the request is nothing more then the result of calling getParameterMap().toString()
on the HttpServletRequest
object in the YADARequest
. The syntax is defined by ServletRequest.getParameterMap:
an immutable java.util.Map containing parameter names as keys and parameter values as map values. The keys in the parameter map are of type String. The values in the parameter map are of type String array.
This String
is marshalled into a JSON object with parameter names as keys, and parameter value arrays as values, and then passed to the script as a string.
The string returned by the script must comply with the same JSON syntax. Here is a Perl example:
#!/usr/bin/perl
# call with an url like the following:
# http://example.com/yada.jsp?q=YADA%20test%20SELECT&pr=scriptPluginPreTest.pl,YADA%20default&c=false
my $qname = $ARGV[0];
# The string returned must be a representation of the
# ServletRequest.getParameterMap() return value, with
# String objects for keys, and String arrays for values.
print '{ "qname" : ["'.$qname.'"] }';
In ScriptPostprocessor
execution, first the String result
value is written to a temporary file. Then the plugin arguments are put into a List
. Then the absolute path the file containing the result is appended. The list is then passed to the plugin script executable.
Here is a perl example of a ScriptPostprocessor.
First the request used to call the plugin:
# some things to note:
# a. no 'pl' or 'plugin' parameter was used
# YADA deduces a script plugin named 'scriptPluginPostTest.pl' is
# required by the used of the use of the 'pa' (postargs) param
# b. no additional arguments follow the script name, so the plugin
# itself should expect the file containing the query result
# in $ARGV[0]
http://example.com/yada.jsp?q=YADA test SELECT&pa=scriptPluginPostTest.pl&c=false
#!/usr/bin/perl
# note the filename is passed, in this case, in $ARGV[0]
# the filename will always be in the last $ARGV index
open INFILE, $ARGV[0] || die "oops";
while (<INFILE>) {
chomp;
$_ =~ s/RESULTSET/PLUGINWASHERE/;
# The output of the script is retrieved by YADA and passed back
# to the Service object as a result
print $_;
}
Obviously, anything is possible in this script. The author could leverage arguments passed in, execute additional queries, call an R
analysis, convert the data to a proprietary format, etc.
The ScriptBypass
is kind of a combination of the pre- and post-processors. It takes the YADARequest
object as an argument to engage
and returns a String
suitable for a response.
Here is the content of ScriptPluginBypassTest.pl
. You can see it dispenses entirely with input, and returns an abitrary string.
#!/usr/bin/perl
print "What is this, velvet?\n";
Obviously, anything is possible in this script. The author could leverage arguments passed in the url, or in other YADA parameters.
Here is flowchart of the Service._execute
method as it relates to plugin discovery and execution. Some steps have been omitted for clarity.
Here is the pertinent snippet from Service._execute
. Some code has been omitted for clarity.
// engage global bypass
gResult = engageBypass(this.getYADARequest());
if (gResult != null)
{
// if a Bypass is discovered and returns anything non-null
// the Service will stop processing and return immediately
return gResult;
}
// engage global preprocessor
engagePreprocess(getYADARequest());
// iterate over the queries in the request
for(YADAQuery yq : this.qMgr.getQueries())
{
// engage query bypass
engageBypass(yq);
YADAQueryResult yqr = yq.getResult();
if (yqr != null)
{
// if there is now a result in the query,
// skip further processing and move on to the next query
continue;
}
// engage query preprocessor
engagePreprocess(yq);
// execute query
yq.getAdaptor().execute(yq);
.
.
.
// engage query postprocessor
engagePostprocess(yq);
}
.
.
.
// build response (creating response payloads)
gResult = composeResponse();
// engage global postprocessor
gResult = engagePostprocess(gResult);
.
.
.