Extensible JIRA mail parser using Milyn Smooks
JIRA can be configured to automatically create issues or comments on existing issues based on incoming messages received by a mail server or external mail service. The way JIRA extracts issue fields from the incoming emails cannot be modified nor can it be configured to extract data to the custom fields. This is problematic in the environments where JIRA represents a sink that receives email based notifications from external systems (for example Network Monitoring Systems, such as Zenoss or Nagios). One can write his own mail handler and have the complete freedom to use the desired incoming email format, along with any number of custom fields. Disadvantage of this approach is that all data processing and custom field mappings has to be hardcoded into the custom mail handler. This can be avoided using the software called Milyn Smooks. Smooks is an extensible framework for building applications for processing XML and non XML data (CSV, EDI, Java etc) using Java.
Smooks Reader
We have extended Smooks with a simple reader which parses key-value formatted emails and produces XML data. This reader implements SmooksXMLReader class, and its parse method is the most interesting:
public void parse(InputSource customInputSource) throws IOException, SAXException {
if(contentHandler == null) {
throw new IllegalStateException("'contentHandler' not set. Cannot parse stream.");
}
if(executionContext == null) {
throw new IllegalStateException("'executionContext' not set. Cannot parse stream.");
}
try {
.
.
// initialize stream reader and line reader
.
.
// Start the document and add the root element...
contentHandler.startDocument();
contentHandler.startElement(XMLConstants.NULL_NS_URI, rootName, StringUtils.EMPTY, EMPTY_ATTRIBS);
while (true)
{
String elementName;
String elementValue;
.
.
// parse elementName and elementValue
.
.
AttributesImpl attrs = new AttributesImpl();
if (elementName != null && !elementName.trim().equals("")
&& elementValue != null && !elementValue.trim().equals(""))
{
contentHandler.startElement(XMLConstants.NULL_NS_URI, elementName, StringUtils.EMPTY, attrs);
contentHandler.characters(elementValue.toCharArray(), 0, elementValue.length());
contentHandler.endElement(XMLConstants.NULL_NS_URI, elementName, StringUtils.EMPTY);
}
}
// Close out the root element and end the document..
contentHandler.endElement(XMLConstants.NULL_NS_URI, rootName, StringUtils.EMPTY);
contentHandler.endDocument();
} finally {
// These properties need to be reset for every execution (e.g. when reader is pooled).
contentHandler = null;
executionContext = null;
}
}Reader also implements a simple configuration, allowing to enumerate keys and key to XML element mappings.
XSD configuration
XML data is then validated and transformed by Smooks, using only Smooks configuration files. Main configuration file is called smooks-config.
For this purpose our configuration namespace is called ‘optimit-custom’. We have created configuration XSD called optimit-custom-1.0.xsd, configuration XSD for your component that extends the base http://www.milyn.org/xsd/smooks-1.1.xsd configuration namespace. Also, we have created Smooks configuration namespace mapping configuration file that maps the custom name-space configuration into a SmooksResourceConfiguration instance (optimit-custom-1.0.xsd-smooks.xml). Details on extending Smooks configuration XSDs can be found here.
xmlns:optimit-custom="http://www.milyn.org/xsd/smooks/optimit-custom-1.0.xsd"
xmlns:rules="http://www.milyn.org/xsd/smooks/rules-1.0.xsd"
xmlns:validation="http://www.milyn.org/xsd/smooks/validation-1.0.xsd"
xmlns:jb="http://www.milyn.org/xsd/smooks/javabean-1.4.xsd">Reader configuration
Here we configure our custom reader’s allowed keys and their mapping to XML element names.
<optimit-custom:reader rootName="issue">
<optimit-custom:keyMap>
<optimit-custom:key from="Issue type" to="issue-type" />
<optimit-custom:key from="Heading" to="heading" />
<optimit-custom:key from="Priority" to="priority" />
<optimit-custom:key from="Comments" to="comments" />
<optimit-custom:key from="Description" to="description" />
<optimit-custom:key from="Submitted by" to="submitted-by" />
<optimit-custom:key from="Email" to="email" />
<optimit-custom:key from="Site" to="site" />
<optimit-custom:key from="Entity Code" to="entity-code" />
<optimit-custom:key from="Start Date Of Problem" to="start-date-of-problem" />
<optimit-custom:key from="End Date Of Problem" to="end-date-of-problem" />
</optimit-custom:keyMap>
</optimit-custom:reader>Rule definitions
Rules provide validation for input data. Smooks supports simple regex rules or more powerful MVEL rules. Rules are grouped into ruleBases.
<!-- Define the ruleBases that are used by the validation rules... -->
<rules:ruleBases>
<!-- Order business rules using MVEL expressions... -->
<rules:ruleBase name="Standard Issue" src="rules/standard-issue.csv" provider="org.milyn.rules.mvel.MVELProvider"/>
</rules:ruleBases>CSV rules file contains rule name and MVEL code for the rule. Example file for “Standard Issue”:
"issue-type","issue.issueType == "Standard Issue""
"entity","issue.customFields["Entity"] != null"Bean configuration
We are using Smooks Java binding to generate objects, namely objects of custom bean class hr.optimit.CustomIssueCreationHelperBean. This bean contains properties for built-in JIRA fields and a map containing custom fields. In Smooks configuration we are using jb:bean tag to describe Java binding beans. Note how the custom fields map is wired to the issue bean. Also, you can see the example use of value mappings, expressions and date decoders.
<jb:bean beanId="issue" class="hr.optimit.CustomIssueCreationHelperBean" createOnElement="issue">
<jb:value property="issueType" data="issue/issue-type" decoder="Mapping">
<jb:decodeParam name="SI">Standard Issue</jb:decodeParam>
<jb:decodeParam name="CR">Change Request</jb:decodeParam>
<jb:decodeParam name="AR">Annual Report</jb:decodeParam>
</jb:value>
<jb:value property="heading" data="issue/heading" />
<jb:value property="priority" data="issue/priority" decoder="Mapping">
<jb:decodeParam name="1">Blocker</jb:decodeParam>
<jb:decodeParam name="2">Critical</jb:decodeParam>
<jb:decodeParam name="3">Major</jb:decodeParam>
<jb:decodeParam name="4">Minor</jb:decodeParam>
</jb:value>
<jb:value property="comments" data="issue/comments" />
<jb:value property="description" data="issue/description" />
<jb:value property="submittedBy" data="issue/submitted-by" />
<jb:value property="email" data="issue/email" />
<jb:wiring property="customFields" beanIdRef="customFields" />
</jb:bean>
<jb:bean beanId="customFields" class="java.util.HashMap" createOnElement="issue">
<jb:expression property="Entity" execOnElement="issue/entity-code">
Entity = [ "{\"id\":\"" + _VALUE + "\",\"type\":\"entity\"}" ];
</jb:expression>
<jb:value property="Site" data="issue/site" />
<jb:value property="Start Date Of Problem" data="issue/start-date-of-problem" decoder="Date">
<jb:decodeParam name="format">yyyy/MM/dd HH:mm</jb:decodeParam>
<jb:decodeParam name="locale">en_US</jb:decodeParam>
</jb:value>
<jb:value property="End Date Of Problem" data="issue/end-date-of-problem" decoder="Date" >
<jb:decodeParam name="format">yyyy/MM/dd HH:mm</jb:decodeParam>
<jb:decodeParam name="locale">en_US</jb:decodeParam>
</jb:value>
</jb:bean>Validation configuration
Validation configuration defines what rules are applying on which fragments. Also, it defines the behaviour in case of rule failure.
<validation:rule executeOn="issue" name="Standard Issue.issue-type" onFail="ERROR"/>
<validation:rule executeOn="issue" name="Standard Issue.entity" onFail="ERROR"/>
</smooks-resource-list>Once we have Smooks configuration file, it’s time to integrate Smooks into our JIRA mail handler. First we obtain Smooks object via ComponentManager (it is deployed as an OSGi bundle). Then we create execution context, set default locale, create Java result object and validation result object and execute smooks.filterSource to transform the source email. Java bean is retrieved from the execution context. Validation result object should be inspected in the case of validation errors.
Smooks smooks = ComponentManager.getOSGiComponentInstanceOfType(Smooks.class);
ExecutionContext executionContext = smooks.createExecutionContext();
Locale.setDefault(new Locale("en", "US"));
JavaResult javaResult = new JavaResult();
ValidationResult validationResult = new ValidationResult();
String messageBody = MailUtils.getBody(message);
smooks.filterSource(executionContext, new StringSource(messageBody), javaResult, validationResult);
logger.debug(javaResult.getBean("issue"));
CustomIssueCreationHelperBean reportBean = executionContext.getBeanContext().getBean(CustomIssueCreationHelperBean.class);
for (OnFailResult result : validationResult.getErrors())
{
RuleEvalResult rule = result.getFailRuleResult();
// Look if this rule applies to our report type
if (rule.getRuleProviderName().equals(reportBean.getReportType()))
{
String mesg = "Email validation failed for " + rule.getRuleProviderName() + "." + rule.getRuleName() + ":" + result.getFailFragmentPath();
logger.error(mesg);
throw new ValidationException(mesg);
}
}Creating issue from the CustomIssueCreationHelperBean is straightforward, the only interesting bit is the creation of the custom field values:
Map<String, Object> customFields = reportBean.getCustomFields();
for (String customFieldName : customFields.keySet())
{
CustomField customField = componentManager.getCustomFieldManager().getCustomFieldObjectByName(customFieldName);
if (customField != null)
{
Object valueObject = customFields.get(customFieldName);
// Dates cannot be directly set
if (valueObject instanceof Date)
{
Timestamp timestamp = new Timestamp(((Date) valueObject).getTime());
customField.createValue(issueObj, timestamp);
}
else
{
customField.createValue(issueObj, valueObject);
}
}
else
{
logger.warn("Email report has value for non-existant custom field \"" + customFieldName + "\"");
}
}