OptimIT

One of our projects used legacy log4j 1.2 as the main logging framework, which has been not been developed since 2015. Thus we needed to migrate large codebase to newer logging framework. Our decision was to migrate API calls to SLF4J facade, and core implementation to latest log4j2, which seemed more feature rich than its main alternative, logback.

The easy thing was changing pom.xml file. Log4j 1.2 dependency is removed and replaced with appropriate dependencies needed to integrate SLF4J with Log4j2:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.11.2</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
   <artifactId>log4j-core</artifactId>
    <version>2.11.2</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-slf4j-impl</artifactId>
    <version>2.11.2</version>
</dependency>
   

The second easy step was changing API references from legacy log4j loggers in the codebase. That step was done using multifile search/replace operations, where we searched for:

import org.apache.log4j.Logger;

Which we replaced with:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

and

Logger.getLogger

and

LoggerFactory.getLogger

Of course, these bulk search replace operations depend on the common code expressions used in the project.

A little bit more complex step was migration of our custom configuration code, since Log4j2 does not have classes DOMConfigurator or PropertyConfigurator which have been used so far in our configuration class. We refactored it to use LoggerContext and Configuration objects.

Afterward, we needed to replace existing utility bridges between Java logging from java.util.logging package to Log4j and Logback to Log4j with standard JUL to SLF4J bridge. To bridge JUL logs to SLF4J, Log4J2 is used as SLF4J implementation provider. That means all log messages send via JUL API will redirect to SLF4J layer which will then use Log4J2 as implementation provider to output the messages. To enable that, in pom.xml we must add new jul-to-slf4j bridge dependency:

                <dependency>
    		    <groupId>org.slf4j</groupId>
    		    <artifactId>jul-to-slf4j</artifactId>
    		    <version>1.7.2</version>
		</dependency>

Finally, the last step was replacing existing log4j.xml configuration file with log4j2.xml. That was the most complex part of the migration, since new version uses quite different syntax than legacy xml schema. For example, <log4j:configuration xmlns:log4j=’http://jakarta.apache.org/log4j/’> is replaced with <Configuration> and <appender name=”STDOUT” class=”org.apache.log4j.ConsoleAppender”> is replaced with <Console name=”STDOUT” target=”SYSTEM_OUT”>.

For most projects such conversion would be done manually by the developer, but we faced with a requirement to upgrade many existing installations to new format. To automate this process, we wanted to develop a script to make a conversion of XML to new format.

Initially we started with bash script, with a sequence of sed and xmlstarlet invocations to do a conversion.

For example, conversion of Configuration element which is mentioned before is made with the next line/command:

sed -i '/<log4j:configuration/c\<Configuration>' $TARGET_FILE

It will replace the line inside $TARGET_FILE which contains pattern <log4j:configuration with <Configuration> and this change will be made in-place ( -i ), directly inside $TARGET_FILE.

Deleting the line with Sed is performed as follows:

sed -i '/DOCTYPE/d' $TARGET_FILE

Line inside $TARGET_FILE with DOCTYPE pattern will be deleted.

To replace one pattern with another one within the entire file:

sed -i 's/appender-ref/AppenderRef/g' $TARGET_FILE

More complex actions required variables and loops are performed with xmlstarlet. For example, we have more than one RollingFile appender. With xmlstarlet we would first find number of those appenders and then iterate over them to perform individual search replace operations: Because there are more elements named param, we need one more for loop to iterate through all of them to get a certain value based on attribute name:

ROLLINGFILE_COUNT=$(xmlstarlet sel -t -v "count(//RollingFile)" $TARGET_FILE)
for i in `seq 1 $ROLLINGFILE_COUNT`
do
   DATE_PATTERN=""
   PARAM_COUNT=$(xmlstarlet sel -t -v 'count(//RollingFile'[$i]'/param)' $TARGET_FILE)
   for j in `seq 1 $PARAM_COUNT`
   do
      PARAM_NAME=$(xmlstarlet sel -t -v '//RollingFile'[$i]'/param'[$j]'/@name' $TARGET_FILE)
      if [ "File" == $PARAM_NAME ]
      then
         FILENAME=$(xmlstarlet sel -t -v '//RollingFile'[$i]'/param'[$j]'/@value' $TARGET_FILE)
         xmlstarlet ed --inplace -i '//RollingFile'[$i] -t attr -n fileName -v $FILENAME $TARGET_FILE
      elif [ "ConversionPattern" == $PARAM_NAME ]
      then
         …
      done
      xmlstarlet ed --inplace -d '//RollingFile'[$i]'/param' $TARGET_FILE
done

Although this concept can be straightforward, we wanted to use more scalable approach that would rely on XSL transformations. After writing appropriate XSL file as a transform file, processing was done with the xsltproc which is a command line tool for applying XSLT stylesheets to XML documents.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output indent="yes"/>
<xsl:template match="*[local-name() = 'configuration']">
<xsl:element name="Configuration">
   <xsl:element name="Appenders">

      <!-- CONSOLE -->
      <xsl:for-each select="appender[@class='org.apache.log4j.ConsoleAppender']">
         <xsl:element name="Console">
            <xsl:attribute name="name"><xsl:value-of select="@name"/></xsl:attribute>
            <xsl:attribute name="target">SYSTEM_OUT</xsl:attribute>
            <xsl:element name="PatternLayout">
                 <xsl:attribute name="pattern">
                    <xsl:if test="layout/param/@name='ConversionPattern'">
                     <xsl:value-of select="layout/param/@value"/>
                  </xsl:if>
                 </xsl:attribute>
              </xsl:element>
          </xsl:element>
       </xsl:for-each>

       <!-- ROLLING FILE -->
      <xsl:for-each select="appender[@class='org.apache.log4j.DailyRollingFileAppender']">
         <xsl:element name="RollingFile">
            <xsl:attribute name="name"><xsl:value-of select="@name"/></xsl:attribute>
            <xsl:choose>
                 <xsl:when test="param[@name='DatePattern']">
                   <xsl:attribute name="filePattern">
                       <xsl:value-of select="concat(param[@name='File']/@value, '.%d{', param[@name='DatePattern']/@value, '}.gz')"/>
                       </xsl:attribute>
                 </xsl:when>
                 <xsl:otherwise>
                    <xsl:attribute name="filePattern"><xsl:value-of select="concat(param[@name='File']/@value, '.%i.gz')"/></xsl:attribute>
                 </xsl:otherwise>
             </xsl:choose>
            <xsl:for-each select="param">
               <xsl:if test="@name='File'">
                  <xsl:attribute name="fileName">
                     <xsl:value-of select="@value"/>
                     </xsl:attribute>
               </xsl:if>
               <xsl:if test="@name='Append'">
                  <xsl:attribute name="append">
                     <xsl:value-of select="@value"/>
                  </xsl:attribute>
               </xsl:if>
               <xsl:if test="@name='DatePattern'">
                  <xsl:element name="Policies">
                       <xsl:element name="TimeBasedTriggeringPolicy"/>
                    </xsl:element>
               </xsl:if>
               <xsl:if test="@name='MaxBackupIndex'">
                  <xsl:element name="DefaultRolloverStrategy">
                     <xsl:attribute name="max">
                          <xsl:value-of select="@value"/>
                       </xsl:attribute>
                    </xsl:element>
               </xsl:if>
               <xsl:if test="@name='MaxFileSize'">
                  <xsl:element name="Policies">
                     <xsl:element name="SizeBasedTriggeringPolicy">
                        <xsl:attribute name="size">
                             <xsl:value-of select="@value"/>
                          </xsl:attribute>
                       </xsl:element>
                    </xsl:element>
               </xsl:if>
            </xsl:for-each>
            <xsl:element name="PatternLayout">
                 <xsl:attribute name="pattern">
                    <xsl:if test="layout/param/@name='ConversionPattern'">
                     <xsl:value-of select="layout/param/@value"/>
                  </xsl:if>
                 </xsl:attribute>
              </xsl:element>
          </xsl:element>
       </xsl:for-each>

       <xsl:for-each select="appender[@class='org.apache.log4j.RollingFileAppender']">
         <xsl:element name="RollingFile">
            <xsl:attribute name="name"><xsl:value-of select="@name"/></xsl:attribute>
            <xsl:choose>
                 <xsl:when test="param[@name='DatePattern']">
                    <xsl:attribute name="filePattern">
                       <xsl:value-of select="concat(param[@name='File']/@value, '.%d{', param[@name='DatePattern']/@value, '}.gz')"/>
                       </xsl:attribute>
                 </xsl:when>
                 <xsl:otherwise>
                    <xsl:attribute name="filePattern">
                       <xsl:value-of select="concat(param[@name='File']/@value, '.%i.gz')"/>
                    </xsl:attribute>
                 </xsl:otherwise>
             </xsl:choose>
            <xsl:for-each select="param">
               <xsl:if test="@name='File'">
                  <xsl:attribute name="fileName">
                     <xsl:value-of select="@value"/>
                     </xsl:attribute>
               </xsl:if>

               <xsl:if test="@name='Append'">
                  <xsl:attribute name="append">
                     <xsl:value-of select="@value"/>
                  </xsl:attribute>
               </xsl:if>
               <xsl:if test="@name='DatePattern'">

                  <xsl:attribute name="datePattern">
                     <xsl:value-of select="@value"/>
                  </xsl:attribute>
                  <xsl:element name="Policies">
                       <xsl:element name="TimeBasedTriggeringPolicy"/>
                    </xsl:element>
               </xsl:if>
               <xsl:if test="@name='MaxBackupIndex'">
                  <xsl:element name="DefaultRolloverStrategy">
                     <xsl:attribute name="max">
                          <xsl:value-of select="@value"/>
                       </xsl:attribute>
                    </xsl:element>
               </xsl:if>
               <xsl:if test="@name='MaxFileSize'">
                  <xsl:element name="Policies">
                     <xsl:element name="SizeBasedTriggeringPolicy">
                        <xsl:attribute name="size">
                             <xsl:value-of select="@value"/>
                          </xsl:attribute>
                       </xsl:element>
                    </xsl:element>
               </xsl:if>
            </xsl:for-each>
            <xsl:element name="PatternLayout">
                 <xsl:attribute name="pattern">
                    <xsl:if test="layout/param/@name='ConversionPattern'">
                     <xsl:value-of select="layout/param/@value"/>
                  </xsl:if>
                 </xsl:attribute>
              </xsl:element>
          </xsl:element>
       </xsl:for-each>

       <!-- FILE -->
       <xsl:for-each select="appender[@class='org.apache.log4j.FileAppender']">
         <xsl:element name="File">
            <xsl:attribute name="name"><xsl:value-of select="@name"/></xsl:attribute>
            <xsl:choose>
                 <xsl:when test="param[@name='DatePattern']">
                    <xsl:attribute name="filePattern">
                       <xsl:value-of select="concat(param[@name='File']/@value, '.%d{', param[@name='DatePattern']/@value, '}.gz')"/>
                       </xsl:attribute>
                 </xsl:when>
                 <xsl:otherwise>
                    <xsl:attribute name="filePattern">
                       <xsl:value-of select="concat(param[@name='File']/@value, '.%i.gz')"/>
                    </xsl:attribute>
                 </xsl:otherwise>
             </xsl:choose>

            <xsl:for-each select="param">
               <xsl:if test="@name='File'">
                  <xsl:attribute name="fileName">
                     <xsl:value-of select="@value"/>
                     </xsl:attribute>
               </xsl:if>
               <xsl:if test="@name='Append'">
                  <xsl:attribute name="append">
                     <xsl:value-of select="@value"/>
                  </xsl:attribute>
               </xsl:if>
               <xsl:if test="@name='DatePattern'">
                  <xsl:attribute name="datePattern">
                     <xsl:value-of select="@value"/>
                  </xsl:attribute>
                  <xsl:element name="Policies">
                       <xsl:element name="TimeBasedTriggeringPolicy"/>
                    </xsl:element>
               </xsl:if>
               <xsl:if test="@name='MaxBackupIndex'">
                  <xsl:element name="DefaultRolloverStrategy">
                     <xsl:attribute name="max">
                          <xsl:value-of select="@value"/>
                       </xsl:attribute>
                    </xsl:element>
               </xsl:if>
               <xsl:if test="@name='MaxFileSize'">
                  <xsl:element name="Policies">
                     <xsl:element name="SizeBasedTriggeringPolicy">
                        <xsl:attribute name="size">
                             <xsl:value-of select="@value"/>
                          </xsl:attribute>
                       </xsl:element>
                    </xsl:element>
               </xsl:if>
            </xsl:for-each>
            <xsl:element name="PatternLayout">
                 <xsl:attribute name="pattern">
                    <xsl:if test="layout/param/@name='ConversionPattern'">
                     <xsl:value-of select="layout/param/@value"/>
                  </xsl:if>
                 </xsl:attribute>
              </xsl:element>
          </xsl:element>
       </xsl:for-each>
   </xsl:element>

   <!-- LOGGERS -->
   <xsl:element name="Loggers">
      <xsl:for-each select="logger">
         <xsl:element name="Logger">
              <xsl:attribute name="name">
                 <xsl:value-of select="@name"/>
              </xsl:attribute>
              <xsl:attribute name="level">
               <xsl:value-of select="level/@value"/>
            </xsl:attribute>
            <xsl:if test="appender-ref">
            <xsl:element name="AppenderRef">
                 <xsl:attribute name="ref">
                    <xsl:value-of select="appender-ref/@ref"/>
                 </xsl:attribute>
              </xsl:element>
           </xsl:if>
           </xsl:element>
      </xsl:for-each>
   </xsl:element>
</xsl:element>
</xsl:template>
</xsl:stylesheet>

Similar to bash script, we need two for loops for each type of appender. One loop is to iterate through all the appenders of the same type and one to iterate through all the elements param inside the appender.

For each type of appender that we support, we have a separate part of transformation, but that mapping of the attributes and parameters is quite similar. The attached transformation covers typical appenders that were common in our setup (File, Console and RollingFile appenders), but it can be easily extended to handle additional appenders if needed.

After selection of element param, we can check what is its name and based on the name, put its value inside corresponding attribute of appender. We found it easier to manage xml elements with XSLT, so it took less time to write XSLT converter than Bash script.