Jenkins
- Benjamin Daniels
The Jenkins CI server provides Continuous Integration build and deployment services for Kuali and Kuali-related applications. It is structured to automate the SDLC and monthly release processes for KFS and RICE services as outlined below.
Build Process
All instances currently build as Maven projects in Jenkins.
- Kuali RICE builds as a native Maven project. It is composed of a parent POM.XML with multiple child POM's
- Kuali KFS builds natively in ANT. However, the UConn DevOps team has devised a Maven "wrapper" (pom.xml) that invokes ANT using the Maven ant-run plugin. This allows the use of Maven targets in the project while still invoking ANT for the core of the KFS build.
In the legacy Jenkins environment, the builds were performed on jen2.uits.uconn.edu (Jenkins slave node)
Deploy Process
All systems currently deploy through Jenkins. They are built locally to the Jenkins node (devops.uconn.edu) under /jenkins_home/jobs and are deployed at build time by each job itself.
- Kuali RICE has all deployment configuration built into the Jenkins build/deploy process.
- Kuali KFS has a post-processing shell script that runs as part of the KFS jobs in Jenkins. When the .WAR file "lands" at its destination, the script "kfsdeployer.sh" is run and it performs multiple actions.
- Sources the local environment variables from the Tomcat configuration file
- Manually unzips the .WAR file (this preempts the Tomcat instance from doing it on startup)
- Sets the KFS version & revision numbers for the GUI
- Adjusts the .XSD files to utilize the "localhost:8080" path, so they will work on any environment, including developer's local systems, since we do not employ a Static Content Server for XSD files
Jenkins Deployment Template Scripts
The below scripts are used as "post-build steps" in the Jenkins jobs. These are executed for the Deploy portion of the jobs after the Build portion successfully completes.
RICE
Non load balanced RICE template:
Load balanced RICE template:
KFS
Non load balanced KFS template:
Load balanced KFS template:
Release Process
Creating a Release is a special process where a numbered version of RICE and/or KFS is "released" into the production environment. There are extra steps involved in creating a Release that go beyond standard build and deployment. For information on those steps please review the section linked below.
PERFORMING A RELEASE
Configuration Notes
Notes for the configuration of the current Jenkins installation including steps to install the instance.
Plugins
- Ant Plugin
- CAS Plugin
- Credentials Plugin
- Email Ext Plugin
- External Monitor Job Type Plugin
- Javadoc Plugin
- Jenkins CVS Plugin
- Jenkins Jira Issue Updater Plugin
- Jenkins JIRA Plugin
- Jenkins Mailer Plugin
- Jenkins Subversion Plug-n
- Jenkins svnmerge Plugin
- Jenkins Translation Assistance Plugin
- LDAP Plugin
- Maven Project Plugin
- PAM Authentication plugin
- SSH Credentials Plugin
- SSH Slaves Plugin
- Token Macro Plugin
New Installation
start with a new copy of template server.
stop tomcat (sudo /sbin/service tomcat6 stop)
wget jenkins.war from jenkins-ci.org
move jenkins.war to /usr/share/tomcat6/webapps if not already there
set $JENKINS_HOME environment variable in /etc/tomcat6/tomcat6.conf add the following line (at the bottom is fine):
CATALINA_OPTS="-DJENKINS_HOME=/srv/jenkins_home" |
change this:
<Connector port="8080" protocol="HTTP/1.1"connectionTimeout="20000"redirectPort="8443" /> |
to this:
<Connector port="8080" protocol="HTTP/1.1"connectionTimeout="20000"redirectPort="8443" URIEncoding="UTF-8"/> |
move or delete /usr/share/tomcat6/lib/cas*.jar
now start tomcat (sudo /sbin/service tomcat6 start)
and go to the URL:
http://jenkins1.uits.uconn.edu:8080/jenkins/
:
Setup cas: (currently not working reporting ("Application Not Authorized to Use CAS")
go to 'manage jenkins' -> "manage plugis" -> "available tab" -> search for "CAS" -> check "CAS plugin" (not cas1), and press the install button.
after restart "manage jenkins" -> "setup security" -> under security realm choose CAS protocal, and fill in cas url as: https://login.uconn.edu/cas/
setup ldap:
"manage jenkins" -> "setup security" ->
Build Node Setup (including master)
in the $HOME/.m2 folder must have a settings.xml file:
<settings xmlns="http://maven.apache.org/POM/4.0.0"<servers> <server> <id>uconn-maven</id> <username>jenkins</username> </server> <server> <id>uconn-maven-snapshots</id> <username>jenkins</username> </server></servers> </settings> |
What this does is tells maven (the mvn command) that when it is connecting to the uconn-maven (or uconn-maven-snapshots) repository to use the username 'jenkins' durring the ssh file copy
Will also need to add the oracle jar to the maven repository
if it's not on the system, export it from svn (requires kinit) and then run maven install:
svn export https://svn0.uits.uconn.edu/kuali/KFS/vendor/rice/kul-cfg-dbs/tags/v1.0.3/rev198/export-to-svn/lib/ojdbc14.jarmvn install:install-file -DgroupId=com.oracle -DartifactId=ojdbc14 -Dversion=10.2.0.3.0 -Dpackaging=jar -Dfile=ojdbc14.jar -DgeneratePom=true |
MVN setup:
wget latest maven
cd /usr/share/;tar -xvf *.tar.gz;rm *.tar.gz; |
add this to .bashrc
export M2_HOME=/usr/share/apache-maven-3.1.0export M2=$M2_HOME/binexport JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64PATH=$M2:$JAVA_HOME/bin:$PATH:$HOME/bin |
source .bashrc
New Jenkins setup
need axis jars for the JIRA configuration to work: http://mirrors.ibiblio.org/apache/axis/axis/java/1.4/axis-bin-1_4.tar.gz
wget the above url then tar -xvf it, then copy the jars in axis/lib mentioned below to to /usr/share/tomcat/lib :
for maven add a file /srv/jenkins_home/settings.xml
for setting Global MAVEN_OPTS under Maven Project Configuration
also ext email plugin will need mail jar otherwise we get mime message error (wget http://java.net/projects/javamail/downloads/download/javax.mail.jar)
also for mail make a /srv/uconn_configs/email-templates folder
make a file in there called uconn-html.jelly
<j:jelly xmlns:j="jelly:core" xmlns:st="jelly:stapler" xmlns:d="jelly:define">
<STYLE>
BODY, TABLE, TD, TH, P {
font-family:Verdana,Helvetica,sans serif;
font-size:11px;
color:black;
}
h1 { color:black; }
h2 { color:black; }
h3 { color:black; }
TD.bg1 { color:white; background-color:#0000C0; font-size:120% }
TD.bg2 { color:white; background-color:#4040FF; font-size:110% }
TD.bg3 { color:white; background-color:#8080FF; }
TD.test_passed { color:blue; }
TD.test_failed { color:red; }
TD.console { font-family:Courier New; }
</STYLE>
<BODY>
<j:set var="spc" value="&nbsp;&nbsp;" />
<!-- GENERAL INFO -->
<TABLE>
<TR><TD align="right">
<j:choose>
<j:when test="${build.result=='SUCCESS'}">
<IMG SRC="${rooturl}static/e59dfe28/images/32x32/blue.gif" />
</j:when>
<j:when test="${build.result=='FAILURE'}">
<IMG SRC="${rooturl}static/e59dfe28/images/32x32/red.gif" />
</j:when>
<j:otherwise>
<IMG SRC="${rooturl}static/e59dfe28/images/32x32/yellow.gif" />
</j:otherwise>
</j:choose>
</TD><TD valign="center"><B style="font-size: 200%;">BUILD ${build.result}</B></TD></TR>
<TR><TD>Build URL</TD><TD><A href="${rooturl}${build.url}">${rooturl}${build.url}</A></TD></TR>
<TR><TD>Project:</TD><TD>${project.name}</TD></TR>
<TR><TD>Date of build:</TD><TD>${it.timestampString}</TD></TR>
<TR><TD>Build duration:</TD><TD>${build.durationString}</TD></TR>
</TABLE>
<BR/>
<!-- CHANGE SET -->
<j:set var="changeSet" value="${build.changeSet}" />
<j:if test="${changeSet!=null}">
<j:set var="hadChanges" value="false" />
<TABLE width="100%">
<TR><TD class="bg1" colspan="2"><B>CHANGES</B></TD></TR>
<j:forEach var="cs" items="${changeSet}" varStatus="loop">
<j:set var="hadChanges" value="true" />
<j:set var="aUser" value="${cs.hudsonUser}"/>
<TR>
<TD colspan="2" class="bg2">${spc}Revision <B>${cs.commitId?:cs.revision?:cs.changeNumber}</B> by
<B>${aUser!=null?aUser.displayName:cs.author.displayName}: </B>
<B>(${cs.msgAnnotated})</B>
</TD>
</TR>
<j:forEach var="p" items="${cs.affectedFiles}">
<TR>
<TD width="10%">${spc}${p.editType.name}</TD>
<TD>${p.path}</TD>
</TR>
</j:forEach>
</j:forEach>
<j:if test="${!hadChanges}">
<TR><TD colspan="2">No Changes</TD></TR>
</j:if>
</TABLE>
<BR/>
</j:if>
<!-- ARTIFACTS -->
<j:set var="artifacts" value="${build.artifacts}" />
<j:if test="${artifacts!=null and artifacts.size()>0}">
<TABLE width="100%">
<TR><TD class="bg1"><B>BUILD ARTIFACTS</B></TD></TR>
<TR>
<TD>
<j:forEach var="f" items="${artifacts}">
<li>
<a href="${rooturl}${build.url}artifact/${f}">${f}</a>
</li>
</j:forEach>
</TD>
</TR>
</TABLE>
<BR/>
</j:if>
<!-- MAVEN ARTIFACTS -->
<j:set var="mbuilds" value="${build.moduleBuilds}" />
<j:if test="${mbuilds!=null}">
<TABLE width="100%">
<TR><TD class="bg1"><B>BUILD ARTIFACTS</B></TD></TR>
<j:forEach var="m" items="${mbuilds}">
<TR><TD class="bg2"><B>${m.key.displayName}</B></TD></TR>
<j:forEach var="mvnbld" items="${m.value}">
<j:set var="artifacts" value="${mvnbld.artifacts}" />
<j:if test="${artifacts!=null and artifacts.size()>0}">
<TR>
<TD>
<j:forEach var="f" items="${artifacts}">
<li>
<a href="${rooturl}${mvnbld.url}artifact/${f}">${f}</a>
</li>
</j:forEach>
</TD>
</TR>
</j:if>
</j:forEach>
</j:forEach>
</TABLE>
<BR/>
</j:if>
<!-- JUnit TEMPLATE -->
<j:set var="junitResultList" value="${it.JUnitTestResult}" />
<j:if test="${junitResultList.isEmpty()!=true}">
<TABLE width="100%">
<TR><TD class="bg1" colspan="2"><B>JUnit Tests</B></TD></TR>
<j:forEach var="junitResult" items="${it.JUnitTestResult}">
<j:forEach var="packageResult" items="${junitResult.getChildren()}">
<TR><TD class="bg2" colspan="2">
Name: ${packageResult.getName()} Failed:
${packageResult.getFailCount()} test(s), Passed:
${packageResult.getPassCount()} test(s), Skipped:
${packageResult.getSkipCount()} test(s), Total:
${packageResult.getPassCount()+packageResult.getFailCount()+packageResult.getSkipCount()}
test(s)</TD></TR>
<j:forEach var="failed_test" items="${packageResult.getFailedTests()}">
<TR bgcolor="white"><TD class="test_failed" colspan="2"><B><li>Failed: ${failed_test.getFullName()} </li></B></TD></TR>
</j:forEach>
</j:forEach>
</j:forEach>
</TABLE>
<BR/>
</j:if>
<!-- COBERTURA TEMPLATE -->
<j:set var="coberturaAction" value="${it.coberturaAction}" />
<j:if test="${coberturaAction!=null}">
<j:set var="coberturaResult" value="${coberturaAction.result}" />
<j:if test="${coberturaResult!=null}">
<table width="100%"><TD class="bg1" colspan="2"><B>Cobertura Report</B></TD></table>
<table width="100%"><TD class="bg2" colspan="2"><B>Project Coverage Summary</B></TD></table>
<table border="1px" class="pane">
<tr>
<td>Name</td>
<j:forEach var="metric" items="${coberturaResult.metrics}">
<td>${metric.name}</td>
</j:forEach>
</tr>
<tr>
<td>${coberturaResult.name}</td>
<j:forEach var="metric" items="${coberturaResult.metrics}">
<td data="${coberturaResult.getCoverage(metric).percentageFloat}">${coberturaResult.getCoverage(metric).percentage}%
(${coberturaResult.getCoverage(metric)})
</td>
</j:forEach>
</tr>
</table>
<j:if test="${coberturaResult.sourceCodeLevel}">
<h2>Source</h2>
<j:choose>
<j:when test="${coberturaResult.sourceFileAvailable}">
<div style="overflow-x:scroll;">
<table class="source">
<thead>
<tr>
<th colspan="3">${coberturaResult.relativeSourcePath}</th>
</tr>
</thead>
${coberturaResult.sourceFileContent}
</table>
</div>
</j:when>
<j:otherwise>
<p>
<i>Source code is unavailable</i>
</p>
</j:otherwise>
</j:choose>
</j:if>
<j:forEach var="element" items="${coberturaResult.childElements}">
<j:set var="childMetrics" value="${coberturaResult.getChildMetrics(element)}"/>
<table width="100%"><TD class="bg2" colspan="2">Coverage Breakdown by ${element.displayName}</TD></table>
<table border="1px" class="pane sortable">
<tr>
<td>Name</td>
<j:forEach var="metric" items="${childMetrics}">
<td>${metric.name}</td>
</j:forEach>
</tr>
<j:forEach var="c" items="${coberturaResult.children}">
<j:set var="child" value="${coberturaResult.getChild(c)}"/>
<tr>
<td>
${child.xmlTransform(child.name)}
</td>
<j:forEach var="metric" items="${childMetrics}">
<j:set var="childResult" value="${child.getCoverage(metric)}"/>
<j:choose>
<j:when test="${childResult!=null}">
<td data="${childResult.percentageFloat}">${childResult.percentage}%
(${childResult})
</td>
</j:when>
<j:otherwise>
<td data="101">N/A</td>
</j:otherwise>
</j:choose>
</j:forEach>
</tr>
</j:forEach>
</table>
</j:forEach>
</j:if>
<BR/>
</j:if>
<!-- CONSOLE OUTPUT -->
<j:getStatic var="resultFailure" field="FAILURE" className="hudson.model.Result"/>
<j:if test="${build.result==resultFailure}">
<TABLE width="100%" cellpadding="0" cellspacing="0">
<TR><TD class="bg1"><B>CONSOLE OUTPUT</B></TD></TR>
<j:forEach var="line" items="${build.getLog(100)}"><TR><TD class="console">${line}</TD></TR></j:forEach>
</TABLE>
<BR/>
</j:if>
</BODY>
</j:jelly>
Local Repository Maintenance
The local .M2 repository on the Jenkins server (devops.uconn.edu) needs periodic maintenance to ensure that the /home file system does not fill up. This is due to Maven (on Jenkins) storing artifacts it creates in its local repo. If the file system reaches 80% usage or higher it needs to be cleaned. This can be done by going to path /home/tomcat/.m2/repository/edu/uconn and cleaning out the tagged and snapshot artifacts to a level of N -2. Artifacts should be removed for both KFS and RICE.
Ariel Kogan says:
For those ones getting the following exceptions while configuring Jira access in...