Kuali Batch Processing Configuration

The KFS Production and UAT batch servers each contain two special scripts that are owned by the Linux OS user "kfsctmuser". This is the user that executes communications between the KFS application and the Control-M scheduling application.

The scripts wrapper.ksh and runDbScripts.sh both reside in /home/kfsctmuser and are needed for proper batch execution.

There is also a cron job that periodically runs on this server under the ROOT crontab that adjusts folder level permissions to ensure certain batch jobs (that transfer files) operate properly. The script /usr/local/sbin/correct_perms.sh validates permissions on an ACL to allow the "kfsctmuser" resource id to access contents of the directories.

[root@kfs20 prd.batch]# crontab -l
0,15 3,10,18,21 * * * /usr/local/sbin/dsm_check
#0,15 3,10,18,21 * * * /usr/sbin/rhn_check
0,15,30,45,59 * * * * /usr/local/sbin/correct_perms.sh
0 * * * * /usr/local/sbin/correct_perms_nathanhale.sh
0 6 * * * /root/jasper/restore-jasper-files.sh
## for the power outage
#0 2 31 5 * chkconfig tomcat6 off
#0 1 1 6 * chkconfig tomcat6 on
# Run system activity accounting tool every minute
*/1 * * * * /usr/lib64/sa/sa1 1 1
# 0 * * * * /usr/lib64/sa/sa1 600 6 &
# Generate a daily summary of process accounting at 23:53
53 23 * * * /usr/lib64/sa/sa2 -A

Ad Hoc Batch Processing

"Ad Hoc" batch jobs can be run through the KFS front end by users, which makes a manual call to the batch nodes (uat.batch or prd.batch) via an sshfs file system mount.

Nightly Batch Processing

Nightly batch processing is driven by jobs that have been configured in Control-M.

Control-M kicks off the scheduled jobs and will remote to the batch server node via ssh.  It uses the ".wrapper.ksh" script along with the particular job name. This will invoke Ant on the batch node to run the job using a "jobrunner" class that is within the Kuali code on the batch server.

Scripts for the Control-M batch scheduling are located in the Home folder of the KFSCTMUSER on the batch servers (~/kfsctmuser)

Tomcat's Role in Batch Processing

UAT.BATCH and PRD.BATCH both have Tomcat instances.

The Tomcat instance is there as a servlet container to house the .WAR file on the batch server. The expanded .WAR file MUST be on the Batch nodes because that houses the code for some of the libraries and dependencies needed by the batch jobs that are run by Control-M.

The Tomcat instance on both of the Batch nodes should remain in DOWN state. Having Tomcat running will not disrupt or damage batch operations, but Tomcat will consume resources while running that can otherwise be utilized by batch processing.

Within the Tomcat configuration on the Batch nodes, under the file configuration-defaults-config.xml the Rice batch mode configuration is set to TRUE. This prevents the server node from distributing service calls to itself and inadvertently polluting the KSB (Kuali Service Bus).

<param name="rice.ksb.batch.mode">true</param>

Wrapper.ksh script

The script 'wrapper.ksh' (AIX korn shell) exists on each batch node and defines configuration and parameters for how Control-M executes batch jobs and interacts with the KFS system.

• job.status.directory=/home/kfsctmuser
• wrapper.ksh in /home/kfsctmuser
• permissions to /srv/uconn_configs/kfs-config/ logs and work

Errors in Batch Processing

Batch jobs can occasionally encounter errors or hang due to certain known issues.

• Control-M never receives a 'complete' or 'fail' signal from the wrapper.sh script that operates batch processing.
• The Java process that wrapper.sh script invokes never exits completely or gracefully, which causes the job to hang and not complete.