Wednesday, May 20, 2015

Oracle Log Rotation


Oracle Log Rotation by Caleb Small

Copyright Ó 2008, Caleb.com

Three types of logs to deal with:

1)      Typical log files (eg. alert, sqlnet, listener, etc) that constantly grow
2)      Trace files (.trc) that are constantly being created
3)      Audit trace files.  One trc file is created for each process PID when a session starts.  When the same PID is used again in the future (which it will) the existing file is appended.

The Linux find utility is useful for locating trace files older than a certain number of days.  It can be easily configured to identify and then delete these files, for example:

$ find /oracle/app/admin -name '*.trc' -mtime +361 -exec rm -f {} \; >/dev/null 2>&1

This can be applied to the audit trace files as well but the results are unpredictable.  Trace files older than the threshold will be deleted.  However, if the threshold is very long the probability that a process PID will be re-used and hence the file date updated becomes more likely.  If create date is used instead then we are deleting audit information before its time.

Typical log files can be handled with the Linux logrotate utility.  Configuration files determine which files are rotated, how often, and when they are purged.  The setup is as follows:

1)      Create a script to be placed in the appropriate cron directory.  This script can perform the find-deletes described above, and then launch logrotate utility with configuration files specifically for Oracle logs
2)      Create a logrotate.conf file to list the Oracle files to be rotated.  Wildcards can be used but they must specifically identify the logfiles to be rotated, not just *.
3)      Create a logrotate.status file which is used by logrotate to determine when each file was last rotated.  This can either be done manually, or automatically by running logrotate with the -f (force) option the first time.  Note, however, that if additional log files are added in the future (even if wildcards would identify them) they will not be rotated unless they are listed in the status file.

Typical cron file placed in /etc/cron.daily


-rwxr-xr-x  1 oracle dba    894 Dec  6 16:59 oralogrotate

#!/bin/bash
#
# Rotate Oracle log files and purge trace files
# Copyright December 2008, Caleb.com
#
# Place this file in /etc/cron.weekly (or whatever period is desired), with root execute privs
# Modify -mtime retention settings as necessary for trace file purging
#
# Place oralogrotate.conf in /etc directory, and modify as necessary to include log files
# Run logrotate once with -f (force) option to create status file
# status file must be updated manually if new logs are added in the future
#
# STEP 1 - Purge trace files
#
# Use find to locate files older than threshold number of days and delete them
# This is useful for the likes of trace files which are constantly being created
#
/usr/bin/find /oracle/app/admin -name '*.trc' -mtime +361 -exec rm -f {} \; >/dev/null 2>&1
/usr/bin/find /oracle/app/admin -name '*.aud' -mtime +31 -exec rm -f {} \; >/dev/null 2>&1
/usr/bin/find /oracle/app/product/10.2.0.1/db_1/gamma1.psoug.org_orcl/sysman/recv -name '*.err*' -mtime +31 -exec rm -f {} \; >/dev/null 2>&1
#
# Some logs (eg core dumps) create a subdir which can be emptied by the above
# commands, but the subdirs will remain.  This will clean out empty dirs:
#
/bin/rmdir --ignore-fail-on-non-empty /oracle/app/admin/orcl/cdump/core* >/dev/null 2>&1
#
# STEP 2 - Rotate log files
#
# Use the Linux log rotator for typical log files (alert log, listener.log, etc)
# that constantly grow.
# Note that the -f (force) option may be *required* the first time in order to
# generate a status file.  If new logs are added in the future they must be added
# to the status file or they will not be rotated.
#
/usr/sbin/logrotate -v -s /var/lib/oralogrotate.status /etc/oralogrotate.conf >/home/oracle/oralogrotate_`date +%F:%T`_`hostname -s`.log 2>&1

Typical oralogrotate.conf file in /etc:

-rw-r--r-- 1 oracle dba 423 Dec  6 17:25 oralogrotate.conf

monthly
rotate 12
copytruncate
compress
/oracle/app/admin/+ASM/bdump/alert*.log {
}
/oracle/app/admin/ORCL/bdump/alert*.log {
}
/oracle/app/admin/ORCL/bdump/drc*.log {
}
/oracle/app/product/10.2.0.1/db_1/network/log/*.log {
}
/oracle/app/product/10.2.0.1/crs_1/log/gamma1/*.log {
}
/oracle/app/product/10.2.0.1/db_1/oc4j/j2ee/OC4J_DBConsole_gamma1_ORCL1/log/*.log {
}
/oracle/app/product/10.2.0.1/db_1/gamma1_ORCL1/sysman/log/emdb.nohup {
}
Note: on systems that use ASM or RAC, or other additional Oracle homes, each Oracle home must be scanned for candidate log files and added to this file.  Be careful of root permission issues on some Clusterware log files. 

Typical oralogrotate.status file in /var/lib:

-rw-r--r-- 1 root root 1110 Dec  6 17:11 oralogrotate.status

logrotate state -- version 2
"/oracle/app/admin/orcl/bdump/alert_orcl.log" 2008-12-6
"/oracle/app/product/10.2.0.1/db_1/sqlnet.log" 2008-12-6
"/oracle/app/product/10.2.0.1/db_1/network/log/listener.log" 2008-12-5
"/oracle/app/product/10.2.0.1/db_1/network/log/sqlnet.log" 2008-12-5
"/oracle/app/product/10.2.0.1/db_1/gamma1.psoug.org_orcl/sysman/log/emagent.log" 2008-12-6
"/oracle/app/product/10.2.0.1/db_1/gamma1.psoug.org_orcl/sysman/log/emagentfetchlet.log" 2008-12-6
"/oracle/app/product/10.2.0.1/db_1/gamma1.psoug.org_orcl/sysman/log/emdctl.log" 2008-12-6
"/oracle/app/product/10.2.0.1/db_1/gamma1.psoug.org_orcl/sysman/log/emoms.log" 2008-12-6
"/oracle/app/product/10.2.0.1/db_1/gamma1.psoug.org_orcl/sysman/log/emagent.trc" 2008-12-6
"/oracle/app/product/10.2.0.1/db_1/gamma1.psoug.org_orcl/sysman/log/emagentfetchlet.trc" 2008-12-6
"/oracle/app/product/10.2.0.1/db_1/gamma1.psoug.org_orcl/sysman/log/emdctl.trc" 2008-12-6
"/oracle/app/product/10.2.0.1/db_1/gamma1.psoug.org_orcl/sysman/log/emoms.trc" 2008-12-6
"/oracle/app/product/10.2.0.1/db_1/gamma1.psoug.org_orcl/sysman/log/emdb.nohup" 2008-12-6

Note: this file can be created manually, or by running oralogrotate the first time with the -f (force) option which causes logs to be rotated whether it is needed or not.   As new log files are added to the configuration, or to the system under wildcard directories, they must be added to the status file or they will not be rotated.