[Placeholder for all my notes about the RAC srvctl and crs_* commands] * In 10g, if sqlplus can start/stop instance but srvctl cannot, check $ORACLE_HOME/log/`hostname -s`/racg/imon.log for any message. 11g doesn't have this file. See https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=844272.1 * The best way to turn on srvctl trace is SRVM_TRACE=1 srvctl ... That is, if using bash (or any Bourne-derived shell), precede srvctl with `SRVM_TRACE='. This is better than `export SRVM_TRACE=' followed by srvctl because turning off trace is `unset SRVM_TRACE', not `export SRVM_TRACE=false', which still turns on trace. Read srvctl script to know why. * srvctl should always run as the database software owner, most likely oracle, not root. If you accidentally run it as root, e.g. `srvctl add database...', the `crs_getperm ora..db' shows the owner to be root. Correct it with `crs_setperm ora..db -o oracle'. Also check file and directory ownership from $ORACLE_HOME down, particularly $ORACLE_HOME/log: find $ORACLE_HOME -user root -ls Correct them as needed: find $ORACLE_HOME -user root -exec chown oracle:oinstall {} \; (A few files such as oradism are better owned by root so you can change them back to root, if desired.) * In 11g before 11.2.0.2, due to Bug 9251136 "INSTANCE WILL NOT USE HUGEPAGE IF STARTED BY SRVCTL". The root cause is process ulimit settings (as in /etc/security/limits.conf) are not used. To confirm, check /proc//limits on Linux or plimit on Solaris. Add ulimit -l to $CRS_HOME/bin/ohasd or /etc/init.d/ohasd, but not /etc/profile or /etc/init.d/init.ohasd to work around the bug. The bug is fixed in 11.2.0.2. * Sometimes a resource cannot be removed ("does not exist") and cannot be added ("already exists"). Try `srvctl remove ... -f' to force remove first. If that does not work, use crs_unregister command. * When a resource's state is UNKNOWN, try `srvctl stop ...'. The state should change to OFFLINE. Then you can `srvctl start ...'. * VIP should depend on listener, and instance on ASM. To add dependency of, e.g. instance, on ASM: srvctl modify instance -d -i -s +ASM In 10g, there should not be any other dependency, such as ASM on VIP. If yes, remove it per Note:4865736.8: ASM resource name is in the form of ora...asm. ASM instance name does not include "+" sign VIP resource name is in the form of ora..vip - Run "crs_stat -p > /tmp/.cap". Note file name must be .cap - Edit /tmp/.cap to remove VIP resource name from the REQUIRED_RESOURCES attribute. I.e. leave the right part of the equal sign empty. - Run "crs_register -u -dir /tmp" to actually remove the dependency info in OCR - Use "crs_stat -p | grep REQUIRED" to verify REQUIRED_RESOURCE attribute is updated (i.e. emptied). If you don't remove it, stopping nodeapps will crash ASM and then instance. * Use these commands to check all dependencies: 10g: crs_stat -p | grep -v =$ | egrep '^NAME|^REQUIRED_RESOURCES' or with indentation for easy reading: crs_stat -p | grep -v =$ | perl -nle 'if(/^NAME/){print}elsif(/^REQUIRED_RESOURCES/){print "\t$_"}' 11g: crsctl stat res -p | grep -v =$ | egrep '^NAME|DEPENDENCIES' or with indentation for easy reading: crsctl stat res -p | grep -v =$ | perl -nle 'if(/^NAME/){print}elsif(/DEPENDENCIES/){print "\t$_"}'