Issues about Program Startup Sequence on UNIX/Linux

[Update 2013-02]

Oracle 11.2.0.3, the simplest case: single node database, no ASM. All we need to do is 
Make sure the database has Y flag in /etc/oratab.
cd /etc/rc3.d; ln -s ../init.d/oracle S99oracle
cd /etc/rc6.d; ln -s ../init.d/oracle K01oracle
and in /etc/init.d/oracle,
change
	su - $ORA_OWNER -c "/u01/app/oracle/product/11.2.0/db/ora_start.sh"
	...
	su - $ORA_OWNER -c "/u01/app/oracle/product/11.2.0/db/ora_stop.sh"
to
	su - $ORA_OWNER -c "/u01/app/oracle/product/11.2.0/db/bin/dbstart /u01/app/oracle/product/11.2.0/db > /tmp/dbstart.log 2>&1"
	...
	su - $ORA_OWNER -c "/u01/app/oracle/product/11.2.0/db/bin/dbshut /u01/app/oracle/product/11.2.0/db > /tmp/dbshut.log 2>&1"
and append
	;;
    esac
to the end. Adjust path as needed. (First argument to dbstart|dbshut is passed to ORACLE_HOME_LISTNER [sic] 
variable to start|stop listener. ora_start.sh and ora_stop.sh don't even exist. Oracle's sloppiness in 
scripting is beyond my imagination!) dbstart and dbshut have their own logs under $ORACLE_HOME. But for 
troubleshooting purposes I like to catch any remaining messages, hence /tmp/dbstart.log and /tmp/dbshut.log.
By the way, Oracle's dbstart and dbshut finally got processing oratab right: W flag is honored!

If /etc/init.d/oracle does not exist, create it as follows:

#!/bin/bash

ORA_OWNER=oracle

RETVAL=0
case "$1" in
    'start')
         su - $ORA_OWNER -c "/u01/app/oracle/product/11.2.0/db/bin/dbstart /u01/app/oracle/product/11.2.0/db > /tmp/dbstart.log 2>&1"
         touch /var/lock/subsys/oracle
         ;;
     'stop')
         su - $ORA_OWNER -c "/u01/app/oracle/product/11.2.0/db/bin/dbshut /u01/app/oracle/product/11.2.0/db > /tmp/dbshut.log 2>&1"
         rm -f /var/lock/subsys/oracle
         ;;
     *)
         echo $"Usage: $0 {start|stop}"
         RETVAL=1
         ;;
esac

[End of Update 2013-02]

[Update 2011-02]
Even in 11gR2 documentation
http://download.oracle.com/docs/cd/E11882_01/server.112/e10839/strt_stp.htm
Oracle still puts crap in dbora:
1) The "rsh $HOST..." part most likely will fail; if using ssh, have to configure key to avoid 
password prompt; better yet, just comment out the whole big if-block
2) The line
$ORACLE_HOME/bin/dbstart $ORACLE_HOME &
should be changed to
su - oracle -c "$ORACLE_HOME/bin/dbstart $ORACLE_HOME" &
Do the same to dbshut line.
3) Symbolic links should include runlevel 6, e.g.
cd /etc/rc6.d; ln -s /etc/init.d/dbora K01dbora
[End of Update 2011-02]

[Update 2009-10]
Key points about non-RAC DB instance using ASM:

* In /etc/inittab, move the init.cssd from bottom to above the "rc 3" line:

#Change according to Note:264235.1
h1:35:respawn:/etc/init.d/init.cssd run >/dev/null 2>&1 </dev/null
l3:3:wait:/etc/rc.d/rc 3

* /etc/rc3.d/S99dbora points to a script with line like:

        su - $ORACLE -c "$ORACLE_HOME/bin/dbstart $ORACLE_HOME"

where $ORACLE_HOME is for the ASM, not DB, so the listener is started from ASM's $ORACLE_HOME 
(this argument doesn't affect DB or ASM startup, only the listener). No extra line for ASM. 
It's better to sleep tens of seconds in the script as suggested in Note:264235.1.

* Both ASM and DB instances in /etc/oratab are set to Y in the last field. Ignore the advice in 
the buggy $ORACLE_HOME/bin/dbstart to set DB instance to W! The ASM instance (probably) should 
be above the DB instance in the file.

* dbstart has its own log at $ORACLE_HOME/startup.log. If you need to capture stdout and stderr 
of init.cssd in /etc/inittab, change ">/dev/null" to ">/tmp/initcssd.log". If you need to do 
that for dbstart, change in dbora
       su - $ORACLE -c "$ORACLE_HOME/bin/dbstart $ORACLE_HOME"
to
       su - $ORACLE -c "$ORACLE_HOME/bin/dbstart $ORACLE_HOME > /tmp/dbstart.log 2>&1"
If you need to log dbora itself, add 
exec 1> /tmp/dbora.log #redirect stdout to a log file
exec 2>&1              #redirect stderr to it too
to the top of the script.
[End of Update 2009-10]


It's easy to control program startup order on UNIX (or Linux). Even traditionally BSD style UNIX is 
moving toward SYSV startup strategy, using /etc/rcX.d, etc. I only document some special cases here.


* 10gR2 Linux non-RAC using ASM. Database doesn't startup on reboot.

There may be a lot of problems in documentation about this. For instance, Bug 4722280 (DBSTART 
SCRIPT IS NOT CHECKING FOR THE ORATAB 'W' OPTION) talks about a “W” option in the oratab file. We 
verified this by reading our dbstart script that 'W' option is equivalent to anything other than 
'Y'. Documentation at
http://download-east.oracle.com/docs/cd/B19306_01/server.102/b15658/strt_stp.htm#sthref255
is obviously wrong or is ahead of the dbstart script maintainer. The dbora script in documentation 
(scroll down a little) is questionable: (1) When root runs the script as "dbora start", it directly 
runs dbstart, but isn't dbstart supposed to be run by oracle? (2) When root runs it as "dbora start 
ORA_DB", it connects to this same server using remote shell as oracle, but I suspect it's going to 
infinitely loop doing the remote shell connection and running "dbora start ORA_DB" again... I 
didn't test it.

So much for that, and dodgy support (see Service Request 5117088.992 if you can). The solution for 
this is still creating a startup script as usual:
...
case "$1" in
 start)
  su - oracle -c "$ORACLE_HOME/bin/dbstart start"
  su - oracle -c "ORACLE_HOME=/u01/app/oracle/product/10.2.0/asm $ORACLE_HOME/bin/lsnrctl start"
...
and place it somewhere, e.g., /etc/rc.d/init.d/mydbstart. Note that my oracle user's shell uses DB's 
home as $ORACLE_HOME and, although not required, I prefer to start listener from my ASM's home 
(change path as needed, or preferably set a variable up front and reference it here, and down below 
in the script when stopping and checking status of listener).

Then you append a line like this:

db:35:once:/etc/rc.d/init.d/mydbstart start >> /var/log/oracle.log 2>&1

to the very end of /etc/inittab. And of course the entries for ASM and DB in oratab should have the 
Y flag for auto start on boot.

Here's why. Suppose your default runlevel is 3 (check by `grep initdefault /etc/inittab`). You must 
have an Sxx link to startup ASM at /etc/rc3.d (probably named Sxxoracleasm). You also have

h1:35:respawn:/etc/init.d/init.cssd run >/dev/null 2>&1 </dev/null

in /etc/inittab. I believe the correct order is init.cssd, ASM, then DB. Since DB has 
to come after cssd, you can't create a Sxx link in /etc/rc3.d pointing to mydbstart, because 
"/etc/rc.d/rc 3" is still way above init.cssd inside inittab. This means even if you create a link 
S99 to mydbstart (or equivalently put mydbstart in /etc/rc.local), it's still run before init.cssd. 
I wish we could move init.cssd out of inittab into /etc/rc3.d and give a number less than 99. But 
we would lose the capability of init to respawn cssd when it dies. All this pushes us to the only 
remaining solution, run mydbstart after init.cssd inside inittab. The line I showed above says give 
a unique ID to this line, e.g. "db", and when server goes to runlevel 3 or 5, run `mydbstart start` 
once and redirect stdout and stderr to /var/log/oracle.log.

Additional notes:
* This problem may be related to Bug:4074759. It was fixed and re-introduced. 
* According to Note:336299.1, dbstart throws error "ORACLE_HOME_LISTNER is not SET, unable to 
auto-start Oracle Net Listener" (note wrong spelling of LISTNER). Ignore the message, or fix it per 
the note.


* On 10gR2 RAC, DB and ASM don't auto-start on bootup.

The best solution is to make sure `srvctl start database` and `srvctl start asm` work. If not, 
cleanup OCR and re-register. `srvctl remove/add` is hard to use. Use `ocr_unregister/_register` 
instead. See ./RAC_OCR_Cleanup.txt.