Oracle Binary ($ORACLE_HOME/bin/oracle) Group Ownership

DBAs normally don't worry about the oracle binary's group ownership. Either oinstall or dba is fine. But beginning with a certain version (maybe 12c), on servers using ASM for storage, Oracle starts to enforce some role separation even against your willingness, and the group ownership now matters. If the group ownership is messed up, launching certain tools, notably DBCA, DBUA (for upgrade) and "srvctl start", will change the group ownership of DB_HOME/bin/oracle, *with absolutely no regard to* a running instance; an instance that is running this oracle binary will hang, with this error in alert.log:

ORA-27140: attach to post/wait facility failed
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: Operation not permitted
ORA-27302: failure occurred at: skgpwinit6
ORA-27303: additional information: startup egid = 1001 (oinstall), current egid = 1002 (dba)

The clean solution is to bounce the instance, or instances if multiple. But as a temporary workaround in emergency, you can manually run `chgrp' to change the group back immediately, e.g.:
chgrp 1001 oracle #"startup egid" in alert.log, or if not in alert.log yet, run `grep ^Gid /proc/<pid>/status' where <pid> is pid of a process such as pmon
chmod 6751 oracle #chgrp will remove setgid and setuid bits, so put them back on with this command, or `chmod u+s oracle; chmod g+s oracle'
ls -l oracle #should show mode -rwsr-s--x
If you apply the above emergency workaround, remember to change group ownership to the correct one (1002 in the example, or the result of the first command shown below) when you have a chance to shut down the instance (instances) in the future.

I have developed two commands to check for this vulnerability on databases that use ASM as storage:

grep "define SS_ASM_GRP" $(awk -F: '/^+ASM/{print $2}' /etc/oratab)/rdbms/lib/config.c | cut -d" " -f3
for i in $(egrep -v '^#|+ASM|+APX|-MGMTDB' /etc/oratab | awk -F: '/:/{print $2}' | sort | uniq); do ls -l $i/bin/oracle; done

The first command checks the defined group for GI (i.e. GRID) ownership. The second command checks the group ownership of all the DB_Home/bin/oracle binaries. The rule is that the first and second checks must match. Otherwise, when you next run DBCA, DBUA, or `srvctl start' database or instance (but not stop or any other action of srvctl), the tool will change the DB oracle binary's group to match the GI group. You can grep setasmgidwrap in the tool's log files in e.g.
$ORACLE_BASE/cfgtoollogs/dbca/<dbname>/trace.log*
$ORACLE_BASE/cfgtoollogs/dbua/upgrade<time>/trace.log*
$ORACLE_BASE/diag/crs/`hostname -s`/crs/trace/crsd_oraagent_oracle.trc or
/u01/app/grid/log/`hostname -s`/agent/crsd/oraagent_oracle/oraagent_oracle.log
to verify. The consequence will be observed only if an instance is currently running from the affected DB Home; if all the instances from that Home are down, no harm is done.

Here's a check on one server:

$ grep "define SS_ASM_GRP" $(awk -F: '/^+ASM/{print $2}' /etc/oratab)/rdbms/lib/config.c | cut -d" " -f3
"dba"
$ for i in $(egrep -v '^#|+ASM|+APX|-MGMTDB' /etc/oratab | awk -F: '/:/{print $2}' | sort | uniq); do ls -l $i/bin/oracle; done
-rwsr-s--x 1 oracle dba 240272328 Sep 15 10:34 /u01/app/oracle/product/11.2.0.4/dbhome_1/bin/oracle
-rwsr-s--x 1 oracle dba 328151990 Jul 11 11:25 /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle
-rwsr-s--x 1 oracle dba 436693766 Jun 29 16:21 /u01/app/oracle/product/18.2.0.0/dbhome_1/bin/oracle

On this server, I have 3 DB Homes and all have the correct group, dba. (In fact, two of them used to be oinstall and we recently got the egid mismatch error when one of the setasmgid-"contaminated" tools ran, changing the oracle binary group and froze the instances.)

Here's another server:

$ grep "define SS_ASM_GRP" $(awk -F: '/^+ASM/{print $2}' /etc/oratab)/rdbms/lib/config.c | cut -d" " -f3
"sudoers"
$ for i in $(egrep -v '^#|+ASM|+APX|-MGMTDB' /etc/oratab | awk -F: '/:/{print $2}' | sort | uniq); do ls -l $i/bin/oracle; done
-rwsr-s--x. 1 oracle oinstall 239674040 Aug  1 20:55 /u01/app/oracle/product/11.2.0/db/bin/oracle

This server shows a possible problem waiting to surface (saying "possible" because I'm not positive if 11g has this problem). It's funny that SS_ASM_GRP was set to sudoers, one of the supplementary groups user oracle belongs to. (If I could reinstall, I would choose dba, because sudoers looks funny.) The important thing is the mismatch between this group and the DB oracle binary's group, oinstall.

The "good" news is that this error only happens once for the DB Home whose oracle binary's group does not match GI's group. Once it's "corrected", it won't happen again. The bad news is that there's no documented switch or option to disable this behavior, even if you don't intend to separate GI and DBA roles. (How many shops in the world do, anyway?) The undocumented way is to rename GI_Home/bin/setasmgidwrap or the actual Linux executable setasmgid to something else (and you'll see error "setasmgidwrap not found" in oraagent log or trace). Obviously, this hack won't be supported.

Note that it's not that either dba or oinstall alone as group for DB_Home/bin/oracle is correct or wrong, or even better or not as good. The key is that this group must match the group designated by GI, which is recorded during GI installation as SS_ASM_GRP in GI_Home/rdbms/lib/config.c. Either dba, or oinstall, or something else (sudoers for instance on some databases I installed) is fine, as long as the SS_ASM_GRP string says the same. Of course, if the database does not use ASM therefore there's no GI, then there's no such problem at all. Also note that GI group ownership is not the same as ASM disks' group ownership (as shown in `ls -l /dev/...' i.e. the disks shown in v$asm_disk.path, optionally pointed to by symbolic links), which should always be dba, at least on non-Exadata servers.

To prevent this problem from happening again, it's important to choose the right group in GI installation, specifically, on the screen to choose "Oracle ASM Administrator (OSASM) Group". It's better to choose "dba" here, because later when you install Oracle database software, on the screen "Privileged Operating System groups", the default is "dba". The screen to choose groups during GI installation sorts groups in gid order so "dba" may not be at the top (i.e. does not appear to be the "default") and could easily be missed.

2018-10,2021-01

Relevant documents:
Starting the database using srvctl changes the group setting for oracle binary (Doc ID 1508027.1)
SRVCTL Fails to Start Instance with ORA-17503 ORA-27303 But sqlplus Startup is Fine (Doc ID 1322959.1)
After Dbca Database Creation, Other Database Terminated With ORA-27300/ORA-27301/ORA-27302/ORA-27303 (Doc ID 2406287.1)
ORA-600 [spstp: ORACLE_HOME uid does not match euid] When Changing Permissions On $ORACLE_HOME/bin/oracle (Doc ID 747456.1)
dbua changing ownership of ORACLE_HOME/bin/oracle (community.oracle.com/thread/3250328)
SR 3-18509050281