[Update 2015-07]
Instead of using my svc_on_pref.sh script, which "Check(s) to see if services are running on their preferred instances",
Chad Cleveland's SQLs to query Oracle Enterprise Manager respository can also do this, if you use EM to monitor your database.
See http://www.freelists.org/post/oracle-l/11g-RAC-TAPICustom-services-how-to-query-in-data-dictionary,1
But the SQL can improved as follows:
col database_unique_name for a10
col cluster_name for a20
col service_name for a30
col preferred_instances for a35
col running_instances for a35
select database_unique_name, cluster_name, service_name, preferred_instances, running_instances
from sysman.mgmt_rac_services
where preferred_instances != running_instances
order by 1, 2, 3, 4;

[Update 2011-05]
In 11gR2, create service with srvctl only (or the failover and load balance attributes added by dbms_service would be wiped out if DB is bounced):
srvctl add service -d dbname -s servicename -r preferredinstance1,preferredinstance2 -m BASIC -e SESSION -j SHORT -B SERVICE_TIME
srvctl start service -d dbname
Before 11gR2, create it with srvctl followed by dbms_service.modify_service inside database. See below.



#Create it in OCR first
#Make sure no instance appears in both preferred and available instance lists (10g clusterware bug 8256800)
#or it would only be in available list and stopping it would crash instance or cause other issues (Note:783703.1)
srvctl add service -d dbname -s servicename -r preferredinstance1,preferredinstance2 -a backupinstance1,backupinstance2
srvctl start service -d dbname -s servicename
srvctl status service -d dbname -s servicename

--Make it failover-able
exec dbms_service.modify_service('servicename', dbms_service.goal_service_time, aq_ha_notifications=>true, failover_method=>'BASIC', failover_type=>'SESSION', failover_retries=>3, failover_delay=>5)
select * from dba_services;

--Make it usable
exec dbms_service.start_service('servicename', dbms_service.all_instances) --not needed if `srvctl start' already started it
select * from v$active_services;

----------------------------------------------------------------------------------------------------

Service does not register with SCAN listener: see "Service does not register" at ../SingleClientAccessName.txt

----------------------------------------------------------------------------------------------------

If you get into a situation add service says it already exists but remove service says it doesn't, try

srvctl remove service -d dbname -s servicename
srvctl remove service -d dbname -s servicename -f
Yes, the first one must be run; sometimes directly running the second will NOT remove service.
If that does not remove, then
crs_unregister ora.dbname.servicename.instance1.srv
crs_unregister ora.dbname.servicename.instance2.srv

Ref: http://apunhiran.blogspot.com/2008/08/prkp-1029-crs-0211-on-srvctl-modifyadd.html

----------------------------------------------------------------------------------------------------

#!/usr/bin/perl -w
#svc_on_pref.sh: Check to see if services are running on their preferred instances. Notify us if not.
#Tested on 11.2.0.3

$DB = "ORCL";
$RECIPIENTEMAIL = 'yong321,fellowdba';

$ENV{PATH}="/usr/bin:/bin:/u01/app/11.2.0/grid/bin";

$_=`crs_stat | awk -F. '/^NAME.*\.svc\$/ {print \$3}' | sort | uniq`; #Get all service names (domain stripped)
@line = split /\n/;
foreach (@line)
{
  $service = "$_.mycompany.com"; #Our domain names are guaranteed to be this.
  $prefinst = `srvctl config service -d $DB -s $service | awk -F": " '/^Preferred/{print \$2}'`;
  chomp($prefinst);     #there's a newline at the end
  $statusline = `srvctl status service -d $DB -s $service`;
  if ($statusline =~ /is running on instance\(s\) (.*)$/)
  { $runinst = $1;
    if ($prefinst ne $runinst)
    { $msg = "Service $service preferred instance list differs from service running instance list:
Preferred: $prefinst
Running on: $runinst\n";
      print $msg;
      system "mail -s 'Service not running on all preferred instances' $RECIPIENTEMAIL <<EOF
$msg
EOF";
    }
    else
    { print "Service $service preferred instance list equals service running instance list:
Preferred: $prefinst
Running on: $runinst\n";
    }
  }
  else #This service is not even running.
  { print $statusline;
    system "mail -s 'Service not running' $RECIPIENTEMAIL <<EOF
$statusline
EOF";
  }
  @prefinst = ();
  @runinst = ();
}

----------------------------------------------------------------------------------------------------

#!/usr/bin/perl -w
#svc_on_pref.sh: Check to see if services are running on their preferred instances. Notify us if not.
#Tested on 10gR2 and 11gR2

$DB = "ORCL";
$RECIPIENTEMAIL = 'yong321,fellowdba';

$ENV{PATH}="/usr/bin:/bin:/u01/crs/oracle/product/11.2.0/crs/bin";

$_=`crs_stat | awk -F. '/^NAME.*\.srv\$/ {print \$3}' | sort | uniq`; #Get all service names (domain stripped)
@line = split /\n/;
foreach (@line)
{
 $service = "$_.mycompany.com"; #Our domain names are guaranteed to be this.
 `srvctl config service -d $DB -s $service` =~ /^$service PREF: (.*) AVAIL:/;
 $prefinst = $1; $prefinst =~ s/ /, /g; #Just change to the same field delimiter as in $runinst string
       #I decide not to use arrays, assuming string comparison suffices.
       #@prefinst = split / /,$1; #preferred instance list (assume it's always defined)
       #foreach(@prefinst) { print "Our service $service preferred on $_.\n"; } #debug
 $statusline = `srvctl status service -d $DB -s $service`;
 if ($statusline =~ /is running on instance\(s\) (.*)$/)
 { $runinst = $1;
       #@runinst = split /, /,$1;
       #foreach(@runinst) { print "Our service $service running on $_.\n"; } #debug
   if ($prefinst ne $runinst)
   { $msg = "Service $service preferred instance list differs from service running instance list:
Preferred: $prefinst
Running on: $runinst\n";
     print $msg;
     system "mail -s 'Service not running on all preferred instances' $RECIPIENTEMAIL <<EOF
$msg
EOF";
   }
 }
 else #This service is not even running.
 { print $statusline;
   system "mail -s 'Service not running' $RECIPIENTEMAIL <<EOF
$statusline
EOF";
 }
 @prefinst = ();
 @runinst = ();
}

----------------------------------------------------------------------------------------------------

#!/bin/bash
#ck_lsnr_svc_upd.sh: 11g only (10g script is quite different). Check listener service update making 
#sure pmon's on all instances update their info with all SCAN listeners.
#NormalServiceList can be initialled created by `cp ActualServiceList NormalServiceList' after 
#running this script once; it should not be altered in any way (chmod 444 or 400). It stores names 
#of the instance "services" that register with the listeners when everything is running normally 
#(excluding +ASMx and PLSExtProc instances).
#20100813: 11g listener service should check the scan listener; regular listener no longer has all 
#instances' registered. Also, because of complexity of syntax, check it only on local node. So 
#schedule to run this job on all nodes of RAC

cd /u01/app/oracle/scripts/ck_lsnr_svc_upd
export ORACLE_HOME=/u01/app/grid
export PATH=/bin:/usr/bin:$ORACLE_HOME/bin
RECIPIENTS="you@example.com"

function process_diff
{ SCANLSNR=$1
  diff ActualServiceList NormalServiceList > diff.out
  #If ActualServiceList differs from NormalServiceList, alert us. Email content is diff output.
  [[ -s diff.out ]] && mail -s "Some services did not report to $(hostname -s) listener $SCANLSNR. 'diff ActualServiceList NormalServiceList' follows" $RECIPIENTS < diff.out
}

#11g needs lsnrctl service LISTENER_SCAN{number}, not just lsnrctl service
for i in $(ps -ef | grep [S]CAN | perl -nle 'print $& if /LISTENER_SCAN\d/'); do
 if [[ $i ]]; then
  lsnrctl service $i | awk '/^Service "/{print $2}' | egrep -v -- '+ASM|PLSExtProc' | sort | uniq > ActualServiceList
  process_diff $i
 fi
done

----------------------------------------------------------------------------------------------------

Not sure if this claim is correct:

http://www.itpub.net/thread-1243195-2-1.html

ToddBao:
goal=>none, LBA checks increase in v$servicemetric.CPUPERCALL
goal=>SERVICE_TIME, LBA checks increase in v$serviemetric.DBTIMEPERSEC/v$serviemetric.CALLSPERSEC
goal=>THROUGHPUT, LBA checks increase in v$serviemetric.DBTIMEPERCALL

Suppose the service has goal set to SERVICE_TIME, we should see dbtimepersec/callspersec to be in the same trend as goodness:
select inst_id, begin_time, elapsedpercall, cpupercall, dbtimepercall, callspersec, dbtimepersec, dbtimepersec/callspersec, goodness, delta
from gv$servicemetric
where (callspersec != 0 or dbtimepersec != 0) and service_name = '&servicename' and flags = 0
order by goodness;

But my test in 10.2.0.4 RAC doesn't seem to show that. Semantically, DBTIMEPERSEC/CALLSPERSEC is the same as DBTIMEPERCALL anyway. And the result of the query proves that.

I wish we could find the algorithm for "internally computed" goodness.