SystemTap to Probe Oracle Process SystemTap is commonly used to probe or trace kernel space system or function calls. Excellent examples are at https://sourceware.org/systemtap/SystemTap_Beginners_Guide/useful-systemtap-scripts.html Beginning with certain Linux kernel level, the uprobes kernel module allows SystemTap to probe user space processes. Newer kernel has uprobes module instead. Make sure you have one of the modules configured: grep CONFIG_UTRACE /boot/config-`uname -r` grep CONFIG_UPROBES /boot/config-`uname -r` Other people have used SystemTap to probe Oracle user processes. See examples at https://sourceware.org/systemtap/wiki/HomePage Find (press ^F) "oracle" in browser to see them. But they all require that you already know the specific function to probe, such as kslwtbctx that signifies the beginning of a wait event. Here, I'll use the para-callgraph.stp script from https://www.sourceware.org/systemtap/SystemTap_Beginners_Guide/paracallgraph.html to probe Oracle server processes with no prior knowledge of which function to probe; I just want to see what function is called. Since I don't intend to pass the 2nd optional argument (used as trigger), I simplify the para-callgraph.stp script by removing the conditional compilation lines as follows (I renamed the filename to cg.stp to save typing): # cat cg.stp #! /usr/bin/env stap function trace(entry_p, extra) { printf("%s%s%s %s\n", thread_indent (entry_p), (entry_p>0?"->":"<-"), ppfunc (), extra) } probe $1.call { trace(1, $$parms) } probe $1.return { trace(-1, $$return) } Let's say I want to probe all Oracle server processes calling functions named like kks*: # stap cg.stp 'process("/u01/app/oracle/product/11.2.0/db/bin/oracle").function("kks*")' 0 oracle(5588):->kksCompileCallNotifier 17 oracle(5588):<-kksCompileCallNotifier 0 oracle(5860):->kksMapCursor 16 oracle(5860):<-kksMapCursor 0 oracle(5860):->kksMapCursor 5 oracle(5860):<-kksMapCursor ^C Two processes were captured calling kksCompileCallNotifier and kksMapCursor functions, respectively. We can limit the probe to a specific process with -x option: # stap cg.stp 'process("/u01/app/oracle/product/12c/db/bin/oracle").function("kks*")' -x 62148 0 oracle_62148_sa(62148):->kksMapCursor 43 oracle_62148_sa(62148): ->kksParentCursor 64 oracle_62148_sa(62148): <-kksParentCursor 71 oracle_62148_sa(62148): ->kksParentCursor 81 oracle_62148_sa(62148): <-kksParentCursor 90 oracle_62148_sa(62148):<-kksMapCursor 0 oracle_62148_sa(62148):->kksCloseCursor 19 oracle_62148_sa(62148): ->kksGetStats 36 oracle_62148_sa(62148): <-kksGetStats 47 oracle_62148_sa(62148): ->kksdip 64 oracle_62148_sa(62148): <-kksdip 89 oracle_62148_sa(62148): ->kksIsVPDCursorSessionCacheable 106 oracle_62148_sa(62148): <-kksIsVPDCursorSessionCacheable 121 oracle_62148_sa(62148):<-kksCloseCursor 0 oracle_62148_sa(62148):->kksMapCursor ^C One issue is that because the oracle binary has tens of thousands of global and local functions (check with `nm $ORACLE_HOME/bin/oracle | egrep ' [Tt] ' | wc -l), if you simply pass * as the argument to function, stap hogs the CPU for a long time to enumerate the functions and no probe will be done. You have to limit the functions to a small set. My test shows that even trying to probe kk*, not alone k*, takes stap too long to prepare and I have to abort. Focusing on function main can be a start: stap cg.stp 'process("/u01/app/oracle/product/11.2.0/db/bin/oracle").function("main")' If you probe other binaries, such as sqlplus or tnslsnr or lsnrctl, passing * is fine. They don't contain too many functions. Due to this issue, I didn't quite achieve the goal of "want to see what function is called", because I can't pass a simple * and have to qualify it to some extent. Another issue is that $$parms and $$return, the two so-called pretty printing target variables, are always null for me.[note1] If they worked, they would show function parameters and function return value, respectively. See https://sourceware.org/systemtap/SystemTap_Beginners_Guide/targetvariables.html#targetprettyprinting I was able to use ulong_arg or s64_arg function to get a specific argument (e.g. the first one) in number format, e.g. function trace(entry_p, extra) { printf("%s%s%s %ld\n", thread_indent (entry_p), (entry_p>0?"->":"<-"), ppfunc (), extra) } function trace2(entry_p) { printf("%s%s%s\n", thread_indent (entry_p), (entry_p>0?"->":"<-"), ppfunc ()) } //probe $1.call { trace(1, ulong_arg(1)) } probe $1.call { trace(1, s64_arg(1)) } probe $1.return { trace2(-1) } I have to omit the second argument of the trace function used by return so I changed it to trace2. The result is 0 oracle(5588):->kksCompileCallNotifier 2 10 oracle(5588):<-kksCompileCallNotifier 0 oracle(5860):->kksMapCursor 139787894174976 9 oracle(5860):<-kksMapCursor ^C _______________________ [note1] This may be due to GCC bug 51358. See http://stackoverflow.com/questions/20770199/how-to-inspect-the-variables-of-user-space-functions-in-systemtap March 2016