Run a command on many hosts

From time to time, you want to run a simple command on hundreds of servers, such as `ps -ef|grep abcd' to see if a process named abcd is running on them, or `cat /etc/redhat-release' to see Red Hat Linux version. Without manually logging into each of them, how do you do it? My scripts automate the whole process that would otherwise be extremely tedious.


* Prepare ck_host.hostlist file. Each host is one line. Commented line will be ignored.
host1
host2.example.com
#host3.example.com #this host will be skipped


* Assume the Expect interpreter is installed. (`which expect' should show the path, probably /usr/bin/expect. If not installed, install it with `yum install expect' or `apt-get ...'.) Create the Expect script, ck_host.exp, as follows:

$ cat ck_host.exp
#!/usr/bin/expect
#This Expect script is run by ck_host.sh. It takes two arguments, host to connect to (through ssh) and password of the same user as you or `whoami'. It connects to the host and runs the command you specify below. Please limit the command to read-only type just to be safe, e.g. ls -l somefile, cat /etc/redhat-release, /usr/sbin/ntpq -p, ps -ef etc.

set password [lindex $argv 1]   ;#Do not change this line; must be here if sudo will be run

#Please change the command inside the braces; need full path unless cmd is in /usr/bin
#Comment must be on its own line, or preceded by ;# (not just #)
#set cmd {cat /etc/redhat-release}
#set cmd {ps -ef|grep pmon|grep -v grep}        ;#Check which Oracle instances are running
#set cmd {grep -v ^# /etc/oratab|grep -v ^$}    ;#Check which DBs are installed
#set cmd {df -m}
#set cmd {who -b}                               ;#Check server bootup time
#set cmd {grep ccsscan /etc/passwd}             ;#Check if a given user exists
set cmd {grep ^SELINUX= /etc/selinux/config}
#How to use env var on remote host; this example checks which Oracle SID is on that host (default if multiple)
#set cmd {. /home/oracle/.bash_profile; echo $ORACLE_SID}
#How to run a command as an argument of sudo; for this to work, you must have connected to the host
#before, so ~/.ssh/known_hosts will have the hostname, plus something I don't know yet!!!
#set cmd {echo $password | sudo -S /sbin/multipath -l}  ;#Check multipaths
#set cmd {rpm -q kernel |tail -n1|sed -e 's/kernel-//';rpm -qf /boot/vmlinuz-`uname -r` |tail -n1|sed -e 's/kernel-//'}

#Normally you don't need to change anything below
set host [lindex $argv 0]
puts "$host --------------------------------------------------"
spawn /usr/bin/ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $host $cmd
set timeout 10
expect_before eof { exit 2 }    ;#basic network connection fails
expect -re "assword:"
set timeout 2                   ;#give more time for initial connection
send "$password\r"
interact


* Lastly, create the shell script you'll actually run, ck_host.sh, which calls ck_host.exp, in which you have specified the Linux command to be run on each host. Suppose you launch ck_host.sh as oracle on this host. It will login to each of the remote hosts in ck_host.hostlist as oracle with the same password. Please change the password in here, MyPassword123\$, shown below. Remember to escape $ if it's part of the password. The bash read has to work on a non-default file descriptor (9 here) to avoid interference with Expect.

$ cat ck_host.sh
#!/bin/bash
#ck_host.sh: Run a command on remote hosts automatically. The hosts are in file passed as argument on command line, or ck_host.hostlist if not passed.
#Usage: ck_host.sh [hostlistfile]
#Each line in hostlistfile is one host. Commented lines are ignored.
#The actual command to run on remote hosts is set in ck_host.exp. Please modify that file.
#WARNING: Make sure this file is only readable by you: chmod 700 ck_host.sh

HL=$1 #hostlist from cmdline
HL=${HL:=ck_host.hostlist} #if hostlist is not passed, use ck_host.hostlist

while read -r -u9 hst; do
  [[ $hst =~ ^# ]] && continue
  ./ck_host.exp $hst MyPassword123\$ | sed '/^spawn/,/assword: /d'
done 9< $HL

Note that you can pass a different host list file of any name as an argument to this shell script. If not, it reads ck_host.hostlist.