Feeds:
Posts
Comments

Archive for the ‘Monitoring Tools’ Category

OpenIPMI is an effort to create a full-function IPMI system to allow full access to all IPMI information on a server and to abstract it to a level that will make it easy to use.

The Open IPMI Linux device driver is designed as a full-function IPMI device driver with the following features:

  • Allows multiple users.
  • Allows multiple interfaces.
  • Allows both kernel and userland things to use the interface.
  • Fully supports the watchdog timer.
  • It works like IPMI drivers are supposed to. It tracks outgoing messages and matches up their responses automatically. It automatically fetches events, received messages, etc.
  • It supports interrupts (I have tested them now).
  • It has backwards-compatability modules for supporting the Radisys IPMI driver and the Intel IMB driver.
  • It’s modular. You don’t have to have the standard userland interface. You don’t have to have the watchdog. Etc.
  • It supports generating an event on a panic.

 

Download OpenIPMI from http://openipmi.sourceforge.net/

Intel source http://www.intel.com/design/servers/ipmi/index.htm

IBM also now supports OpenIPMI drier beginning with the following version of servers.

The OpenIPMI driver ships with these and later operating systems and IBM’s applications will talk directly to the OpenIPMI driver without requiring a mapping layer.These operating systems also ship with an OS Watchdog application, which interfaces to the OpenIPMI driver, so that native watchdog application should be used in place of IBM’s Automatic Server Restart-IPMI application on these and later operating systems.

  • Red Hat Enterprise Linux 3.7
  • Red Hat Enterprise Linux 4.3
  • Red Hat Enterprise Linux 5
  • SUSE Linux Enterprise Server 9.3
  • SUSE Linux Enterprise Server 10
  • VMware ESX Server 2.5.4 (32-bit operating system)
  • VMware ESX Server 3 (32-bit operating system)

IBM Source: http://www-304.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-66668&brandind=5000008

# rpm -qa | grep -i ipmi
OpenIPMI-1.4.19-1.8
OpenIPMI-devel-1.4.19-1.8
ipmitool-1.8.2-0.5

ipmitool download: http://www-304.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5069538&brandind=5000008

http://ipmitool.sourceforge.net

ipmitool – utility for controlling IPMI-enabled devices

You can use IPMItool to configure and manage devices that support the Intelligent Platform Management Interface versions 1.5 and 2.0. IPMItool provides a simple, command-line interface to the Baseboard Management Controller, featuring the ability to read the sensor data repository (SDR) and print sensor values, display the contents of a system event log, print field-replaceable unit inventory information, read and set LAN configuration parameters, and perform remote chassis power control.

To check the Management Controller status and global enables using ipmitool utility

# ipmitool mc info
Device ID                 : 32
Device Revision           : 0
Firmware Revision         : 3.10
IPMI Version              : 2.0
Manufacturer ID           : 2
Product ID                : 251 (0x00e8)
Device Available          : yes
Provides Device SDRs      : no
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    IPMB Event Receiver
    IPMB Event Generator
    Chassis Device
Aux Firmware Rev Info     :
    0x5a
    0x53
    0x42
    0x54

Use sensor option to Print detailed sensor information

# ipmitool sensor
Ambient Temp     | 16.000     | degrees C  | ok    | 64.000    | na        | 64.000    | -128.000  | na        | na
Fan 1 Tach       | 2888.000   | RPM        | ok    | 760.000   | 1976.000  | 1976.000  | na        | na        | na
Fan 2 Tach       | 2888.000   | RPM        | ok    | 760.000   | 1976.000  | 1976.000  | na        | na        | na
Fan 3 Tach       | 2618.000   | RPM        | ok    | 770.000   | 2002.000  | 2002.000  | na        | na        | na
Fan 4 Tach       | 2541.000   | RPM        | ok    | 770.000   | 2002.000  | 2002.000  | na        | na        | na
Fan 5 Tach       | 2888.000   | RPM        | ok    | 760.000   | 1976.000  | 1976.000  | na        | na        | na
Fan 6 Tach       | 2964.000   | RPM        | ok    | 760.000   | 1976.000  | 1976.000  | na        | na        | na
Fan 7 Tach       | 2541.000   | RPM        | ok    | 770.000   | 2002.000  | 2002.000  | na        | na        | na
Fan 8 Tach       | 2618.000   | RPM        | ok    | 770.000   | 2002.000  | 2002.000  | na        | na        | na
CPU 1 Temp       | 31.000     | degrees C  | ok    | na        | na        | na        | 85.000    | na        | 95.000
Fan 1 Presence   | 0x0        | discrete   | 0x0280| na        | na        | na        | na        | na        | na
Fan 2 Presence   | 0x0        | discrete   | 0x0280| na        | na        | na        | na        | na        | na
Fan 3 Presence   | 0x0        | discrete   | 0x0280| na        | na        | na        | na        | na        | na
Fan 4 Presence   | 0x0        | discrete   | 0x0280| na        | na        | na        | na        | na        | na
Fan 5 Presence   | 0x0        | discrete   | 0x0280| na        | na        | na        | na        | na        | na
Fan 6 Presence   | 0x0        | discrete   | 0x0280| na        | na        | na        | na        | na        | na
Fan 7 Presence   | 0x0        | discrete   | 0x0280| na        | na        | na        | na        | na        | na
Fan 8 Presence   | 0x0        | discrete   | 0x0280| na        | na        | na        | na        | na        | na

…. truncated

Some available options for ipmitool util

# ipmitool help
Commands:
        raw           Send a RAW IPMI request and print response
        i2c           Send an I2C Master Write-Read command and print response
        lan           Configure LAN Channels
        chassis       Get chassis status and set power state
        event         Send pre-defined events to MC
        mc            Management Controller status and global enables
        sdr           Print Sensor Data Repository entries and readings
        sensor        Print detailed sensor information
        fru           Print built-in FRU and scan SDR for FRU locators
        sel           Print System Event Log (SEL)
        pef           Configure Platform Event Filtering (PEF)
        sol           Configure IPMIv2.0 Serial-over-LAN
        isol          Configure IPMIv1.5 Serial-over-LAN
        user          Configure Management Controller users
        channel       Configure Management Controller channels
        session       Print session information
        sunoem        OEM Commands for Sun servers
        exec          Run list of commands from file
        set           Set runtime variable for shell and exec

Find the following link to set up the IPMI watchdog Timer , Turning off the Watchdog and additional commands for Watchdog Timer administration

http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/liaai/ipmi/liaaiipmisetupwatchdog.htm&tocNode=toc%3Afront%2Ffront.cmb%2F0%2F2%2F7%2F5%2F5%2F0%2F

Advertisements

Read Full Post »

Just happened on couple of SLES 9 servers after installing the followings 
the server just came back with postfix plugin.

perl-Net-Server-0.87-29.4.i586.rpm
munin-node-1.2.4-1.sles9.rpm

 
But by default munin plugins are
cpu entropy forks if_eth0 iostat memory mysql_slowqueries  open_files processes df exim_mailqueue
if_err_eth0  if_eth1 irqstats mysql_bytes mysql_threads open_inodes
swap df_inode  exim_mailstats if_err_eth1 interrupts load mysql_queries 
netstat postfix_mailvolume vmstat

Then I manually did symbolic links from /etc/munin/plugins to /usr/share/munin/plugins

beastie:/etc/munin/plugins # ln -s /usr/share/munin/plugins/cpu cpu
beastie:/etc/munin/plugins # ln -s /usr/share/munin/plugins/df df
beastie:/etc/munin/plugins # ln -s /usr/share/munin/plugins/df_inode df_inode
beastie:/etc/munin/plugins # ln -s /usr/share/munin/plugins/load load
beastie:/etc/munin/plugins # ln -s /usr/share/munin/plugins/memory memory
beastie:/etc/munin/plugins # ln -s /usr/share/munin/plugins/processes processes
beastie:/etc/munin/plugins # ln -s /usr/share/munin/plugins/if_eth0 if_eth0
beastie:/etc/munin/plugins # ln -s /usr/share/munin/plugins/if_err_eth0 if_err_eth0
beastie:/etc/munin/plugins # ln -s /usr/share/munin/plugins/if_eth1 if_eth1
beastie:/etc/munin/plugins # ln -s /usr/share/munin/plugins/if_err_eth1 if_err_eth1

Now after this the munin-node crashed and failed to show graph even for postfix.

While checking the log for munin in /var/log/munin/munin-node.log, I got the following error message

Can’t exec “/etc/munin/plugins/df”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:09:52 Plugin timeout: df config: Interrupted system call
(pid 16888)
2008/08/07-18:09:52 Server closing!
Can’t exec “/etc/munin/plugins/cpu”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:10:02 Plugin timeout: cpu config: Interrupted system call
(pid 16894)
2008/08/07-18:10:02 Server closing!
Can’t exec “/etc/munin/plugins/load”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:10:12 Plugin timeout: load config: Interrupted system call
(pid 16901)
2008/08/07-18:10:12 Server closing!
Can’t exec “/etc/munin/plugins/swap”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:10:22 Plugin timeout: swap config: Interrupted system call
(pid 16908)
2008/08/07-18:10:22 Server closing!
Can’t exec “/etc/munin/plugins/forks”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:10:32 Plugin timeout: forks config: Interrupted system call
(pid 16911)
2008/08/07-18:10:32 Server closing!
Can’t exec “/etc/munin/plugins/entropy”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:10:42 Plugin timeout: entropy config: Interrupted system call
(pid 16912)
2008/08/07-18:10:42 Server closing!
Can’t exec “/etc/munin/plugins/netstat”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:10:52 Plugin timeout: netstat config: Interrupted system call
(pid 16917)
2008/08/07-18:10:52 Server closing!
Can’t exec “/etc/munin/plugins/iostat”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:11:02 Plugin timeout: iostat config: Interrupted system call
(pid 16986)
Can’t exec “/etc/munin/plugins/df_inode”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:11:14 Plugin timeout: df_inode config: Interrupted system call
(pid 16991)
2008/08/07-18:11:14 Server closing!
Can’t exec “/etc/munin/plugins/memory”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:11:24 Plugin timeout: memory config: Interrupted system call
(pid 16997)
2008/08/07-18:11:24 Server closing!
Can’t exec “/etc/munin/plugins/vmstat”: Permission denied at /usr/sbin/munin-node line 562.
2008/08/07-18:11:34 Plugin timeout: vmstat config: Interrupted system call
(pid 17014)
2008/08/07-18:11:34 Server closing!
2008/08/07-18:14:33 CONNECT TCP Peer: “XX.XX.XX.7:52286” Local: “XX.XX.XX.145:4949”
Can’t exec “/etc/munin/plugins/df_inode”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 2.
Can’t exec “/etc/munin/plugins/entropy”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 3.
Can’t exec “/etc/munin/plugins/cpu”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 4.
Can’t exec “/etc/munin/plugins/postfix_mailvolume”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 5.
Can’t exec “/etc/munin/plugins/postfix_mailqueue”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 6.
Can’t exec “/etc/munin/plugins/forks”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 7.
Can’t exec “/etc/munin/plugins/iostat”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 8.
Can’t exec “/etc/munin/plugins/memory”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 9.
Can’t exec “/etc/munin/plugins/df”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 10.
Can’t exec “/etc/munin/plugins/netstat”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 11.
Can’t exec “/etc/munin/plugins/swap”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 12.
Can’t exec “/etc/munin/plugins/vmstat”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 13.
Can’t exec “/etc/munin/plugins/load”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 14.
Can’t exec “/etc/munin/plugins/cpu”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 15.
Can’t exec “/etc/munin/plugins/entropy”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 16.
Can’t exec “/etc/munin/plugins/df_inode”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 17.
Can’t exec “/etc/munin/plugins/postfix_mailqueue”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 18.
Can’t exec “/etc/munin/plugins/postfix_mailvolume”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 19.
Can’t exec “/etc/munin/plugins/forks”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 20.
Can’t exec “/etc/munin/plugins/iostat”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 21.
Can’t exec “/etc/munin/plugins/memory”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 22.
Can’t exec “/etc/munin/plugins/df”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 23.
Can’t exec “/etc/munin/plugins/netstat”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 24.
Can’t exec “/etc/munin/plugins/swap”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 25.
Can’t exec “/etc/munin/plugins/vmstat”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 26.
Can’t exec “/etc/munin/plugins/load”: Permission denied at /usr/sbin/munin-node line 562, <STDIN> line 27.
2008/08/07-18:14:33 Plugin “swap” exited with status 13. —-
2008/08/07-18:14:33 Plugin “netstat” exited with status 13. —-
2008/08/07-18:14:33 Plugin “df” exited with status 13. —-
2008/08/07-18:14:33 Plugin “memory” exited with status 13. —-
2008/08/07-18:14:33 Plugin “iostat” exited with status 13. —-
2008/08/07-18:14:33 Plugin “forks” exited with status 13. —-
2008/08/07-18:14:33 Plugin “postfix_mailvolume” exited with status 13. —-
2008/08/07-18:14:33 Plugin “postfix_mailqueue” exited with status 13. —-
2008/08/07-18:14:33 Plugin “df_inode” exited with status 13. —-
2008/08/07-18:14:33 Plugin “entropy” exited with status 13. —-
2008/08/07-18:14:33 Plugin “cpu” exited with status 13. —-
2008/08/07-18:14:33 Plugin “load” exited with status 13. —-
2008/08/07-18:14:33 Plugin “vmstat” exited with status 13. —-
2008/08/07-18:14:33 Plugin “swap” exited with status 13. —-
2008/08/07-18:14:33 Plugin “netstat” exited with status 13. —-
2008/08/07-18:14:33 Plugin “df” exited with status 13. —-
2008/08/07-18:14:33 Plugin “memory” exited with status 13. —-
2008/08/07-18:14:33 Plugin “iostat” exited with status 13. –
2008/08/07-18:14:33 Plugin “forks” exited with status 13. —-
2008/08/07-18:14:33 Plugin “postfix_mailqueue” exited with status 13. —-
2008/08/07-18:14:33 Plugin “postfix_mailvolume” exited with status 13. —-
2008/08/07-18:14:33 Plugin “cpu” exited with status 13. —-
2008/08/07-18:14:33 Plugin “entropy” exited with status 13. —-
2008/08/07-18:14:33 Plugin “df_inode” exited with status 13. —-

 

If you ever get these kinds of errors don’t forget to add plugins and permissions in /etc/munin/plugin-conf.d/munin-node file.

$cat /etc/munin/plugin-conf.d/munin-node
#
# This file contains configuration options for the plugins. Three
# options are understood by munin-node itself:
#
#       user <user>         # Set the user to run the plugin as
#       group <group>       # Set the group to run the plugin as
#       command <command>   # Run <command> instead of the plugin. %c
#                             expands to what would normally be run.
#   env.<variable>      # Sets <variable> in the plugin’s environment, see the
#                         individual plugins to find out which variables they
#                         care about.
#
#

[mysql*]
#env.mysqlopts -u someuser

[exim*]
group mail

[cps*]
user root

[apt]
user root

[vlan*]
user root

[postfix*]
user root
env.logfile mail

[cpu*]
user root

[memory*]
user root

[load*]
user root

[if_*]
user root

[if_err_*]
user root

[df*]
user root

[process*]
user root

[dhcpd]
env.leasefile /var/lib/dhcp/db/dhcpd.leases

 

You have to do this only when you get those error messages ūüôā Cheers

Read Full Post »

Ethereal

Ethereal (or the new July 2006 name of Wireshark) is a much easier tool for use than tcpdump (or snoop). However, the GUI tool need to be installed to the machine you run on. It is typically easiest to run tcpdump to capture to a file, then open it with the GUI ethereal running on Linux or Windows.

ethereal (GUI)
tethereal (CLI)

most flags work for both.

snoop-like behaviour (mostly for ethereal):
-l    : scroll capture
-S     : update as capture is in progress.
-k     : start capture immediately  (disable interaction?)

-i [IF] : specify interface, eg eth0, hme0
-n     : no dns resolution, use ip Number

-V     : more verbose output, captured data displayed in tree mode instead of 1 line per packet.

-f     : capture filter expression  (tcpdump notation needed), eg:

>    tcp port 23 and host 10.0.0.5
>   src net 10.0.15.0/24
>   dst net 10.0.15.0 mask 255.255.255.0
>>     [src|dst] host
>>    ether [src|dst] host 00:E0:2B:DE:0E:00
>>     [tcp|udp] [src|dst] port

host 10.215.20.152 || host 10.215.2.21 || host 10.215.19.73

————————————————————

ethereal view filter expression
[ work in GUI filter box when viewing,
NOT as capture filter (which is tcpdump format ]

operatos:
eq, ==    Equal
ne, !=    Not equal
gt, >     Greater than
lt, <     Less Than
ge, >=    Greater than or Equal to
le, <=    Less than or Equal to

and, &&   Logical AND
or, ||    Logical OR
not, !    Logical NOT

boolean: true (1) or false (0)

some commonly used filter fields:

eth.src == aa-aa-aa-aa-aa-aa
ip.dst eq http://www.mit.edu
ip.src == 192.168.1.1
ip.addr == 129.111.0.0/16
eth.src == aa-aa-aa-aa-aa-aa
eth.src[0:3] == 00:00:83            # filter by vendor by use of slide
tcp.port == 80 and ip.src == 192.168.2.1
ip.addr is for both src or dest, these multiple ocurring field is a bit confusing for packet filtering.

for generic filter dealing with a specific host, but not necessary filtering by tcp/udp/icmp
ip.dst
ip.src
ip.addr

udp
udp.port
udp.dstport
udp.srcport

tcp
tcp.port
tcp.dstport
tcp.srcport
tcp.seq

icmp

bootp.dhcp==true        : frame is dhcp
bootp.hw.addr

smb.cmd==(unsigned 8 bit int)    : smb protocol command number
smb.cmd == 0x06          : cmd is smb unlink
smb.status != 0x0000    : Error code, 4 bytes aka status, lot of items.
smb.errcls != 0x0        : error class, 1 byte represent the categories
0x0       = Success
0x1       = DOS Error
0x2       = Server Error
0x3     = hardware error
0x4    = not a smb cmd
Note, netBench Fail code 32 maybe in Dos or Hrd.
smb.pid
smb.mid        (multiplex id)
smb.uid        (user id, maybe per process)
nfs.*
nfs.fh.version != 3        = not sure what this is, not nfs protocol version!
rpc.programversion != 3        = all packet that are rpc program nfs version 3.

lot of higher level protocol stuff available, including vlan on switches, etc.
see the man page on ethereal or tethereal (very long!)

GUI version, filter can just enter a protocol type.  eg: smb
That means smb protocol is present.  A protocol in the filter w/o any comparison operator means filter packets where such field is present in the packet.
eg: smb.errcls  filter packet that contain smb error class.

Network trace capture with tcpdump or snoop, save to file for viewing with ethereal

tcpdump -i [interface] -s 1500 -w [some-file]
tcpdump -s 8192 -w netuse.tcpdump ‘host 10.0.71.232 or host 10.0.71.15’
snoop -d hme0  -o /tmp/snoop.out host 10.215.55.211

editcap can be used to trim captured file, or convert between formats
(tcpdump, ethereal, snoop, ms netmon, etc).

Good read on ethereal: http://www.ns.aus.com/ethereal/user-guide/ch03capfilt.html

Read Full Post »

Snoop

snoop host [IP]            # traffic with a given host (as src or dst)
snoop -r port 25        # all traffic in port 25 (smtp),
# do not resolve ip to dns names
-s     = sniplet length (def is whole packet)
= 80 ip hdr only, 120 = nfs header only

-V    = layer info
-v    = more verbose than -V, lot of info.

from cli :
Usage:  snoop
[ -a ]                  # Listen to packets on audio
[ -d device ]           # settable to le?, ie?, bf?, tr?
[ -s snaplen ]          # Truncate packets
[ -c count ]            # Quit after count packets
[ -P ]                  # Turn OFF promiscuous mode
[ -D ]                  # Report dropped packets
[ -S ]                  # Report packet size
[ -i file ]             # Read previously captured packets
[ -o file ]             # Capture packets in file
[ -n file ]             # Load addr-to-name table from file
[ -N ]                  # Create addr-to-name table
[ -t  r|a|d ]           # Time: Relative, Absolute or Delta
[ -v ]                  # Verbose packet display
[ -V ]                  # Show all summary lines
[ -p first[,last] ]     # Select packet(s) to display
[ -x offset[,length] ]  # Hex dump from offset for length
[ -C ]                  # Print packet filter code

Capture traffic on NIC hme0 specific to a host, capture up 8K of the packet,
and dump result to an output file:
snoop -d hme0 -s 8192 -o /tmp/snoop.out host 10.215.55.211

Read input file back.  May wish to use ethereal to read this file for easier access.
snoop -i /tmp/snoop.out

snoop -s 120 port 25 host 211.196.53.194

titaniumleg.com  mail server traffic monitor
snoop -r -D -P -s 1500 -c 100000 -o /export/tmp/smtp01.20030122.snoop port 25

snoop -n /dev/null  -D -P -s 1500 -c 100000 -o /export/tmp/smtp01.20030122.snoop port 25
snoop -D -s 9000 -c 100000 -o jumpstartclient.snoop host jumpstartclient
-r = do not resolve hostname  # not in sol 7 snoop
-D = display num of dropped packets
-P = non promiscuous mode capture¬†¬† (don’t use in troubleshooting jumpstart problems).
-s snipplet length
-c count num of backets to capture
-o output file

Read Full Post »

Tool to collect performance stat, very similar in HP-UX, Sun, AIX…
Also check out kSar ( http://ksar.atomique.net/) and sar2rrd ( http://www.trickytools.com/php/sar2rrd.php )

Basic SAR Setup (from HP-UX sys admin handbook and tooltips, p503):

sar -o /tmp/sar.data 60 300         # run sar every 60 sec for 300 count,
-o store info in file (bin)

sar -u -f /var/adm/sa/saXX        # read data from file (Solaris, XX = date number)
sar -u -f /tmp/sar.data            # read data from file (HP-UX)
-u display cpu info (similar to iostat and vmstat)
-b buffer cache activity, imp for oracle
-d disk activity
-q avg queue length (if run queue > num of cpu, will have to wait).
-w swap info

Solaris starts sadc in /etc/rc2.d/S21perf , a deamon to collect sar info.

————

Setup SAR data collection for HP-UX (should also work for other platform):

http://www.sarcheck.com/sarhowto.htm    (Actually SarCheck.com, but cost money!)

mkdir  /var/adm/sa,
then setup root crontab:

#collect sar data      # every 20 min 8-5, hourly outside normal work
0 * * * * /usr/lbin/sa/sa1
20,40 8-17 * * 1-5 /usr/lbin/sa/sa1
#reduce the sar data    # generate pre-formated report focus for business hrs
5 18 * * * /usr/lbin/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A

# sample for SF + Minsk work hours
0 * * * * /usr/lbin/sa/sa1
15,30,45 0-8,10-19,23 * * 1-5 /usr/lbin/sa/sa1
05 21 * * * /usr/lbin/sa/sa2 -i 3600 -A

# sa1 is data collection to /var/adm/sa/saXX
# sa2 really produce condense version of report to /var/adm/sa/sarXX (sar vs sa)

# filenames are reused every month.
# use sar -A -f /var/adm/sa/saXX to get more detail report than std summary.

————-
AIX has preset entries in crontab for ‘adm’.¬† Check to ensure script exsit.
sar logs are stored in /var/adm/sa
————-

Minsk is PT + 10 hours (ahead).

SF  vs  Minsk work hours
—- ¬†¬†¬† ——
8am    6pm
10am    8pm
6pm    4am

10pm     8am
midnite    10am
8am     6pm

Read Full Post »

netstat -ta     show current intenet services/connections
        -a  : show (a)ll  (include listening port process)
        -n  : ip (n)umber only (no dns lookup)
        -r  : (r)outing table   (change with route cmd)
        -i  : show stat for diff nic (i)nterfaces
        -k ce0 : lot of interface specific info, ce NIC will have duplex stat.

netstat -p : print ip to mac address table known to host
netstat -k : print lot of kernel stat, among it hme0 is for the sun’s build in
   happy meal ethernet nic (see sunsolve infodoc 17416 for explanation of these
  undocumented stats, good for trobleshooting network latency, comare agaist cisco stat.
netstat -s : show high level packet send/receive/fragment info

vmstat -a   : all
       -n   :
       -p   : process owning port

iostat -xn 30 
 : check for disk activity, anything more than 5% busy and avg resp time > 30 ms is bad.

nfsstat

mpstat 10 
: processor stats, repeat every 10 seconds
: In Solaris, it reports context switch, interrupt, mutex spin, xcal, etc
: see http://sunsite.uakom.sk/sunworldonline/swol-08-1998/swol-08-perf.html

cpustat¬†¬†¬†¬† : find out what cpu is doing…

lockstat sleep 5 
: gather kernel lock stats during the sleep period (5 sec)
: solaris, run as root

truss -c -p PID : find number of system call and usr time for a process (sol).  ??

top
protocol
sysmon

trapstat, thread list, kmastat, kmausers

GUDS

Guds is a script to gather performance stats for Solaris.
Sample usage is
./guds_2.4.5
./guds_2.4.5 -qX -H3 -s65040465

-qX is for quite mode
-H3 is for running it for 3 hours
-sNNNNN is the sun case number (info embeded in dirs created by guds to
store the files).

It collects lots of info in /var/tmp/CASEID/guds-DATE-TIME/…

May need lot of know how to analyze data.
Having a baseline when things is good and when there are
performance problems would help.
date; mkfile 1000m test; date     
 # create a 1 GB file (filled with 0)
date; dd if=/dev/urandom of=test bs=1024 count=100000  
# same, file has random data.

Read Full Post »