Solaris Builtin Tools

netstat -ta     show current intenet services/connections
        -a  : show (a)ll  (include listening port process)
        -n  : ip (n)umber only (no dns lookup)
        -r  : (r)outing table   (change with route cmd)
        -i  : show stat for diff nic (i)nterfaces
        -k ce0 : lot of interface specific info, ce NIC will have duplex stat.

netstat -p : print ip to mac address table known to host
netstat -k : print lot of kernel stat, among it hme0 is for the sun’s build in
   happy meal ethernet nic (see sunsolve infodoc 17416 for explanation of these
  undocumented stats, good for trobleshooting network latency, comare agaist cisco stat.
netstat -s : show high level packet send/receive/fragment info

vmstat -a   : all
       -n   :
       -p   : process owning port

iostat -xn 30 
 : check for disk activity, anything more than 5% busy and avg resp time > 30 ms is bad.

nfsstat

mpstat 10 
: processor stats, repeat every 10 seconds
: In Solaris, it reports context switch, interrupt, mutex spin, xcal, etc
: see http://sunsite.uakom.sk/sunworldonline/swol-08-1998/swol-08-perf.html

cpustat     : find out what cpu is doing…

lockstat sleep 5 
: gather kernel lock stats during the sleep period (5 sec)
: solaris, run as root

truss -c -p PID : find number of system call and usr time for a process (sol).  ??

top
protocol
sysmon

trapstat, thread list, kmastat, kmausers

GUDS

Guds is a script to gather performance stats for Solaris.
Sample usage is
./guds_2.4.5
./guds_2.4.5 -qX -H3 -s65040465

-qX is for quite mode
-H3 is for running it for 3 hours
-sNNNNN is the sun case number (info embeded in dirs created by guds to
store the files).

It collects lots of info in /var/tmp/CASEID/guds-DATE-TIME/…

May need lot of know how to analyze data.
Having a baseline when things is good and when there are
performance problems would help.
date; mkfile 1000m test; date     
 # create a 1 GB file (filled with 0)
date; dd if=/dev/urandom of=test bs=1024 count=100000  
# same, file has random data.

Read Full Post »

VCS config files

/etc/VRTSvcs/conf/config/main.cf
config file for vcs, usually changed using hagui or haclus command. 
Once cluster is live, config is in memory, and this file is only backup. 
Changes to it will be ignored if cluster is up. Cluster start does read this file, 
so easy manual chage of config if cluster is down.
/etc/init.d/vx*
vxvm-relocover
 starts several deamon, which also take argument and email root at local machines.
 change these!
3 files in /etc need to be copied to each of the node in the cluster
(rsh of install should create these if doing multinode install w/ install script).
/etc/llttab ::
set-node oaprod1 # diff for each node, reflect local node name
set-cluster 1
link ce1 /dev/ce:1 – ether – -
link ce3 /dev/ce:3 – ether – -
link-lowpri ce0 /dev/ce:0 – ether – - 

/etc/llthosts ::
0 oaprod1
1 oaprod2 

/etc/gabtab ::
/sbin/gabconfig -c -n2

Read Full Post »

Veritas Cluster Config

gabconfig -a : display link config info.  a = ??gab port, ie loaded ok by kernel.    
b = iofencing port.  h = cluster port. 

GAB Port Memberships
===============================================================
Port a gen   1ea001 membership 01
Port b gen   1ea00f membership 01
Port h gen   1ea012 membership 01
vxfenadm -i  /dev/rdsk/emcpower0c
  Display serial number of the device (LUN, disk)
vxfenadm -g  /dev/rdsk/emcpower0c
  Show IO Fencing info
graceful shutdown of cluster
hastop -all
# stop vcs for the whole cluster, ready for both machine to shutdown.
hastop -local
# stop vcs on local machine only, it will stop the services, no migration by default.
hastop -local -evacuate 
# stop vcs, migrate (evacuate) service to another node
# evacuate a single node, just single node clean exit out of cluster.
hastatus # monitor cluster status, no arg act like tail -f
  -sum # display summary and exit.

lltstat  # general summary
 -nvv # see cluster interconnect link info (heartbeat).

hares -online Mount_u02 -sys oaprod1   
# online the give resource at the specified system
# resource name is as per config (Main.cf)
# resource and group name are listed by hastatus cmd.  
hares -offline Oracle_oaprod -sys oaprod2 
# offline the whole resrouce group on the specified system
# migration to another node will NOT happen for -offline.
 hagrp -switch oracle_group -to oaprod1   
# switch a service group to the specified system hares -modify Oracle_oaprod Owner oracle 
# change resouce=Oracle_oaprod attribute=Owner new_value=oracle
haconf -dump       
# save vcs config to Main.cf (edited via special command)
# do not edit Main.cf while cluster is up, it will be ignored. haconf -dump -makero    
# equiv of “close config” of hagui, config still kept by conf editor

seq of offline commands:
hares -offline Netlsnr_oaprod -sys oaprod2
hares -offline Oracle_oaprod  -sys oaprod2
hares -offline Mount_u02      -sys oaprod2
hares -offline Mount_u03      -sys oaprod2
hares -offline Volume_u02     -sys oaprod2
hares -offline Volume_u03     -sys oaprod2
hares -offline DiskGroup_oracledg -sys oaprod2  
# some sort of high level container wrapper.
hares -offline IPMultiNICB_oaprod -sys oaprod2
hares -clear   Oracle_oaprod  -sys oaprod1
hares -online  Oracle_oaprod  -sys oaprod1  
 # bring up oracle service group, w/ all deps
hagrp -switch  oracle_group   -to  oaprod2
—
config eg, for adding oracle test group.

haconf -makerw
hagrp -freeze oracle_group
hares -modify Oracle_oaprod User veritas_monitor
hares -modify Oracle_oaprod Pword veritas_password
hares -modify Oracle_oaprod Table monitor
hares -modify Oracle_oaprod MonScript “./bin/Oracle/SqlTest.pl”
hares -modify Oracle_oaprod DetailMonitor 1
haconf -dump -makero
hagrp -unfreeze oracle_group
—
config commands (typically located in /opt/VRTS/bin):

hacf -verify /etc/VRTSvcs/conf/config/
 verify that the main.cf config file is correct, parseable.

haconf -makerw
turn config to be read write, so that changes can be made via haclus
haconf -dump -makero
save and close config from rw, must remember to do this, or else reboot will have issues!
haclus …
change cluster config param.  (CLI change instead of gui).

hauser -add vcsuser
Add a new user that can use hagui, it will prompt for the new password of the new user.

Read Full Post »

Solaris Hardware commands

format  = slice/partition disk, surface scan, etc.  Linux/DOS call this fdisk.
Note that under part submenu, use “label” to save changes to the partition table to disk.
Use “volname” to add a name to the disk volume (shown in format disk list)
prtvtoc : print the volume table of content (vtoc, ie the partition table + disk geometry data)
swap -l                                 list swap info
swap -a /dev/dsk/c…   add slide as swap
swap -d /dev/dsk/c…   delete  slide as swap
drvconfig; disks        : create entries in /dev/dsk/c*t* …
drvconfig; tapes        : create entries for backup tape drives in /dev/rmt
     : sometime drvcnofig cause problem, device config need boot -r to fix.
devfsadm     : “new” solaris command for scanning new storage devices.
drvconfig; tapes; devlinks  : tell system to reconfigure for new tape drive, eg /dev/rmt/0cbn etc
Fiber Channel commands:
cfgadm -c configure [c3]     
  # configure controller 3 (HBA), scan san for LUN
  # run devfsadm if needed, then see new “disks” in format
cfgadm -c unconfigure c3        # remove all config of the given controller
cfgadm -c unconfigure c0::dsk/c0t11d0 
# unconfigure internal scsi disk (eg E250)
# so that dead disk no longer show up in “format”
# but still shows up in cfgadm -al
# (may need a reconfigure reboot to completely clear it)
cfgadm -c unconfigure c3::wwn  
# remove spurious entries in /etc/cfg/fp/fabric_WWN_map devices.
# such device cause boot warnings if left in there.
cfgadm -o force_update -c unconfigure cX::wwn   # forceful manner of above
cfgadm -c unconfigure -o unusable_FCP_dev cX:wwn
luxadm fcode_download -p
        display HBA firmware version and driver/path info.
        luxadm is probably only for 880 w/ sse dev, and some sun array products.
luxadm probe
        display WWN of fc dev

luxadm display [logical_dev]
        …

Read Full Post »

NIC Configuration in Solaris

ndd -get /dev/hme status_link   # query nic speed, see ndd ref in email
ndd -get /dev/hme \?            # list all possible param

ndd -get /dev/hme \? | fgrep -v ‘?’ |  awk ‘{print “echo ” $1 “; ndd -get /dev/hme ” $1 }’  | sh
  # display all NIC parameters, must run as root

ndd -get /dev/ip \? | fgrep -v ‘?’ |  awk ‘{ print $1 }’ | awk -F\( ‘{print “echo; echo —- ” $1 ” —-; ndd -get /dev/ip ” $1 ” ; echo”}’ | sh
  # display lot of IP info.  May want to pipe it to less…

ndd -get /dev/tcp \? | egrep -v ‘\?|obsolete’ | awk ‘{print “echo; echo —- $1 ” —-; ndd -get /dev/tcp ” $1 ” ; echo”}’ | sh
  # display lot of TCP info.

kstat -p hme:0::’/collisions|framing|crc|code_violations|tx_late_collisions/’
kstat -p dmfe:0::’/collisions|framing|crc|code_violations|tx_late_collisions/’
 # get NIC collision stat from kernel stat.  Runnable as user. 
/etc/hostname.hme0 # default hostname/IP
/etc/hosts              # solaris is actually /etc/inet/hosts
/etc/nodename
/etc/inet/ipnodes    # solaris 10 also put IP address in here, manual update!

ifconfig -a
ifconfig hme0 plumb
ifconfig hme0 10.10.0.101 broadcast 10.10.0.255 netmask 255.255.255.0 up

ifconfig hme0 dhcp     for DHCP instead of static IP (see USAH).

hostname

adding statig roures in dual homed host:
route add net [network number] [gateway], eg
route add net 172.17.224.0 172.17.160.1
Note that [gateway] is within the local network (ie 1 hop)
from one of the interfaces in the computer.
In this case, this computer had hme1=172.17.160.8.

solaris adding default route (usually in /etc/defaultrouter)
route add default [IP]

Read Full Post »