FREE、TOPなどでもシステム状況を把握して問題解決やリソースのプランニングが可能だが、
他にもいくつか参考になるコマンドがあるのでメモ。

CPU使用率TOP10
ps auxw | sort -k3 -nr | head -n 10
cpu利用率

MEM使用率TOP10
ps auxw | sort -k4 -nr | head -n 10
mem利用率

パフォーマンス確認に便利なdstat
Usage: dstat [-afv] [options..] [delay [count]]

インストール
[root@ip-xx-xxx-xx-xxx ec2-user]# yum install dstat
Loaded plugins: fastestmirror, priorities, security, update-motd
Loading mirror speeds from cached hostfile
* amzn-main: packages.ap-northeast-1.amazonaws.com
* amzn-updates: packages.ap-northeast-1.amazonaws.com
amzn-main | 2.1 kB 00:00
amzn-updates | 2.3 kB 00:00
Setting up Install Process
Resolving Dependencies
–> Running transaction check
—> Package dstat.noarch 0:0.7.0-1.5.amzn1 will be installed
–> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================
Package Arch Version Repository Size
=======================================================================================
Installing:
dstat noarch 0.7.0-1.5.amzn1 amzn-main 182 k

Transaction Summary
=======================================================================================
Install 1 Package(s)

Total download size: 182 k
Installed size: 660 k
Is this ok [y/N]: y
Downloading Packages:
dstat-0.7.0-1.5.amzn1.noarch.rpm | 182 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : dstat-0.7.0-1.5.amzn1.noarch 1/1

Installed:
dstat.noarch 0:0.7.0-1.5.amzn1

Complete!
[root@ip-xx-xxx-xx-xxx ec2-user]#

dstat –top-bioで最もブロックI/Oしているプロセス確認。
dstat –top-ioで最もI/Oが多いプロセスを確認。

[root@ip-xx-xxx-xx-xxx ec2-user]# dstat --top-bio
—-most-expensive—-
block i/o process
init 66k 726B

[root@ip-xx-xxx-xx-xxx ec2-user]# dstat --top-io
—-most-expensive—-
i/o process
bash 82k 1351B
sshd: ec2-u 146B 248B
sshd: ec2-u 78B 168B
sshd: ec2-u 78B 168B
sshd: ec2-u 78B 168B
sshd: ec2-u 78B 168B
sshd: ec2-u 78B 168B

dstat –tcp でTCPコネクション確認
dstat –net でネットワーク確認

[ip-xx-xxx-xx-xxx]#dstat --tcp --net

net状態

[root@ip-xx-xxx-xx-xxx ec2-user]# dstat --top-cpu
-most-expensive-
cpu process
httpd 0.1
mysqld 1.0

[root@ip-xx-xxx-xx-xxx ec2-user]# dstat --net
-net/total-
recv send
0 0
80B 308B
40B 260B
40B 260B
40B 260B
40B 260B
5925B 35k
1710B 24k
4120B 37k
256B 298B
40B 154B
[root@ip-xx-xxx-xx-xxx ec2-user]#

CPUとネットの状況を同時に確認

[root@ip-xx-xxx-xx-xxx ec2-user]# dstat -a
—-total-cpu-usage—- -dsk/total- -net/total- —paging– —system–
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
1 0 98 0 0 0| 92k 12k| 0 0 | 0 0 | 36 66
0 0 100 0 0 0| 0 0 | 80B 916B| 0 0 | 20 46
0 0 100 0 0 0| 0 0 | 40B 452B| 0 0 | 18 44
0 0 100 0 0 0| 0 0 | 40B 452B| 0 0 | 18 46
0 0 100 0 0 0| 0 0 | 40B 452B| 0 0 | 20 46
0 0 100 0 0 0| 0 0 | 40B 452B| 0 0 | 19 47
0 0 100 0 0 0| 0 0 | 40B 452B| 0 0 | 16 42
0 0 100 0 0 0| 0 0 | 40B 452B| 0 0 | 19 46
1 0 99 0 0 0| 0 0 | 40B 452B| 0 0 | 18 43
[root@ip-xx-xxx-xx-xxx ec2-user]#

-a, --all equals -cdngy (default)
all

dtatオプション

[ip-xx-xxx-xx-xxx]#dstat –help
Usage: dstat [-afv] [options..] [delay [count]]
Versatile tool for generating system resource statistics

Dstat options:
-c, –cpu enable cpu stats
-C 0,3,total include cpu0, cpu3 and total
-d, –disk enable disk stats
-D total,hda include hda and total
-g, –page enable page stats
-i, –int enable interrupt stats
-I 5,eth2 include int5 and interrupt used by eth2
-l, –load enable load stats
-m, –mem enable memory stats
-n, –net enable network stats
-N eth1,total include eth1 and total
-p, –proc enable process stats
-r, –io enable io stats (I/O requests completed)
-s, –swap enable swap stats
-S swap1,total include swap1 and total
-t, –time enable time/date output
-T, –epoch enable time counter (seconds since epoch)
-y, –sys enable system stats

–aio enable aio stats
–fs, –filesystem enable fs stats
–ipc enable ipc stats
–lock enable lock stats
–raw enable raw stats
–socket enable socket stats
–tcp enable tcp stats
–udp enable udp stats
–unix enable unix stats
–vm enable vm stats

–plugin-name enable plugins by plugin name (see manual)
–list list all available plugins

-a, –all equals -cdngy (default)
-f, –full automatically expand -C, -D, -I, -N and -S lists
-v, –vmstat equals -pmgdsc -D total

–bw, –blackonwhite change colors for white background terminal
–float force float values on screen
–integer force integer values on screen
–nocolor disable colors (implies –noupdate)
–noheaders disable repetitive headers
–noupdate disable intermediate updates
–output file write CSV output to file

delay is the delay in seconds between each update (default: 1)
count is the number of updates to display before exiting (default: unlimited)

[ip-xx-xxx-xx-xxx]#

上記オプションを適切に使い分けて、最適なシステム状況診断と
対応が出来るようにしておくと良さそうです。

dstat --cpu --mem --disk --page --int --load --net
general

dstat -N eth0
interface

dstat --time --proc --swap --sys --ipc
date

dstat --full
full

dstat --vmstat
vmstat


--integer 整数値を表示します
--nocolor カラー表示を無効にします
--noheaders ヘッダー表示を無効にします
--noupdate 仲介更新を無効にします
--output file 出力結果を CSV ファイルに書き出します

参考
Dstat: Versatile resource statistics tool


nuttcpでネットワークの状態を簡易的に調査・検証する事が出来ます。
—————————————————————-
nuttcp is a TCP/UDP network testing tool, much like iperf.
I think it’s the best such tool available, for its simplicity, ease of use, and feature set.

http://www.lcp.nrl.navy.mil/nuttcp/
対象ドメイン navy.mil (mil:IANAトップドメイン情報)
組織・国・地域 military 米軍事機関用


Phil Dykstra’s nuttcp quick start guide

今回の検証は、最新のSTABLEバージョンで行いましたが、
6.2.7以降では再送処理も取得出来るそうです。
—————————————————————-
TCP retransmit info (Linux only)
Starting in version 6.2.7, nuttcp includes retransmit info for Linux.
This is much more convenient that using tcpdump to determine if the path has lots of congestion.

サーバー側
http://lcp.nrl.navy.mil/nuttcp/stable/rpm/nuttcp-5.3.1-1.i386.rpm
検証ではLinuxにてサーバーを実行

[root@colinux ~]# wget http://www.lcp.nrl.navy.mil/nuttcp/stable/rpm/nuttcp-5.3.1-1.i386.rpm
--2012-06-03 07:53:20--  http://www.lcp.nrl.navy.mil/nuttcp/stable/rpm/nuttcp-5.3.1-1.i386.rpm
www.lcp.nrl.navy.mil をDNSに問いあわせています... 132.250.114.61
www.lcp.nrl.navy.mil|132.250.114.61|:80 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 200 OK
長さ: 52369 (51K) [application/x-rpm]
`nuttcp-5.3.1-1.i386.rpm' に保存中
100%[==========================================================================>] 52,369      82.5K/s 時間 0.6
2012-06-03 07:53:21 (82.5 KB/s) - `nuttcp-5.3.1-1.i386.rpm' へ保存完了 [52369/52369]
[root@colinux ~]# ls -l


[root@colinux ~]# rpm -ivh nuttcp-5.3.1-1.i386.rpm
準備中...                   ########################################### [100%]
   1:nuttcp                 ########################################### [100%]
[root@colinux ~]#


[root@colinux ~]# nuttcp -S
[root@colinux ~]# ps -ef | grep nuttcp
root      2882     1  0 08:01 pts/0    00:00:00 nuttcp -S
root      2884  2837  1 08:01 pts/0    00:00:00 grep nuttcp
[root@colinux ~]#

クライアント側
http://lcp.nrl.navy.mil/nuttcp/stable/bin/nuttcp-5.3.1.win32.zip
検証ではWindowsにてクライアントを実行

Colinuxで検証したので、以下のPortへのアクセス許可を設定してあります。
port number to send to|listen at (default 5001)
port number for control connection (default 5000)

nuttcp client

Help内容(オプション)
—————————————————————-
Usage: nuttcp or nuttcp -h prints this usage info
Usage: nuttcp -V prints version info
Usage: nuttcp -xt [-m] host forward and reverse traceroute to/from server
Usage (transmitter): nuttcp [-t] [-options] host [3rd-party] [ out ]
-4 Use IPv4
-6 Use IPv6
-c## cos dscp value on data streams (t|T suffix for full TOS field)
-l## length of network write|read buf (default 1K|8K/udp, 64K/tcp)
-s use stdin|stdout for data input|output instead of pattern data
-n## number of source bufs written to network (default unlimited)
-w## transmitter|receiver window size in KB (or (m|M)B or (g|G)B)
-ws## server receive|transmit window size in KB (or (m|M)B or (g|G)B)
-wb braindead Solaris 2.8 (sets both xmit and rcv windows)
-p## port number to send to|listen at (default 5001)
-P## port number for control connection (default 5000)
-u use UDP instead of TCP
-m## use multicast with specified TTL instead of unicast (UDP)
-M## MSS for data connection (TCP)
-N## number of streams (starting at port number), implies -B
-R## transmit rate limit in Kbps (or (m|M)bps or (g|G)bps or (p)ps)
-T## transmit timeout in seconds (or (m|M)inutes or (h|H)ours)
-i## receiver interval reporting in seconds (or (m|M)inutes)
-Ixxx identifier for nuttcp output (max of 40 characters)
-F flip option to reverse direction of data connection open
-d set TCP SO_DEBUG option on data socket
-v[v] verbose [or very verbose] output
-b brief output (default)
-D xmit only: don’t buffer TCP writes (sets TCP_NODELAY sockopt)
-B recv only: only output full blocks of size from -l## (for TAR)
Usage (server): nuttcp -S[f][P] [-options]
note server mode excludes use of -s
‘f’ suboption forces server mode (useful with rsh/ssh)
‘P’ suboption makes 3rd party {in,out}bound control ports same
-4 Use IPv4 (default)
-6 Use IPv6
-1 oneshot server mode (implied with inetd/xinetd), implies -S
-P## port number for server connection (default 5000)
note don’t use with inetd/xinetd (use services file instead)
–no3rdparty don’t allow 3rd party capability
–nofork don’t fork server
Format options:
-fxmitstats also give transmitter stats (MB) with -i (UDP only)
-frunningtotal also give cumulative stats on interval reports
-f-drops don’t give packet drop info on brief output (UDP)
-f-percentloss don’t give %loss info on brief output (UDP)
-fparse generate key=value parsable output

Bufferサイズを変更してネットワーク診断

C:\Tools\tmp\nuttcp>nuttcp-5.3.1.exe -v -v -i1 192.168.11.10
nuttcp-t: v5.3.1: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 192.168.11.10
nuttcp-t: time limit = 10.00 seconds
nuttcp-t: connect to 192.168.11.10 with mss=0
nuttcp-t: send window size = 8192, receive window size = 8192
nuttcp-r: v5.3.1: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: interval reporting every 1.00 second
nuttcp-r: accept from 192.168.11.2
nuttcp-r: send window size = 8192, receive window size = 8192
    1.0000 MB /   1.00 sec =    8.3886 Mbps
    1.3125 MB /   1.00 sec =   11.0100 Mbps
    0.8125 MB /   1.00 sec =    6.8157 Mbps
    1.1250 MB /   1.00 sec =    9.4372 Mbps
    2.1250 MB /   1.00 sec =   17.8258 Mbps
    1.3750 MB /   1.00 sec =   11.5343 Mbps
    0.7500 MB /   1.00 sec =    6.2915 Mbps
    1.2500 MB /   1.00 sec =   10.4858 Mbps
    0.8750 MB /   1.00 sec =    7.3400 Mbps
    1.1250 MB /   1.00 sec =    9.4372 Mbps
nuttcp-t: 11.7500 MB in 10.00 real seconds = 1203.20 KB/sec = 9.8566 Mbps
nuttcp-t: 188 I/O calls, msec/call = 54.47, calls/sec = 18.80
nuttcp-t: 0.0user 0.0sys 0:10real 0% 0i+0d 1806maxrss 20+0pf 0+0csw

nuttcp-r: 11.7500 MB in 10.12 real seconds = 1188.35 KB/sec = 9.7349 Mbps
nuttcp-r: 2711 I/O calls, msec/call = 3.82, calls/sec = 267.75
nuttcp-r: 0.0user 0.0sys 0:10real 0% 0i+0d 1690maxrss 0+0pf 0+0csw

---------------------------------------------------------------
http://www.wcisd.hpc.mil/nuttcp/Nuttcp-HOWTO.html
  -i1       to watch tests run (1 second intervals)
  -w8m      to set socket buffers ("window") to 8 MBytes
  -u        for UDP tests
  -R10m     for a 10 Mbps UDP test (or TCP rate limit)
  -l512     to set UDP packet length (or TCP write size)
---------------------------------------------------------------


C:\Tools\tmp\nuttcp>nuttcp-5.3.1.exe -w8m -v -v -i1 192.168.11.10
nuttcp-t: v5.3.1: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 192.168.11.10
nuttcp-t: time limit = 10.00 seconds
nuttcp-t: connect to 192.168.11.10 with mss=0
nuttcp-t: send window size = 8388608, receive window size = 8192
nuttcp-r: v5.3.1: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: interval reporting every 1.00 second
nuttcp-r: accept from 192.168.11.2
nuttcp-r: send window size = 8192, receive window size = 8388608
   14.9651 MB /   1.00 sec =  125.5366 Mbps
   17.3642 MB /   1.00 sec =  145.6613 Mbps
   37.0160 MB /   1.42 sec =  218.3634 Mbps
   21.7070 MB /   0.59 sec =  306.5508 Mbps
   23.5170 MB /   1.00 sec =  197.2752 Mbps
nuttcp-t: 119.5625 MB in 10.00 real seconds = 12243.20 KB/sec = 100.2963 Mbps
nuttcp-t: 1913 I/O calls, msec/call = 5.35, calls/sec = 191.30
nuttcp-t: 0.0user 0.1sys 0:10real 2% 0i+0d 1806maxrss 20+0pf 0+0csw

nuttcp-r: 119.5625 MB in 5.88 real seconds = 20839.49 KB/sec = 170.7171 Mbps
nuttcp-r: 17961 I/O calls, msec/call = 0.33, calls/sec = 3057.19
nuttcp-r: 0.3user 1.3sys 0:05real 27% 0i+0d 1690maxrss 0+0pf 0+0csw

C:\Tools\tmp\nuttcp>


Beta版を利用してのretransmissionの確認
動作確認なので、同じホストで検証しています。

[root@colinux ~]# wget http://www.lcp.nrl.navy.mil/nuttcp/beta/nuttcp-7.1.6.c
--2012-06-03 09:09:54--  http://www.lcp.nrl.navy.mil/nuttcp/beta/nuttcp-7.1.6.c
www.lcp.nrl.navy.mil をDNSに問いあわせています... 132.250.114.61
www.lcp.nrl.navy.mil|132.250.114.61|:80 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 200 OK
長さ: 251878 (246K) 
`nuttcp-7.1.6.c' に保存中

100%[=========================================================================================>] 251,878      113K/s 時間 2.2s

2012-06-03 09:09:57 (113 KB/s) - `nuttcp-7.1.6.c' へ保存完了 [251878/251878]

[root@colinux ~]# 

[root@colinux nuttcp]# cc -O3 -o nuttcp nuttcp-7.1.6.c
[root@colinux nuttcp]# 
[root@colinux nuttcp]# ls -l
合計 392
-rwxr-xr-x 1 root root 138267 2012-06-03 09:11 nuttcp
-rw-r--r-- 1 root root 251878 2012-03-01 06:29 nuttcp-7.1.6.c
[root@colinux nuttcp]# ./nuttcp -S
[root@colinux nuttcp]# ./nuttcp -i1 localhost
   82.7500 MB /   1.00 sec =  694.1462 Mbps     0 retrans
   93.9375 MB /   1.00 sec =  787.9930 Mbps     0 retrans
   94.5625 MB /   1.00 sec =  793.2358 Mbps     0 retrans
   94.8125 MB /   1.00 sec =  795.3330 Mbps     0 retrans
   94.3125 MB /   1.00 sec =  791.1387 Mbps     0 retrans
   94.5625 MB /   1.00 sec =  793.2358 Mbps     0 retrans
   94.4375 MB /   1.00 sec =  792.1873 Mbps     0 retrans
   93.9375 MB /   1.00 sec =  787.9930 Mbps     0 retrans
   93.7500 MB /   1.00 sec =  786.4202 Mbps     0 retrans
   90.7500 MB /   1.00 sec =  761.2548 Mbps     0 retrans

  928.9531 MB /  10.01 sec =  778.4721 Mbps 82 %TX 18 %RX 0 retrans
[root@colinux nuttcp]#

nuttcp compile

nuttcp retrunsmit

nuttcp retrunsmit

retrunsmit

参考:
http://wcisd.hpc.mil/nuttcp/Nuttcp-HOWTO.html
http://fasterdata.es.net/performance-testing/network-troubleshooting-tools/nuttcp/
http://sourceforge.jp/magazine/08/08/22/0159234
http://hylom.net/2011/05/18/nuttcp_thruput_benchmarking/