nuttcpでネットワークの状態を簡易的に調査・検証する事が出来ます。
—————————————————————-
nuttcp is a TCP/UDP network testing tool, much like iperf.
I think it’s the best such tool available, for its simplicity, ease of use, and feature set.

http://www.lcp.nrl.navy.mil/nuttcp/
対象ドメイン navy.mil (mil:IANAトップドメイン情報)
組織・国・地域 military 米軍事機関用


Phil Dykstra’s nuttcp quick start guide

今回の検証は、最新のSTABLEバージョンで行いましたが、
6.2.7以降では再送処理も取得出来るそうです。
—————————————————————-
TCP retransmit info (Linux only)
Starting in version 6.2.7, nuttcp includes retransmit info for Linux.
This is much more convenient that using tcpdump to determine if the path has lots of congestion.

サーバー側
http://lcp.nrl.navy.mil/nuttcp/stable/rpm/nuttcp-5.3.1-1.i386.rpm
検証ではLinuxにてサーバーを実行

[root@colinux ~]# wget http://www.lcp.nrl.navy.mil/nuttcp/stable/rpm/nuttcp-5.3.1-1.i386.rpm
--2012-06-03 07:53:20--  http://www.lcp.nrl.navy.mil/nuttcp/stable/rpm/nuttcp-5.3.1-1.i386.rpm
www.lcp.nrl.navy.mil をDNSに問いあわせています... 132.250.114.61
www.lcp.nrl.navy.mil|132.250.114.61|:80 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 200 OK
長さ: 52369 (51K) [application/x-rpm]
`nuttcp-5.3.1-1.i386.rpm' に保存中
100%[==========================================================================>] 52,369      82.5K/s 時間 0.6
2012-06-03 07:53:21 (82.5 KB/s) - `nuttcp-5.3.1-1.i386.rpm' へ保存完了 [52369/52369]
[root@colinux ~]# ls -l


[root@colinux ~]# rpm -ivh nuttcp-5.3.1-1.i386.rpm
準備中...                   ########################################### [100%]
   1:nuttcp                 ########################################### [100%]
[root@colinux ~]#


[root@colinux ~]# nuttcp -S
[root@colinux ~]# ps -ef | grep nuttcp
root      2882     1  0 08:01 pts/0    00:00:00 nuttcp -S
root      2884  2837  1 08:01 pts/0    00:00:00 grep nuttcp
[root@colinux ~]#

クライアント側
http://lcp.nrl.navy.mil/nuttcp/stable/bin/nuttcp-5.3.1.win32.zip
検証ではWindowsにてクライアントを実行

Colinuxで検証したので、以下のPortへのアクセス許可を設定してあります。
port number to send to|listen at (default 5001)
port number for control connection (default 5000)

nuttcp client

Help内容(オプション)
—————————————————————-
Usage: nuttcp or nuttcp -h prints this usage info
Usage: nuttcp -V prints version info
Usage: nuttcp -xt [-m] host forward and reverse traceroute to/from server
Usage (transmitter): nuttcp [-t] [-options] host [3rd-party] [ out ]
-4 Use IPv4
-6 Use IPv6
-c## cos dscp value on data streams (t|T suffix for full TOS field)
-l## length of network write|read buf (default 1K|8K/udp, 64K/tcp)
-s use stdin|stdout for data input|output instead of pattern data
-n## number of source bufs written to network (default unlimited)
-w## transmitter|receiver window size in KB (or (m|M)B or (g|G)B)
-ws## server receive|transmit window size in KB (or (m|M)B or (g|G)B)
-wb braindead Solaris 2.8 (sets both xmit and rcv windows)
-p## port number to send to|listen at (default 5001)
-P## port number for control connection (default 5000)
-u use UDP instead of TCP
-m## use multicast with specified TTL instead of unicast (UDP)
-M## MSS for data connection (TCP)
-N## number of streams (starting at port number), implies -B
-R## transmit rate limit in Kbps (or (m|M)bps or (g|G)bps or (p)ps)
-T## transmit timeout in seconds (or (m|M)inutes or (h|H)ours)
-i## receiver interval reporting in seconds (or (m|M)inutes)
-Ixxx identifier for nuttcp output (max of 40 characters)
-F flip option to reverse direction of data connection open
-d set TCP SO_DEBUG option on data socket
-v[v] verbose [or very verbose] output
-b brief output (default)
-D xmit only: don’t buffer TCP writes (sets TCP_NODELAY sockopt)
-B recv only: only output full blocks of size from -l## (for TAR)
Usage (server): nuttcp -S[f][P] [-options]
note server mode excludes use of -s
‘f’ suboption forces server mode (useful with rsh/ssh)
‘P’ suboption makes 3rd party {in,out}bound control ports same
-4 Use IPv4 (default)
-6 Use IPv6
-1 oneshot server mode (implied with inetd/xinetd), implies -S
-P## port number for server connection (default 5000)
note don’t use with inetd/xinetd (use services file instead)
–no3rdparty don’t allow 3rd party capability
–nofork don’t fork server
Format options:
-fxmitstats also give transmitter stats (MB) with -i (UDP only)
-frunningtotal also give cumulative stats on interval reports
-f-drops don’t give packet drop info on brief output (UDP)
-f-percentloss don’t give %loss info on brief output (UDP)
-fparse generate key=value parsable output

Bufferサイズを変更してネットワーク診断

C:\Tools\tmp\nuttcp>nuttcp-5.3.1.exe -v -v -i1 192.168.11.10
nuttcp-t: v5.3.1: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 192.168.11.10
nuttcp-t: time limit = 10.00 seconds
nuttcp-t: connect to 192.168.11.10 with mss=0
nuttcp-t: send window size = 8192, receive window size = 8192
nuttcp-r: v5.3.1: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: interval reporting every 1.00 second
nuttcp-r: accept from 192.168.11.2
nuttcp-r: send window size = 8192, receive window size = 8192
    1.0000 MB /   1.00 sec =    8.3886 Mbps
    1.3125 MB /   1.00 sec =   11.0100 Mbps
    0.8125 MB /   1.00 sec =    6.8157 Mbps
    1.1250 MB /   1.00 sec =    9.4372 Mbps
    2.1250 MB /   1.00 sec =   17.8258 Mbps
    1.3750 MB /   1.00 sec =   11.5343 Mbps
    0.7500 MB /   1.00 sec =    6.2915 Mbps
    1.2500 MB /   1.00 sec =   10.4858 Mbps
    0.8750 MB /   1.00 sec =    7.3400 Mbps
    1.1250 MB /   1.00 sec =    9.4372 Mbps
nuttcp-t: 11.7500 MB in 10.00 real seconds = 1203.20 KB/sec = 9.8566 Mbps
nuttcp-t: 188 I/O calls, msec/call = 54.47, calls/sec = 18.80
nuttcp-t: 0.0user 0.0sys 0:10real 0% 0i+0d 1806maxrss 20+0pf 0+0csw

nuttcp-r: 11.7500 MB in 10.12 real seconds = 1188.35 KB/sec = 9.7349 Mbps
nuttcp-r: 2711 I/O calls, msec/call = 3.82, calls/sec = 267.75
nuttcp-r: 0.0user 0.0sys 0:10real 0% 0i+0d 1690maxrss 0+0pf 0+0csw

---------------------------------------------------------------
http://www.wcisd.hpc.mil/nuttcp/Nuttcp-HOWTO.html
  -i1       to watch tests run (1 second intervals)
  -w8m      to set socket buffers ("window") to 8 MBytes
  -u        for UDP tests
  -R10m     for a 10 Mbps UDP test (or TCP rate limit)
  -l512     to set UDP packet length (or TCP write size)
---------------------------------------------------------------


C:\Tools\tmp\nuttcp>nuttcp-5.3.1.exe -w8m -v -v -i1 192.168.11.10
nuttcp-t: v5.3.1: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 192.168.11.10
nuttcp-t: time limit = 10.00 seconds
nuttcp-t: connect to 192.168.11.10 with mss=0
nuttcp-t: send window size = 8388608, receive window size = 8192
nuttcp-r: v5.3.1: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: interval reporting every 1.00 second
nuttcp-r: accept from 192.168.11.2
nuttcp-r: send window size = 8192, receive window size = 8388608
   14.9651 MB /   1.00 sec =  125.5366 Mbps
   17.3642 MB /   1.00 sec =  145.6613 Mbps
   37.0160 MB /   1.42 sec =  218.3634 Mbps
   21.7070 MB /   0.59 sec =  306.5508 Mbps
   23.5170 MB /   1.00 sec =  197.2752 Mbps
nuttcp-t: 119.5625 MB in 10.00 real seconds = 12243.20 KB/sec = 100.2963 Mbps
nuttcp-t: 1913 I/O calls, msec/call = 5.35, calls/sec = 191.30
nuttcp-t: 0.0user 0.1sys 0:10real 2% 0i+0d 1806maxrss 20+0pf 0+0csw

nuttcp-r: 119.5625 MB in 5.88 real seconds = 20839.49 KB/sec = 170.7171 Mbps
nuttcp-r: 17961 I/O calls, msec/call = 0.33, calls/sec = 3057.19
nuttcp-r: 0.3user 1.3sys 0:05real 27% 0i+0d 1690maxrss 0+0pf 0+0csw

C:\Tools\tmp\nuttcp>


Beta版を利用してのretransmissionの確認
動作確認なので、同じホストで検証しています。

[root@colinux ~]# wget http://www.lcp.nrl.navy.mil/nuttcp/beta/nuttcp-7.1.6.c
--2012-06-03 09:09:54--  http://www.lcp.nrl.navy.mil/nuttcp/beta/nuttcp-7.1.6.c
www.lcp.nrl.navy.mil をDNSに問いあわせています... 132.250.114.61
www.lcp.nrl.navy.mil|132.250.114.61|:80 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 200 OK
長さ: 251878 (246K) 
`nuttcp-7.1.6.c' に保存中

100%[=========================================================================================>] 251,878      113K/s 時間 2.2s

2012-06-03 09:09:57 (113 KB/s) - `nuttcp-7.1.6.c' へ保存完了 [251878/251878]

[root@colinux ~]# 

[root@colinux nuttcp]# cc -O3 -o nuttcp nuttcp-7.1.6.c
[root@colinux nuttcp]# 
[root@colinux nuttcp]# ls -l
合計 392
-rwxr-xr-x 1 root root 138267 2012-06-03 09:11 nuttcp
-rw-r--r-- 1 root root 251878 2012-03-01 06:29 nuttcp-7.1.6.c
[root@colinux nuttcp]# ./nuttcp -S
[root@colinux nuttcp]# ./nuttcp -i1 localhost
   82.7500 MB /   1.00 sec =  694.1462 Mbps     0 retrans
   93.9375 MB /   1.00 sec =  787.9930 Mbps     0 retrans
   94.5625 MB /   1.00 sec =  793.2358 Mbps     0 retrans
   94.8125 MB /   1.00 sec =  795.3330 Mbps     0 retrans
   94.3125 MB /   1.00 sec =  791.1387 Mbps     0 retrans
   94.5625 MB /   1.00 sec =  793.2358 Mbps     0 retrans
   94.4375 MB /   1.00 sec =  792.1873 Mbps     0 retrans
   93.9375 MB /   1.00 sec =  787.9930 Mbps     0 retrans
   93.7500 MB /   1.00 sec =  786.4202 Mbps     0 retrans
   90.7500 MB /   1.00 sec =  761.2548 Mbps     0 retrans

  928.9531 MB /  10.01 sec =  778.4721 Mbps 82 %TX 18 %RX 0 retrans
[root@colinux nuttcp]#

nuttcp compile

nuttcp retrunsmit

nuttcp retrunsmit

retrunsmit

参考:
http://wcisd.hpc.mil/nuttcp/Nuttcp-HOWTO.html
http://fasterdata.es.net/performance-testing/network-troubleshooting-tools/nuttcp/
http://sourceforge.jp/magazine/08/08/22/0159234
http://hylom.net/2011/05/18/nuttcp_thruput_benchmarking/


サーバーを含むシステムの高速化やビックデータ時代の到来に伴い、
分散処理に注目が集まっている様子。

10年前にHPCCが盛り上がった時にはあまり身近に感じなかったが、
HadoopやMongodbのようにオープンサースで気軽に分散処理出来る
システムが導入出来るようになり、ここ2~3年で再び注目を集めている。
忘れがちだったのだが、ネットワークがボトルネックになる可能性も高いので
システム導入の時点できちんとスケールアウトも含めて設計しておく必要がある。

HPCユーザーが知っておきたいTCP/IPの話
ESnet: http://fasterdata.es.net/
———————————————————-
To make better use of its accumulated knowledge, ESnet has developed this Fasterdata Knowledge Base.
The knowledge base provides provides proven, operationally-sound methods for troubleshooting and
solving performance issues. Our solutions fall into five categories:

Network Architecture, including the Science DMZ model
Host Tuning
Network Tuning
Data Transfer Tools
Network Performance Testing
———————————————————-
上記HPCの資料によるとここら辺もきちんとカスタマイズしておいた方が良さそう。
色々なツールもあるので調査したい場合にインストールして現状把握してみても良いかと思います。
nuttcpなどは再送処理なども見つける事が出来るようです。

■Data Transfer Tools
http://fasterdata.es.net/data-transfer-tools/

■Network Troubleshooting Tools
http://fasterdata.es.net/performance-testing/network-troubleshooting-tools/

■Phil Dykstra’s nuttcp quick start guide
http://wcisd.hpc.mil/nuttcp/Nuttcp-HOWTO.html

例)scamperでMTU含めてネットワークパス確認。
———————————————————————-
http://fasterdata.es.net/performance-testing/network-troubleshooting-tools/scamper/

To install scamper:
wget http://www.wand.net.nz/scamper/scamper-cvs-20110421.tar.gz
tar xvzf scamper-cvs-20110421.tar.gz
./configure; make; make install

[root@ip-xxx-xxx-xxx-xxx1 scamper-cvs-20110421]# ./configure; make; make install
checking for a BSD-compatible install… /usr/bin/install -c
checking whether build environment is sane… yes
checking for a thread-safe mkdir -p… /bin/mkdir -p
checking for gawk… gawk
checking whether make sets $(MAKE)… yes
checking build system type… x86_64-unknown-linux-gnu
checking host system type… x86_64-unknown-linux-gnu
checking how to print strings… printf
checking for style of include used by make… GNU
checking for gcc… gcc
checking whether the C compiler works… yes

[root@ip-xxx-xxx-xxx-xxx1 scamper-cvs-20110421]# dig yahoo.co.jp

; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.15.amzn1 <<>> yahoo.co.jp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24120 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;yahoo.co.jp. IN A ;; ANSWER SECTION: yahoo.co.jp. 287 IN A 203.216.243.240 yahoo.co.jp. 287 IN A 124.83.187.140 ;; Query time: 1 msec ;; SERVER: 172.16.0.23#53(172.16.0.23) ;; WHEN: Sun May 27 08:24:32 2012 ;; MSG SIZE rcvd: 61 [root@ip-xxx-xxx-xxx-xxx1 scamper-cvs-20110421]# [root@ip-xxx-xxx-xxx-xxx1 scamper-cvs-20110421]# /usr/local/bin/scamper -c "trace -M" -i 124.83.187.140 traceroute from 10.157.37.241 to 124.83.187.140 1 10.157.36.2 4.163 ms [mtu: 1500] 2 10.1.22.9 0.378 ms [mtu: 1500] 3 175.41.192.21 0.397 ms [mtu: 1500] 4 27.0.0.165 0.321 ms [mtu: 1500] 5 27.0.0.205 7.595 ms [mtu: 1500] 6 27.0.0.188 10.107 ms [mtu: 1500] 7 61.200.80.201 7.698 ms [mtu: 1500] 8 61.200.80.134 7.857 ms [mtu: 1500] 9 61.200.82.138 7.942 ms [mtu: 1500] 10 124.83.128.26 12.923 ms [mtu: 1500] 11 124.83.128.146 9.725 ms [mtu: 1500] 12 124.83.128.146 9.852 ms !X [mtu: 1500] [root@ip-xxx-xxx-xxx-xxx1 scamper-cvs-20110421]#

その他、サーバー側のNICメモリー設定も環境毎に最適化出来る様子。

[root@colinux ~]# /sbin/sysctl -a | grep mem
net.ipv4.udp_wmem_min = 4096
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_mem = 2324160 3098880 4648320
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_mem = 196608 262144 393216
net.ipv4.igmp_max_memberships = 20
net.core.optmem_max = 20480
net.core.rmem_default = 129024
net.core.wmem_default = 129024
net.core.rmem_max = 131071
net.core.wmem_max = 131071
vm.lowmem_reserve_ratio = 256 256 32
vm.overcommit_memory = 0
[root@colinux ~]#

[root@ip-xxx-xxx-xxx-xxx ec2-user]# /sbin/sysctl -a | grep mem
vm.overcommit_memory = 0
vm.lowmem_reserve_ratio = 256 256 32
net.core.wmem_max = 131071
net.core.rmem_max = 131071
net.core.wmem_default = 229376
net.core.rmem_default = 229376
net.core.optmem_max = 20480
net.ipv4.igmp_max_memberships = 20
net.ipv4.tcp_mem = 14679 19574 29358
net.ipv4.tcp_wmem = 4096 16384 626368
net.ipv4.tcp_rmem = 4096 87380 626368
net.ipv4.udp_mem = 14679 19574 29358
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
[root@ip-xxx-xxx-xxx-xxx ec2-user]#

補足:
Windowsに関しては、Windows2008からいくつか注意しておくべき事がありそうです。

TCP 受信ウィンドウの自動調整機能が機能しない正しくで Windows Server 2008 R2

All the TCP/IP ports that are in a TIME_WAIT status are not closed after 497 days

Scalable Networking Pack をご存知ですか?