First, it is important to remember that buffered IO are irrelevant when testing a client to server data transfer. The only thing we measure is a combination of the NFS buffering, server buffering, network layout and operating system asynchronous IO handling. So, the scaled CPU rates are here useless.

Second, IOzone will not be able to perform large buffer IO tests because of the simple fact that NFS will break IO into it's own block size. Pics should be seen whenever IOzone matches the NFS settings. The only test which may show a server limitation is (once again) to perform local IO tests.

IOPerf results

Local IO on server

Nothing much to say
Block IO Character Random
fwrite fread putc getc seek
Machine/FSKB KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU
_ibmtest_400000 27797.0893896.7129.60 35874.44224719.1015.96 9182.749765.6294.03 8266.179205.9889.79 1457.561457.56100.00
1PV_ibmtest_test_400000 33030.55102040.8232.37 36968.58238095.2415.53 9142.869768.0193.60 8271.309214.4789.76 1415.311415.31100.00
NPV_ibmtest_test_400000 2991.5138876.387.69 3007.73214530.061.41 961.549747.689.86 908.159137.909.94 238.411420.0116.83

Client to Server

Results were obtained using a 8 kByte buffer, 5000 IO 1 pass. The test order in that table is :

Block IO Character Random
fwrite fread putc getc seek
Machine/FSKB KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU
_ibmtest_40000 8810.5772727.2712.11 444444.44444444.44100.00 9049.7710869.5783.26 13201.3213201.32100.00 1627.601953.1283.33
1PV_ibmtest_test_40000 10126.5872727.2713.92 444444.44444444.44100.00 10000.0010443.8695.75 13377.9313377.93100.00 1627.601627.60100.00
NPV_ibmtest_test_40000 990.8826386.473.77 207653.13293150.1869.12 917.669475.379.69 3044.1213205.4423.05 1423.811743.8681.68

For those tests, 8 kByte buffer, 50000 IO 1 pass. All nodes invintoried here started their process at the same time. The first line shows a gross-average over all available nodes. We performed 47 simultaneous IO tests.
Comparing to the base-line ( ~ 9 M/sec), the expected fwrite rate in a straight scaling scenario would be 9/47 i.e 215 kB/sec . The number we woud get in a network-saturated scenario is (100 BT connection) 12 MB/sec and 47 nodes i.e. 261 kB/sec.
Looking at the gross-average of 9 M/sec wth a single thread and 188 kB/sec (47 thread), it appears that the numbers scales well with the client-to-server base line number. The only conclusion to this would be that either the server is IO limited or that the NFS-daemon is the limitation.

Block IO Character Random
fwrite fread putc getc seek
Machine/FSKB KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU KB/sec RealKB/sec CPU%CPU
<rcas6*>400000 187.7543215.410.43 166280.69208008.2566.07 115.656411.001.81 6752.187349.4191.07 956.65957.6699.89
rcas6001400000 173.4743478.260.40 254777.07253164.56100.64 99.086443.301.54 6132.157432.1882.51 966.89976.5699.01
rcas6002400000 144.8943196.540.34 367.7088495.580.42 139.086360.312.19 1069.586850.4915.61 986.43986.43100.00
rcas6005400000 232.0543196.540.54 542.42120481.930.45 125.606433.971.95 7425.287430.8099.93 986.43986.43100.00
rcas6006400000 165.0543907.790.38 253164.56254777.0799.37 98.596420.551.54 7426.667426.66100.00 986.43986.43100.00
rcas6007400000 155.9242060.990.37 464.71120120.120.39 152.756426.742.38 7417.027444.6399.63 986.43986.43100.00
rcas6008400000 207.0842553.190.49 270.3345454.550.59 121.956374.501.91 7441.867447.4099.93 986.43986.43100.00
rcas6009400000 240.4343956.040.55 253164.56253164.56100.00 103.986419.521.62 7426.667426.66100.00 986.43986.43100.00
rcas6010400000 205.0343383.950.47 359.80148148.150.24 121.426413.341.89 7417.027419.7799.96 976.56976.56100.00
rcas6011400000 105.0343010.750.24 253164.56254777.0799.37 140.946409.232.20 7419.777419.77100.00 986.43986.43100.00
rcas6012400000 183.4342780.750.43 250000.00250000.00100.00 91.396398.981.43 7397.827422.5399.67 976.56966.89101.00
rcas6013400000 197.5042643.920.46 251572.33250000.00100.63 102.256431.901.59 7425.287426.6699.98 986.43986.43100.00
rcas6014400000 142.2042507.970.33 393.6599502.490.40 141.926386.722.22 1012.637031.1114.40 986.43996.4998.99
rcas6015400000 251.9643668.120.58 253164.56254777.0799.37 90.506436.041.41 7403.297415.6599.83 996.49986.43101.02
rcas6016400000 169.9043431.050.39 245398.77253164.5696.93 132.406440.192.06 7425.287426.6699.98 986.43986.43100.00
rcas6017400000 138.8842060.990.33 317.4561443.930.52 147.736333.122.33 5056.257441.8667.94 976.56996.4998.00
rcas6018400000 149.5344345.900.34 251572.33251572.33100.00 118.426426.741.84 7425.287428.0499.96 966.89976.5699.01
rcas6019400000 156.8643956.040.36 253164.56254777.0799.37 126.796466.211.96 7425.287428.0499.96 986.43986.43100.00
rcas6020400000 123.0543336.940.28 380.15133333.330.29 160.396333.122.53 7423.907429.4299.93 986.43986.43100.00
rcas6021400000 248.8143668.120.57 248447.20251572.3398.76 118.436407.181.85 7423.907426.6699.96 986.43986.43100.00
rcas6022400000 103.7543383.950.24 239520.96242424.2498.80 140.826382.642.21 7408.787412.9099.94 986.43986.43100.00
rcas6023400000 212.7543668.120.49 245398.77250000.0098.16 98.136443.301.52 7403.297437.7199.54 626.00626.00100.00
rcas6024400000 235.8543811.610.54 251572.33251572.33100.00 109.136443.301.69 7443.257444.6399.98 966.89976.5699.01
rcas6025400000 176.3043668.120.40 253164.56251572.33100.63 119.946379.591.88 7417.027415.65100.02 976.56966.89101.00
rcas6026400000 220.8843010.750.51 254777.07254777.07100.00 90.996429.831.42 7425.287432.1899.91 986.43986.43100.00
rcas6027400000 162.7443431.050.37 246913.58248447.2099.38 110.836400.001.73 6694.567425.2890.16 966.89966.89100.00
rcas6028400000 249.3843196.540.58 250000.00250000.00100.00 109.516392.841.71 7406.047426.6699.72 921.29912.68100.94
rcas6029400000 244.0443383.950.56 22637.24251572.339.00 107.756430.871.68 7423.907426.6699.96 986.43986.43100.00
rcas6030400000 254.9342417.820.60 533.40194174.760.27 100.336425.701.56 7441.867446.0299.94 986.43986.43100.00
rcas6031400000 238.0744004.400.54 253164.56253164.56100.00 87.006430.871.35 7423.907429.4299.93 986.43986.43100.00
rcas6032400000 129.3542964.550.30 253164.56253164.56100.00 111.366405.121.74 7411.527411.52100.00 986.43986.43100.00
rcas6033400000 199.8642964.550.47 493.47122699.390.40 113.146418.491.76 6382.647429.4285.91 976.56986.4399.00
rcas6034400000 179.2944004.400.41 254777.07256410.2699.36 91.566405.121.43 7425.287426.6699.98 986.43976.56101.01
rcas6035400000 254.9842598.510.60 254777.07254777.07100.00 91.326404.101.43 7422.537423.9099.98 976.56976.56100.00
rcas6036400000 241.0543811.610.55 238095.24239520.9699.40 85.006442.261.32 7425.287433.5699.89 986.43986.43100.00
rcas6037400000 144.1543010.750.34 254777.07253164.56100.64 115.186449.531.79 7425.287433.5699.89 986.43986.43100.00
rcas6038400000 137.5142872.450.32 253164.56253164.56100.00 127.396414.371.99 7426.667425.28100.02 986.43986.43100.00
rcas6039400000 188.9743715.850.43 1622.72195121.950.83 110.386428.801.72 7422.537428.0499.93 545.57548.6399.44
rcas6040400000 218.2142149.630.52 247.3753120.850.47 122.856310.141.95 1219.486029.5420.22 904.22948.1295.37
rcas6041400000 173.1840899.800.42 284.8560606.060.47 141.086336.132.23 737.156087.3512.11 986.43986.43100.00
rcas6042400000 155.5043010.750.36 253164.56253164.56100.00 100.696417.461.57 7425.287428.0499.96 996.49986.43101.02
rcas6043400000 152.1243859.650.35 370.16170940.170.22 134.766393.862.11 7425.287433.5699.89 996.49986.43101.02
rcas6044400000 194.5143383.950.45 250000.00248447.20100.62 117.546468.311.82 7425.287433.5699.89 986.43986.43100.00
rcas6045400000 149.6642598.510.35 250000.00251572.3399.38 136.836449.532.12 7426.667425.28100.02 986.43986.43100.00
rcas6046400000 134.2044150.110.30 254777.07254777.07100.00 116.996417.461.82 7426.667426.66100.00 986.43986.43100.00
rcas6047400000 272.7543620.500.63 253164.56254777.0799.37 103.556405.121.62 7423.907425.2899.98 986.43986.43100.00
rcas6048400000 199.4244150.110.45 253164.56254777.0799.37 99.946425.701.56 7198.137419.7797.01 642.48642.48100.00
rcas6049400000 209.8442238.650.50 778.2499750.620.78 107.856405.121.68 7220.227429.4297.18 986.43986.43100.00

IOzone profiling

The profiling gives a gross performance view for several kByte buffer and several file size.
The Read/Write tests are unbuffered while FRead/FWrite tests are buffered.

Client to Server

The truncated distribution here are the result of the inhability of the client to push large file sizes ; the performance drops to about nothing. In addition, the unbuffered IO results are deceptive. They show a very large fwrite rate, larger than the ethernet capability. This is again due to a NFS side effect i.e. large blocks pushed through NFS will be broken in smaller buffer size (see rsize=32768 and wsize=32768 parameters) and what we measure is the NFS client performance and its buffering capabilities. Note that ioperf forces a sync/flush after each IO in write mode and isnot subject to this effect.

Local IO on server

There is a server-side intrinsic limitation when writting to a large file. Note that this axis (log2 of file size) is only the file size and has nothing to do with its internal structure. This limitation however seems to be a buffering effect since the unbuffered IO tests (bottom 2 FWrite and FRead) does not show that of a sharp fall (there is still something fishy for large file and large record size but that's expected).

Local IO on ServerClient to Server (beware of NFS)