性能场景之网络模拟-CFANZ编程社区

模拟网络状况的有很多种，这里说的是Linux上的TC工具。

什么是TC

TC（traffic control)是Linux中的流量控制工具。它是通过控制netem来实现的网络场景模拟。该工具是直接对物理网卡生效的，如果是逻辑网卡，则该控制无效。如果是用的虚拟机，可视虚拟网卡为物理网卡。

什么是HTB

HTB是Hierarchy Token Bucket的缩写。通过在实践基础上的改进，它实现了一个丰富的连接共享类别体系。使用HTB可以很容易地保证每个类别的带宽，虽然它也允许特定的类可以突破带宽上限，占用别的类的带宽。HTB可以通过TBF(Token Bucket Filter)实现带宽限制，也能够划分类别的优先级。

TC使用步骤

要对网卡进行流量控制的配置，需要进行如下的步骤:

为网卡配置一个队列;
在该队列上建立分类;
根据需要建立子队列和子分类;
为每个分类建立过滤器；
建立与过滤器配合的路由表；

操作实例

正常情况下的网络表现

产生流量的语句： server端：

1. iperf -s

Client端：

1. iperf -c 172.17.211.143 -p 5001 -i 2 -P 5

产生的结果：

Server端：

1. [ 12]  0.0-10.1 sec  95.2 MBytes  79.3 Mbits/sec
2. [ 10]  0.0-10.1 sec   158 MBytes   131 Mbits/sec
3. [  6]  0.0-10.1 sec   116 MBytes  95.7 Mbits/sec
4. [  9]  0.0-10.2 sec   143 MBytes   118 Mbits/sec
5. [  4]  0.0-10.3 sec   183 MBytes   150 Mbits/sec
6. [  7]  0.0-10.3 sec   117 MBytes  96.0 Mbits/sec
7. [ 11]  0.0-10.3 sec   156 MBytes   127 Mbits/sec
8. [ 14]  0.0-10.3 sec   138 MBytes   113 Mbits/sec
9. [  8]  0.0-10.3 sec   136 MBytes   111 Mbits/sec
10. [  5]  0.0-10.3 sec   162 MBytes   132 Mbits/sec
11. [SUM]  0.0-10.3 sec  1.37 GBytes  1.14 Gbits/sec

Client端：

1. [ ID] Interval       Transfer     Bandwidth
2. [  6]  0.0- 2.0 sec  88.9 MBytes   373 Mbits/sec
3. [  4]  0.0- 2.0 sec  85.8 MBytes   360 Mbits/sec
4. [  5]  0.0- 2.0 sec  68.2 MBytes   286 Mbits/sec
5. [  7]  0.0- 2.0 sec  52.2 MBytes   219 Mbits/sec
6. [  3]  0.0- 2.0 sec  92.5 MBytes   388 Mbits/sec
7. [SUM]  0.0- 2.0 sec   388 MBytes  1.63 Gbits/sec
8. [  4]  2.0- 4.0 sec  62.4 MBytes   262 Mbits/sec
9. [  6]  2.0- 4.0 sec  48.9 MBytes   205 Mbits/sec
10. [  5]  2.0- 4.0 sec  27.4 MBytes   115 Mbits/sec
11. [  3]  2.0- 4.0 sec  68.9 MBytes   289 Mbits/sec
12. [  7]  2.0- 4.0 sec  44.6 MBytes   187 Mbits/sec
13. [SUM]  2.0- 4.0 sec   252 MBytes  1.06 Gbits/sec
14. [  3]  4.0- 6.0 sec  45.5 MBytes   191 Mbits/sec
15. [  5]  4.0- 6.0 sec  30.0 MBytes   126 Mbits/sec
16. [  4]  4.0- 6.0 sec  54.8 MBytes   230 Mbits/sec
17. [  6]  4.0- 6.0 sec  69.4 MBytes   291 Mbits/sec
18. [  7]  4.0- 6.0 sec  53.1 MBytes   223 Mbits/sec
19. [SUM]  4.0- 6.0 sec   253 MBytes  1.06 Gbits/sec
20. [  4]  6.0- 8.0 sec  40.4 MBytes   169 Mbits/sec
21. [  6]  6.0- 8.0 sec  25.6 MBytes   107 Mbits/sec
22. [  7]  6.0- 8.0 sec  76.1 MBytes   319 Mbits/sec
23. [  3]  6.0- 8.0 sec  59.1 MBytes   248 Mbits/sec
24. [  5]  6.0- 8.0 sec  38.2 MBytes   160 Mbits/sec
25. [SUM]  6.0- 8.0 sec   240 MBytes  1.00 Gbits/sec
26. [  6]  8.0-10.0 sec  37.8 MBytes   158 Mbits/sec
27. [  6]  0.0-10.0 sec   270 MBytes   227 Mbits/sec
28. [  4]  8.0-10.0 sec  39.9 MBytes   167 Mbits/sec
29. [  4]  0.0-10.1 sec   283 MBytes   234 Mbits/sec
30. [  5]  8.0-10.0 sec  40.8 MBytes   171 Mbits/sec
31. [  5]  0.0-10.1 sec   205 MBytes   169 Mbits/sec
32. [  7]  8.0-10.0 sec  48.0 MBytes   201 Mbits/sec
33. [  7]  0.0-10.1 sec   274 MBytes   227 Mbits/sec
34. [  3]  8.0-10.0 sec  84.8 MBytes   355 Mbits/sec
35. [SUM]  8.0-10.0 sec   251 MBytes  1.05 Gbits/sec
36. [  3]  0.0-10.2 sec   351 MBytes   289 Mbits/sec
37. [SUM]  0.0-10.2 sec  1.35 GBytes  1.14 Gbits/sec

做了多遍，都和上面的结果相似。5个线程加在一起每秒大概1Gbits左右。

模拟丢包

模拟命令

1. tc qdisc add dev eth0 root netem loss 10%

丢包率与带宽的关系

模拟丢包率	吞吐量	降幅
关闭模拟	1.14 Gbits/sec
1%	1.14 Gbits/sec	0 Mbits/sec
2%	602 Mbits/sec	538 Mbits/sec
3%	335 Mbits/sec	267 Mbits/sec
4%	215 Mbits/sec	120 Mbits/sec
5%	130 Mbits/sec	85 Mbits/sec
6%	79.9 Mbits/sec	50.1 Mbits/sec
7%	58.9 Mbits/sec	21 Mbits/sec
8%	39.7 Mbits/sec	19.2 Mbits/sec
9%	24.7 Mbits/sec	15 Mbits/sec
10%	19.3 Mbits/sec	5.4 Mbits/sec

模拟延迟

模拟命令

1. tc qdisc add dev eth0 root netem delay 100ms

延迟与带宽的关系

模拟延迟（按10%浮动）	吞吐量
关闭模拟	1.14 Gbits/sec
10 ms	1.02 Gbits/sec
20 ms	483 Mbits/sec
30 ms	351 Mbits/sec
40 ms	254 Mbits/sec
50ms	201 Mbits/sec
60 ms	130 Mbits/sec
70 ms	116 Mbits/sec
80 ms	98.1 Mbits/sec
90 ms	92.0 Mbits/sec
100 ms	66.9 Mbits/sec

HTB队列分层限制

分类配置

1. tc qdisc add dev eth0 root handle 1: htb default 2
2. 
3. tc class add dev eth0 parent 1: classid 1:1 htb rate 100Mbps ceil 100Mbps
4. tc class add dev eth0 parent 1:1 classid 1:2 htb rate 20Mbps ceil 20Mbps
5. tc class add dev eth0 parent 1:1 classid 1:3 htb rate 50Mbps ceil 50Mbps
6. tc class add dev eth0 parent 1:1 classid 1:4 htb rate 20Mbps ceil 20Mbps
7. 
8. tc filter add dev eth0 parent 1:0 protocol ip prio 100 route
9. tc filter add dev eth0 parent 1:0 protocol ip prio 100 route to 2 flowid 1:2
10. tc filter add dev eth0 parent 1:0 protocol ip prio 100 route to 3 flowid 1:3
11. tc filter add dev eth0 parent 1:0 protocol ip prio 100 route to 4 flowid 1:4
12. 
13. ip route add 172.17.211.144 dev eth0 via 172.17.211.143 realm 2
14. [root@7dgroup ~]# tc -s class ls dev eth0
15. class htb 1:1 root rate 800000Kbit ceil 800000Kbit burst 1600b cburst 1600b
16.  Sent 1350897 bytes 6146 pkt (dropped 0, overlimits 0 requeues 0)
17.  rate 0bit 0pps backlog 0b 0p requeues 0
18.  lended: 0 borrowed: 0 giants: 0
19.  tokens: 234 ctokens: 234
20. 
21. class htb 1:2 parent 1:1 prio 0 rate 160000Kbit ceil 160000Kbit burst 1600b cburst 1600b
22.  Sent 1350897 bytes 6146 pkt (dropped 0, overlimits 0 requeues 0)
23.  rate 0bit 0pps backlog 0b 0p requeues 0
24.  lended: 5850 borrowed: 0 giants: 0
25.  tokens: 1170 ctokens: 1170
26. 
27. class htb 1:3 parent 1:1 prio 0 rate 400000Kbit ceil 400000Kbit burst 1600b cburst 1600b
28.  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
29.  rate 0bit 0pps backlog 0b 0p requeues 0
30.  lended: 0 borrowed: 0 giants: 0
31.  tokens: 500 ctokens: 500
32. 
33. class htb 1:4 parent 1:1 prio 0 rate 160000Kbit ceil 160000Kbit burst 1600b cburst 1600b
34.  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
35.  rate 0bit 0pps backlog 0b 0p requeues 0
36.  lended: 0 borrowed: 0 giants: 0
37.  tokens: 1250 ctokens: 1250

限流效果

测试手段从A机往B机用iperf开5个线程发数据包。

不限制

1. [  5]  0.0-10.0 sec   168 MBytes   140 Mbits/sec
2. [  7]  0.0-10.1 sec  75.2 MBytes  62.4 Mbits/sec
3. [  8]  0.0-10.2 sec   210 MBytes   172 Mbits/sec
4. [  4]  0.0-10.3 sec  92.8 MBytes  75.7 Mbits/sec
5. [  6]  0.0-10.3 sec   158 MBytes   129 Mbits/sec
6. [SUM]  0.0-10.3 sec   704 MBytes   574 Mbits/sec

限10M

7. [  8]  0.0-10.2 sec  16.6 MBytes  13.7 Mbits/sec
8. [  4]  0.0-10.2 sec  16.4 MBytes  13.5 Mbits/sec
9. [  5]  0.0-10.2 sec  14.5 MBytes  11.9 Mbits/sec
10. [  6]  0.0-10.2 sec  25.8 MBytes  21.2 Mbits/sec
11. [  7]  0.0-10.2 sec  19.8 MBytes  16.2 Mbits/sec
12. [SUM]  0.0-10.2 sec  93.0 MBytes  76.4 Mbits/sec

限20M

13. [  5]  0.0-10.1 sec  55.6 MBytes  46.0 Mbits/sec
14. [  7]  0.0-10.2 sec  28.9 MBytes  23.8 Mbits/sec
15. [  9]  0.0-10.2 sec  26.1 MBytes  21.6 Mbits/sec
16. [  4]  0.0-10.2 sec  45.0 MBytes  37.1 Mbits/sec
17. [  6]  0.0-10.2 sec  29.5 MBytes  24.3 Mbits/sec
18. [SUM]  0.0-10.2 sec   185 MBytes   153 Mbits/sec

限30M

19. [  4]  0.0-10.2 sec  53.0 MBytes  43.7 Mbits/sec
20. [  6]  0.0-10.2 sec  62.0 MBytes  51.1 Mbits/sec
21. [  8]  0.0-10.2 sec  57.9 MBytes  47.7 Mbits/sec
22. [  5]  0.0-10.2 sec  58.5 MBytes  48.2 Mbits/sec
23. [  7]  0.0-10.2 sec  46.4 MBytes  38.2 Mbits/sec
24. [SUM]  0.0-10.2 sec   278 MBytes   229 Mbits/sec

限40M

25. [  5]  0.0-10.1 sec  76.6 MBytes  63.5 Mbits/sec
26. [  9]  0.0-10.1 sec  76.9 MBytes  63.6 Mbits/sec
27. [  6]  0.0-10.1 sec  72.4 MBytes  59.9 Mbits/sec
28. [  7]  0.0-10.1 sec  70.6 MBytes  58.5 Mbits/sec
29. [  4]  0.0-10.1 sec  72.9 MBytes  60.3 Mbits/sec
30. [SUM]  0.0-10.1 sec   369 MBytes   305 Mbits/sec

限50M

31. [  4]  0.0-10.1 sec  89.9 MBytes  74.5 Mbits/sec
32. [  5]  0.0-10.1 sec  99.6 MBytes  82.5 Mbits/sec
33. [  8]  0.0-10.1 sec  89.9 MBytes  74.3 Mbits/sec
34. [  6]  0.0-10.1 sec  91.9 MBytes  76.0 Mbits/sec
35. [  7]  0.0-10.2 sec  89.8 MBytes  74.1 Mbits/sec
36. [SUM]  0.0-10.2 sec   461 MBytes   381 Mbits/sec

限60M

37. [  4]  0.0-10.1 sec   107 MBytes  89.1 Mbits/sec
38. [  7]  0.0-10.1 sec   121 MBytes   101 Mbits/sec
39. [  9]  0.0-10.1 sec   108 MBytes  89.3 Mbits/sec
40. [  5]  0.0-10.1 sec   107 MBytes  89.1 Mbits/sec
41. [  6]  0.0-10.1 sec   107 MBytes  89.2 Mbits/sec
42. [SUM]  0.0-10.1 sec   550 MBytes   457 Mbits/sec

限70M

43. [  8]  0.0-10.1 sec   178 MBytes   148 Mbits/sec
44. [  7]  0.0-10.1 sec  94.4 MBytes  78.5 Mbits/sec
45. [  4]  0.0-10.1 sec  95.0 MBytes  78.9 Mbits/sec
46. [  6]  0.0-10.1 sec  94.6 MBytes  78.6 Mbits/sec
47. [  5]  0.0-10.1 sec   178 MBytes   148 Mbits/sec
48. [SUM]  0.0-10.1 sec   640 MBytes   531 Mbits/sec

限80M

49. [  7]  0.0-10.0 sec   167 MBytes   140 Mbits/sec
50. [  9]  0.0-10.1 sec   166 MBytes   137 Mbits/sec
51. [  4]  0.0-10.2 sec  99.8 MBytes  82.4 Mbits/sec
52. [  5]  0.0-10.2 sec   157 MBytes   129 Mbits/sec
53. [  6]  0.0-10.2 sec   110 MBytes  90.2 Mbits/sec
54. [SUM]  0.0-10.2 sec   700 MBytes   574 Mbits/sec

限90M

55. [  4]  0.0-10.0 sec   220 MBytes   184 Mbits/sec
56. [  7]  0.0-10.2 sec   124 MBytes   102 Mbits/sec
57. [  5]  0.0-10.2 sec   104 MBytes  85.2 Mbits/sec
58. [  8]  0.0-10.2 sec   117 MBytes  96.2 Mbits/sec
59. [  6]  0.0-10.2 sec   135 MBytes   111 Mbits/sec
60. [SUM]  0.0-10.2 sec   699 MBytes   573 Mbits/sec

限100M

61. [  4]  0.0-10.1 sec   140 MBytes   116 Mbits/sec
62. [  7]  0.0-10.1 sec   139 MBytes   116 Mbits/sec
63. [  6]  0.0-10.1 sec   145 MBytes   121 Mbits/sec
64. [  5]  0.0-10.1 sec   128 MBytes   106 Mbits/sec
65. [  9]  0.0-10.1 sec   146 MBytes   121 Mbits/sec
66. [SUM]  0.0-10.1 sec   698 MBytes   579 Mbits/sec

有很多种手段可以模拟网络丢包、延迟、限流的情况，大家可以自行探索。