0
点赞
收藏
分享

微信扫一扫

媒体服务器(MS)抢救记


文章目录

  • ​​前言​​
  • ​​一、abrt获取core文件​​
  • ​​二、gdb分析coredump文件​​
  • ​​三、sipp模拟呼叫​​
  • ​​三、valrind工具​​
  • ​​总结​​

前言

当媒体服务器发行故障后,先后通过分析SIP信令及媒体服务器抓包及服务器日志,定位媒体服务器网元出现故障。
通过abrt抓取进程异常的core文件,使用sipp模拟大呼进行故障场景复现。采用valgrind进行内存泄漏定位。最终定位问题并修复该问题。

提示:以下是本篇文章正文内容

一、abrt获取core文件

上上周,负责江苏现场交付的同事联系我,客户有反馈通话过程中存在通话中断的现象。
我先让现场的抓取了客户的通话信令,发现通话中断时,收到了核心网交换机发送了拆线的指令。原因值是RAP/RTCP TIME OUT。
然后我让现场的同事采用gdb命令在媒体服务器进行网卡抓包,并进行实际拨测,当通话中断时,停止网络抓包。
通过网络包分析,发现媒体服务器存在媒体包丢包情况。
于是判断媒体服务器存在问题了,通过分析进城的运行日志,发现存在重启现象。当中午发生时会导致媒体包丢包,导致通话中断。
至此问题陡然升级,先联系现场配置abrt服务(service arbtd restart),当进程重启时,抓取到coredump。
经和客户沟通后,先下掉媒体服务器,通话旁路。

二、gdb分析coredump文件

gdb ms coredump

然后输入where命令,frame及相关的print等命令,逐一查看相关变量。

三、sipp模拟呼叫

于是编写sipp的uac和uas的模拟桩,uac模拟主叫,uas模拟被叫。同时uac和uas的模拟桩中加入rtp流,让场景变得更真实。
uac脚本如下:

<?xml version="1.0" encoding="ISO-8859-1" ?>

<!-- This program is free software; you can redistribute it and/or -->
<!-- modify it under the terms of the GNU General Public License as -->
<!-- published by the Free Software Foundation; either version 2 of the -->
<!-- License, or (at your option) any later version. -->
<!-- -->
<!-- This program is distributed in the hope that it will be useful, -->
<!-- but WITHOUT ANY WARRANTY; without even the implied warranty of -->
<!-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -->
<!-- GNU General Public License for more details. -->
<!-- -->
<!-- You should have received a copy of the GNU General Public License -->
<!-- along with this program; if not, write to the -->
<!-- Free Software Foundation, Inc., -->
<!-- 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA -->
<!-- -->
<!-- Sipp default 'uac' scenario. -->
<!-- -->

<scenario name="Basic Sipstone UAC">
<!-- In client mode (sipp placing calls), the Call-ID MUST be -->
<!-- generated by sipp. To do so, use [call_id] token. -->
<send>
<![CDATA[

INVITE sip:[field1]@[remote_ip]:[remote_port] SIP/2.0
Via: SIP/2.0/[transport] [local_ip]:[local_port]
From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number]
To: sut <sip:[field1]@[remote_ip]:[remote_port]>
Call-ID: [call_id]
Cseq: [cseq] INVITE
Contact: sip:sipp@[local_ip]:[local_port]
Route: <sip:tagscpxf03.js.ctcims.cn:[remote_port];call=ldmp;lr>
Route: <sip:[local_ip]:8063;lr>
Max-Forwards: 70
P-Asserted-Identity: <sip:[field0]@cintel.com.cn>
Subject: Performance Test
Content-Type: application/sdp
Content-Length: [len]

v=0
o=- 53655765 2353687637 IN IP4 [media_ip]
s=SIPP
c=IN IP4 [media_ip]
t=0 0
m=audio [media_port] RTP/AVP 115 103 102 101 8 96 97 18 0
b=AS:80
b=RS:600
b=RR:2000
a=rtpmap:115 AMR-WB/16000/1
a=fmtp:115 mode-change-capability=2;max-red=0
a=rtpmap:103 AMR-WB/16000/1
a=fmtp:103 octet-align=1;mode-change-capability=2;max-red=0
a=rtpmap:102 AMR/8000/1
a=fmtp:102 mode-change-capability=2;max-red=0
a=rtpmap:101 AMR/8000/1
a=fmtp:101 octet-align=1;mode-change-capability=2;max-red=0
a=rtpmap:8 PCMA/8000
a=rtpmap:96 telephone-event/16000
a=fmtp:96 0-15
a=rtpmap:97 telephone-event/8000
a=fmtp:97 0-15
a=curr:qos local none
a=curr:qos remote none
a=des:qos mandatory local sendrecv
a=des:qos optional remote sendrecv
a=sendrecv
a=maxptime:240
a=ptime:20
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:0 PCMU/8000

]]>
</send>



<recv response="100" >
</recv>

<recv response="183" >
</recv>
<send>
<![CDATA[

PRACK sip:[field1]@[remote_ip]:[remote_port] SIP/2.0
Via: SIP/2.0/[transport] [local_ip]:[local_port]
From: <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number]
To: <sip:[field1]@192.168.2.228:8700>[peer_tag_param]
Call-ID: [call_id]
CSeq: 2 PRACK
RAck: 1 1 INVITE
Contact: <sip:[local_ip]@[local_ip]:[local_port]>
Max-Forwards: 70
Content-Length: 0

]]>
</send>

<recv response="200" >
</recv>
<nop>
<action>
<!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/-->
<exec play_pcap_audio="pcap/amrwb1_115.pcap"/>
</action>
</nop>
<send>
<![CDATA[

UPDATE sip:[remote_ip]:[remote_port] SIP/2.0
Via: SIP/2.0/[transport] [local_ip]:[local_port]
[last_Call-ID:]
From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number]
To: sut <sip:[field1]@[remote_ip]:[remote_port]>[peer_tag_param]
Allow: INVITE,ACK,CANCEL,BYE,UPDATE,PRACK,MESSAGE,REFER,NOTIFY,INFO,OPTIONS
Cseq: 3 UPDATE
Contact: <sip:[local_ip]@[local_ip]:[local_port]>
Require: precondition
Supported: timer
Content-Length: [len]
Content-Type: application/sdp

v=0
o=HuaweiATS9900 171392092 171392093 IN IP4 5.2.169.97
s=SBC call
c=IN IP4 [media_ip]
b=AS:49
b=RS:600
b=RR:2000
t=0 0
m=audio [media_port] RTP/AVP 115 102 8 96 18 0 97
b=AS:80
b=RS:600
b=RR:2000
a=rtpmap:115 AMR-WB/16000/1
a=fmtp:115 mode-change-capability=2;max-red=0
a=rtpmap:102 AMR/8000
a=fmtp:102 mode-change-capability=2;max-red=0
a=rtpmap:8 PCMA/8000
a=rtpmap:96 telephone-event/16000
a=fmtp:96 0-15
a=curr:qos local sendrecv
a=curr:qos remote none
a=des:qos mandatory local sendrecv
a=des:qos mandatory remote sendrecv
a=sendrecv
a=maxptime:240
a=ptime:20
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:0 PCMU/8000
a=rtpmap:97 telephone-event/8000
a=fmtp:97 0-15
]]>
</send>
<recv response="200">
</recv>
<nop>
<action>
<!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/-->
<exec play_pcap_audio="pcap/g711a.pcap"/>
</action>
</nop>
<recv response="180">
</recv>

<recv response="200">
</recv>

<send>
<![CDATA[

ACK sip:[remote_ip]:[remote_port];transport=udp;Hpt=8f28_16;CxtId=3;TRC=ffffffff-ffffffff SIP/2.0
Via: SIP/2.0/[transport] [local_ip]:[local_port]
From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number]
To: sut <sip:[field1]@[remote_ip]:[remote_port]>[peer_tag_param]
Call-ID: [call_id]
Cseq: [cseq] ACK
Contact: sip:sipp@[local_ip]:[local_port]
Max-Forwards: 70
Subject: Performance Test
Content-Length: 0

]]>
</send>
<nop>
<action>
<!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/-->
<exec play_pcap_audio="pcap/amr102.pcap"/>
</action>
</nop>


<!-- This delay can be customized by the -d command-line option -->
<!-- or by adding a 'milliseconds = "value"' option here. -->
<pause milliseconds = "500"/>

<send>
<![CDATA[

INVITE sip:[field1]@[remote_ip]:[remote_port] SIP/2.0
Via: SIP/2.0/[transport] [local_ip]:[local_port]
From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number]
To: sut <sip:[field1]@[remote_ip]:[remote_port]>[peer_tag_param]
Call-ID: [call_id]
Cseq: 4 INVITE
Contact: sip:sipp@[local_ip]:[local_port]
Max-Forwards: 66
P-Early-Media: supported
Subject: Performance Test
Session-Expires: 1800;refresher=uas
Min-SE: 600
Accept-Contact: *;+g.3gpp.icsi-ref="urn%3Aurn-7%3A3gpp-service.ims.icsi.mmtel";video
Content-Type: application/sdp
Content-Length: [len]

v=0
o=HuaweiATS9900 171392092 171392094 IN IP4 5.2.169.97
s=SBC call
c=IN IP4 [media_ip]
b=AS:2211
b=RS:8600
b=RR:8000
t=0 0
m=audio [media_port] RTP/AVP 115 102 8 96 18 0 97
b=AS:80
b=RS:600
b=RR:2000
a=rtpmap:115 AMR-WB/16000/1
a=fmtp:104 mode-change-capability=2;max-red=0
a=rtpmap:102 AMR/8000
a=fmtp:102 mode-change-capability=2;max-red=0
a=rtpmap:8 PCMA/8000
a=rtpmap:96 telephone-event/16000
a=fmtp:96 0-15
a=sendrecv
a=maxptime:240
a=ptime:20
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:0 PCMU/8000
a=rtpmap:97 telephone-event/8000
a=fmtp:97 0-15
m=video 52998 RTP/AVP 114 113
b=AS:2162
b=RS:8000
b=RR:6000
a=rtpmap:114 H264/90000
a=fmtp:114 profile-level-id=42C01F;packetization-mode=1;sar-understood=16;sar-supported=1;sprop-parameter-sets=Z0LAH9oC0ChoBtChNQ==,aM4G4g==
a=imageattr:114 send
a=rtpmap:113 H264/90000
a=fmtp:113 profile-level-id=42C01F;packetization-mode=0;sar-understood=16;sar-supported=1;sprop-parameter-sets=Z0LAH9oC0ChoBtChNQ==,aM4G4g==
a=imageattr:113 send
a=rtcp-fb:* nack
a=rtcp-fb:* nack pli
a=rtcp-fb:* ccm fir
a=rtcp-fb:* ccm tmmbr
a=sendrecv
a=tcap:1 RTP/AVPF
a=pcfg:1 t=1
a=extmap:2 urn:3gpp:video-orientation

]]>
</send>
<recv response="100" ></recv>

<recv response="200" ></recv>
<send>
<![CDATA[

ACK sip:[remote_ip]:[remote_port];transport=udp;Hpt=8f28_16;CxtId=3;TRC=ffffffff-ffffffff SIP/2.0
Via: SIP/2.0/[transport] [local_ip]:[local_port]
From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number]
To: sut <sip:[field1]@[remote_ip]:[remote_port]>[peer_tag_param]
Call-ID: [call_id]
Cseq: 4 ACK
Contact: sip:sipp@[local_ip]:[local_port]
Max-Forwards: 70
Subject: Performance Test
Content-Length: 0

]]>
</send>
<nop>
<action>
<!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/-->
<exec play_pcap_audio="pcap/amrwb1_115.pcap"/>
</action>
</nop>
<pause milliseconds = "30000"/>
<!-- The 'crlf' option inserts a blank line in the statistics report. -->
<send>
<![CDATA[

BYE sip:[field1]@[remote_ip]:[remote_port] SIP/2.0
Via: SIP/2.0/[transport] [local_ip]:[local_port]
From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number]
To: sut <sip:[field1]@[remote_ip]:[remote_port]>[peer_tag_param]
Call-ID: [call_id]
Cseq: 1 BYE
Contact: sip:sipp@[local_ip]:[local_port]
Max-Forwards: 70
Subject: Performance Test
Content-Length: 0

]]>
</send>


<recv response="200" crlf="true">
</recv>


<!-- definition of the response time repartition table (unit is ms) -->
<ResponseTimeRepartition value="10, 20, 30, 40, 50, 100, 150, 200"/>

<!-- definition of the call length repartition table (unit is ms) -->
<CallLengthRepartition value="10, 50, 100, 500, 1000, 5000, 10000"/>

</scenario>

uac的启动脚本如下:

./sipp -sf uac_jsfhgk.xml -inf calling_number_graylist_speech_template_performance.txt -i 192.168.2.179 -p 5788 192.168.2.112:5062 -m 10000 -r 10 -trace_screen -screen_overwrite true -screen_file uac_jsfhgk.log

uas脚本如下:

<?xml version="1.0" encoding="ISO-8859-1" ?>

<!-- This program is free software; you can redistribute it and/or -->
<!-- modify it under the terms of the GNU General Public License as -->
<!-- published by the Free Software Foundation; either version 2 of the -->
<!-- License, or (at your option) any later version. -->
<!-- -->
<!-- This program is distributed in the hope that it will be useful, -->
<!-- but WITHOUT ANY WARRANTY; without even the implied warranty of -->
<!-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -->
<!-- GNU General Public License for more details. -->
<!-- -->
<!-- You should have received a copy of the GNU General Public License -->
<!-- along with this program; if not, write to the -->
<!-- Free Software Foundation, Inc., -->
<!-- 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA -->
<!-- -->
<!-- Sipp default 'uas' scenario. -->
<!-- -->

<scenario name="Basic UAS responder">
<!-- By adding rrs="true" (Record Route Sets), the route sets -->
<!-- are saved and used for following messages sent. Useful to test -->
<!-- against stateful SIP proxies/B2BUAs. -->
<recv request="INVITE" crlf="true">
<action>
<ereg regexp=".*" search_in="hdr" header="CSeq:" assign_to="cseq_invite" />
<ereg regexp=".*" search_in="hdr" header="Via:" assign_to="via_invite" />
<!--ereg regexp="<.*>" search_in="hdr" header="Contact:" assign_to="uac_addr" /-->
<!--ereg regexp=".*" search_in="hdr" header="From:" assign_to="from_invite" /-->
</action>
</recv>

<!-- The '[last_*]' keyword is replaced automatically by the -->
<!-- specified header if it was present in the last message received -->
<!-- (except if it was a retransmission). If the header was not -->
<!-- present or if no message has been received, the '[last_*]' -->
<!-- keyword is discarded, and all bytes until the end of the line -->
<!-- are also discarded. -->
<!-- -->
<!-- If the specified header was present several times in the -->
<!-- message, all occurences are concatenated (CRLF seperated) -->
<!-- to be used in place of the '[last_*]' keyword. -->
<send>
<![CDATA[

SIP/2.0 100 Trying
[last_Via:]
[last_From:]
To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number]
[last_Call-ID:]
[last_CSeq:]
Contact: <sip:[local_ip]:[local_port];transport=[transport]>
Content-Length: 0

]]>
</send>


<send>
<![CDATA[

SIP/2.0 183 Session Progress
[last_Via:]
[last_From:]
To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number]
[last_Call-ID:]
[last_CSeq:]
Require: 100rel,precondition
RSeq: 1
P-Charging-Vector: icid-value="pbac22bhw.19e.627a.20211025080346";orig-ioi=js.ims.mnc011.mcc460.3gppnetwork.org;term-ioi=js.chinamobile.com
Contact: <sip:[local_ip]:[local_port];transport=[transport]>
Content-Length: [len]
Content-Type: application/sdp

v=0
o=- 1635149026 1635149026 IN IP4 5.92.12.134
s=SBC call
c=IN IP4 [media_ip]
b=AS:41
b=RR:1837
b=RS:600
t=0 0
m=audio [media_port] RTP/AVP 115 96
b=AS:41
b=RR:1837
b=RS:600
a=rtpmap:115 AMR-WB/16000/1
a=fmtp:115 mode-change-capability=2;max-red=0
a=ptime:20
a=maxptime:240
a=sendrecv
a=curr:qos local none
a=curr:qos remote none
a=des:qos mandatory local sendrecv
a=des:qos mandatory remote sendrecv
a=conf:qos remote sendrecv
a=rtpmap:96 telephone-event/16000
a=fmtp:96 0-15

]]>
</send>
<recv request="PRACK"></recv>
<send>
<![CDATA[

SIP/2.0 200 OK
[last_Via:]
[last_From:]
To: <sip:01082012002@[remote_ip]:[remote_port]>;tag=[call_number]
[last_Call-ID:]
CSeq: 2 PRACK
Contact: <sip:[local_ip]:[local_port];transport=[transport]>
Content-Length: [len]

]]>
</send>
<nop>
<action>
<!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/-->
<exec play_pcap_audio="pcap/amrwb1_115.pcap"/>
</action>
</nop>
<recv request="UPDATE"></recv>
<send>
<![CDATA[

SIP/2.0 200 OK
[last_Via:]
[last_Call-ID:]
[last_From:]
[last_To:]
[last_CSeq:]
Allow: INVITE,ACK,OPTIONS,BYE,CANCEL,REGISTER,INFO,UPDATE,PRACK,REFER,SUBSCRIBE,NOTIFY,MESSAGE
Contact: <sip:[field1]@[local_ip]:[local_port];transport=udp;Hpt=8ec2_16;CxtId=3;TRC=ffffffff-ffffffff;srti=d0_1025>;video;+g.3gpp.icsi-ref="urn%3Aurn-7%3A3gpp-service.ims.icsi.mmtel";+g.3gpp.mid-call;+g.3gpp.srvcc-alerting
Server: Ericsson MTAS - CXP2010134/1 R20G11
Supported: timer,100rel,histinfo,join,norefersub,replaces
Session-Expires: 1800;refresher=uas
P-Access-Network-Info: 3GPP-E-UTRAN;utran-cell-id-3gpp="460115E13675381B";access-domain=bac01.+86591.fj.ctcims.cn;network-provided
P-Charging-Vector: icid-value=sg6.psbc12ber.fz.fj.node.epc.-1608-718164-185570-946;orig-ioi=ericsson.com;term-ioi=fj.ctcims.cn;icid-generated-at=sg6.psbc12ber.fz.fj.node.epc.mnc011.mcc460.3gppnetwork.org
Supported: timer
P-Access-Network-Info: 3GPP-E-UTRAN;utran-cell-id-3gpp=4600069167E8F712;sbc-domain=sbc7.0791.100.jx.chinamobile.com;ue-ip=[local_ip];ue-port=[local_port];network-provided
P-Charging-Vector: icid-value="cdpsbc21bhw.19c.4185.20180803081658";orig-ioi=cdscscf6bhw.sc.chinamobile.com;term-ioi=jx.chinamobile.com
P-Early-Media: gated
Content-Length: [len]
Content-Type: application/sdp
P-Charging-Vector: icid-value=sg9.psbc21ber.nc.jx.node.epc.mnc0-1589-273998-798645;orig-ioi=ericsson.com;term-ioi=js.ctcims.cn
P-Early-Media: sendrecv
P-Access-Network-Info: IEEE-802.11;"location-info=5.253.86.82"

v=0
o=- 1635149026 1635149027 IN IP4 5.92.12.134
s=SBC call
c=IN IP4 [media_ip]
b=AS:41
b=RR:1837
b=RS:600
t=0 0
m=audio [media_port] RTP/AVP 8 96
b=AS:41
b=RR:1837
b=RS:600
a=rtpmap:8 PCMA/8000
a=ptime:20
a=maxptime:240
a=sendrecv
a=curr:qos local sendrecv
a=curr:qos remote sendrecv
a=des:qos mandatory local sendrecv
a=des:qos mandatory remote sendrecv
a=rtpmap:96 telephone-event/16000
a=fmtp:96 0-15
]]>
</send>
<nop>
<action>
<!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/-->
<exec play_pcap_audio="pcap/g711a.pcap"/>
</action>
</nop>
<pause milliseconds = "6000"/>
<send>
<![CDATA[

SIP/2.0 180 Ringing
Via:[$via_invite]
[last_Call-ID:]
[last_From:]
[last_To:]
CSeq: 1 INVITE
Allow: INVITE,UPDATE,BYE,PRACK,INFO,OPTIONS,CANCEL,SUBSCRIBE,ACK,REFER,NOTIFY,REGISTER,PUBLISH,MESSAGE
Call-Info: <tel:+8613665171969>;purpose=call-completion;m=NR
Contact: <sip:[local_ip]:[local_port];transport=[transport]>
P-Asserted-Identity: <tel:+8613665171969>
P-Charging-Vector: icid-value="pbac22bhw.19e.627a.20211025080346";orig-ioi=js.ims.mnc011.mcc460.3gppnetwork.org;term-ioi=js.chinamobile.com
P-Early-Media: inactive
Content-Length: 0

]]>
</send>
<pause milliseconds = "6000"/>
<send>
<![CDATA[

SIP/2.0 200 OK
Via:[$via_invite]
[last_From:]
To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number]
[last_Call-ID:]
CSeq: [$cseq_invite]
Contact: <sip:[local_ip]:[local_port];transport=[transport]>
Content-Type: application/sdp
Content-Length: [len]

v=0
o=- 1635149026 1635149027 IN IP4 5.92.12.134
s=SBC call
c=IN IP4 [media_ip]
b=AS:41
b=RR:1837
b=RS:600
t=0 0
m=audio [media_port] RTP/AVP 102 96
b=AS:41
b=RR:1837
b=RS:600
a=rtpmap:102 AMR/8000
a=fmtp:102 mode-change-capability=2;max-red=0
a=ptime:20
a=maxptime:240
a=sendrecv
a=curr:qos local sendrecv
a=curr:qos remote sendrecv
a=des:qos mandatory local sendrecv
a=des:qos mandatory remote sendrecv
a=rtpmap:96 telephone-event/16000
a=fmtp:96 0-15

]]>
</send>

<recv request="ACK"
rtd="true"
crlf="true">
</recv>
<nop>
<action>
<!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/-->
<exec play_pcap_audio="pcap/amr102.pcap"/>
</action>
</nop>
<recv request="INVITE">
</recv>
<send>
<![CDATA[

SIP/2.0 100 Trying
[last_Via:]
[last_From:]
To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number]
[last_Call-ID:]
[last_CSeq:]
Contact: <sip:[local_ip]:[local_port];transport=[transport]>
Content-Length: 0

]]>
</send>
<send>
<![CDATA[

SIP/2.0 200 OK
[last_Via:]
[last_From:]
To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number]
[last_Call-ID:]
[last_CSeq:]
Contact: <sip:[local_ip]:[local_port];transport=[transport]>
Content-Type: application/sdp
Content-Length: [len]

v=0
o=- 1635149026 1635149027 IN IP4 5.92.12.134
s=SBC call
c=IN IP4 [media_ip]
b=AS:41
b=RR:1837
b=RS:600
t=0 0
m=audio [media_port] RTP/AVP 115 96
b=AS:41
b=RR:1837
b=RS:600
a=rtpmap:115 AMR-WB/16000/1
a=fmtp:115 mode-change-capability=2;max-red=0
a=ptime:20
a=maxptime:240
a=sendrecv
a=curr:qos local sendrecv
a=curr:qos remote sendrecv
a=des:qos mandatory local sendrecv
a=des:qos mandatory remote sendrecv
a=rtpmap:96 telephone-event/16000
a=fmtp:96 0-15

]]>
</send>
<recv request="ACK">
</recv>
<nop>
<action>
<!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/-->
<exec play_pcap_audio="pcap/amrwb1_115.pcap"/>
</action>
</nop>
<recv request="BYE">
</recv>

<send>
<![CDATA[

SIP/2.0 200 OK
[last_Via:]
[last_From:]
To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number]
[last_Call-ID:]
[last_CSeq:]
Contact: <sip:[local_ip]:[local_port];transport=[transport]>
Content-Type: application/sdp
Content-Length: 0

]]>
</send>

<!-- Keep the call open for a while in case the 200 is lost to be -->
<!-- able to retransmit it if we receive the BYE again. -->
<pause milliseconds="0"/>


<!-- definition of the response time repartition table (unit is ms) -->
<ResponseTimeRepartition value="10, 20, 30, 40, 50, 100, 150, 200"/>

<!-- definition of the call length repartition table (unit is ms) -->
<CallLengthRepartition value="10, 50, 100, 500, 1000, 5000, 10000"/>

</scenario>

uas的启动脚本如下:

./sipp -sf uas_jsfhgk.xml -i 192.168.2.179 -p 8063 -trace_screen -screen_overwrite true -screen_file uas_jsfhgk.log -inf calling_number_graylist_speech_template_performance.txt

三、valrind工具

通过10 caps的连续呼叫测试,媒体服务器的故障现象开始复现,确定了代码的大致位置后。开始对媒体服务器的代码部分进行修复加锁。加锁完成后,进程不重启了,但是发现有内存泄漏,通过top命令查看,进程占用的内存一直在增长。
于是启用了valgrind工具,用valgrind启动媒体服务进程,uac单呼一次,然后分析valgrind日志。
通过valgrind的日志发现,提示调用的avcodec_send_frame存在内存泄漏,但是我查了代码,已经调用了av_frame_free释放了内存,于是我判断应该不是这个地方存在内存泄漏。通过百度查询avcodec_send_frame内存泄漏问题时,我注意到一个专家写了一句话“当av_read_frame返回值不小于0时,内部会进行缓存操作,需要外界释放。

总结

提示:这里对文章进行总结:
当媒体服务器发行故障后,先后通过分析SIP信令及媒体服务器抓包及服务器日志,定位媒体服务器网元出现故障。
通过abrt抓取进程异常的core文件,使用sipp模拟大呼进行故障场景复现。采用valgrind进行内存泄漏定位。最终定位问题并修复该问题。
希望这个抢救的案例,能够给其他同行起到借鉴作用。


举报

相关推荐

0 条评论