S300 asterisk code behaves weird

Dear All Hi
Suddenly and after 2 years of smooth operation the S300 behaves very weird.
I have 160 user local and remote and it is been two days now that the users at first looses their registration and then they register and registered ,all time.
Finally sip register trunks looses registration too and at the very end all the system crashes. Please find attaches the ssh trace file for your help. This is the first time i see such a weird trace.
Of course i have tried to restart it many times and after that I login with ssh to monitor the asterisk code behavior.
It starts OK but gradually it behave weird and to fast. One of the strange thing is when i type help in the ssh , many of the command are missing. Particularly i have typed "pjsip set logger on" and they response i had was " no such a command". Very Very strange.
Please advise asap

here is some traces

[2019-04-23 16:20:56] WARNING[16577]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:20:57] NOTICE[16576]: res_pjsip_mwi.c:452 send_unsolicited_mwi_notify: No contacts bound to AOR 723. Cannot send unsolicited MWI until a contact registers.

[2019-04-23 16:20:58] NOTICE[16576]: res_pjsip_mwi.c:452 send_unsolicited_mwi_notify: No contacts bound to AOR 872. Cannot send unsolicited MWI until a contact registers.

[2019-04-23 16:20:58] NOTICE[684]: res_pjsip_mwi.c:452 send_unsolicited_mwi_notify: No contacts bound to AOR 619. Cannot send unsolicited MWI until a contact registers.

[2019-04-23 16:20:59] NOTICE[5109]: res_pjsip_mwi.c:452 send_unsolicited_mwi_notify: No contacts bound to AOR 611. Cannot send unsolicited MWI until a contact registers.

[2019-04-23 16:21:00] WARNING[1668]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:00] WARNING[25025]: res_pjsip/pjsip_distributor.c:590 authenticate: the ip 192.168.204.26 is blocked in firewall page to register this account

[2019-04-23 16:21:01] WARNING[16577]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:01] WARNING[25025]: res_pjsip/pjsip_distributor.c:590 authenticate: the ip 192.168.204.26 is blocked in firewall page to register this account

[2019-04-23 16:21:01] WARNING[5110]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:01] WARNING[1668]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:01] WARNING[684]: res_pjsip/pjsip_distributor.c:590 authenticate: the ip 192.168.204.26 is blocked in firewall page to register this account

[2019-04-23 16:21:02] WARNING[16577]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:02] NOTICE[5108]: res_pjsip_mwi.c:452 send_unsolicited_mwi_notify: No contacts bound to AOR 611. Cannot send unsolicited MWI until a contact registers.

[2019-04-23 16:21:02] WARNING[1668]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:03] WARNING[16577]: res_pjsip_t38.c:544 t38_incoming_invite_request: Channel PJSIP/trunk-CytaSip-endpoint-00000056 was unable to handle AST_CONTROL_T38_PARAMETERS indication, faking AST_T38_REFUSED

[2019-04-23 16:21:04] WARNING[25024]: res_pjsip/pjsip_distributor.c:590 authenticate: the ip 192.168.204.26 is blocked in firewall page to register this account

[2019-04-23 16:21:04] WARNING[16577]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:05] WARNING[1668]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:05] ERROR[2698]: res_pjsip/pjsip_configuration.c:401 persistent_endpoint_contact_deleted_observer: Unable to create ast_sip_contact_status for contact 766/sip:613@192.168.201.31:5060
[2019-04-23 16:21:05] ERROR[2698]: res_pjsip/pjsip_configuration.c:401 persistent_endpoint_contact_deleted_observer: Unable to create ast_sip_contact_status for contact 766/sip:613@192.168.201.31:5060

[2019-04-23 16:21:05] WARNING[5110]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:05] WARNING[1668]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:06] WARNING[16577]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:06] ERROR[2698]: res_pjsip/pjsip_configuration.c:401 persistent_endpoint_contact_deleted_observer: Unable to create ast_sip_contact_status for contact 766/sip:613@192.168.201.31:5060

[2019-04-23 16:21:07] WARNING[5110]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:07] WARNING[1668]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:07] WARNING[5112]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:08] WARNING[2698]: res_pjsip_registrar_expire.c:199 contact_expiration_observer_deleted: Unable to cancel schedule ID 98. This is probably a bug (res_pjsip_registrar_expire.c: contact_expiration_observer_deleted, line 199).
[2019-04-23 16:21:08] ERROR[2698]: astobj2.c:129 INTERNAL_OBJ: bad magic number for object 0xbc5e08c. Object is likely destroyed.

[2019-04-23 16:21:08] WARNING[1668]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:08] WARNING[16576]: res_pjsip/pjsip_distributor.c:590 authenticate: the ip 192.168.204.26 is blocked in firewall page to register this account

[2019-04-23 16:21:08] WARNING[16577]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

[2019-04-23 16:21:09] WARNING[684]: res_pjsip/pjsip_distributor.c:590 authenticate: the ip 192.168.204.26 is blocked in firewall page to register this account

[2019-04-23 16:21:09] NOTICE[5108]: res_pjsip_mwi.c:452 send_unsolicited_mwi_notify: No contacts bound to AOR 775. Cannot send unsolicited MWI until a contact registers.

[2019-04-23 16:21:09] WARNING[1668]: pjsip:0 <?>: sip_transactio Unable to register REGISTER transaction (key exists)

Larry Neblett May 10, 2019 12:41

I can only offer the following:

1. What and why is this IP (192.168.204.26) being blocked? It is seen quite often over the course of the 13 seconds of the log.

2. You have other IPs that the system sees that are also having issues, why?

I am wondering of the number of requests and messages that are being block is causing any kind of a CPU utilization issue such that the processor can't keep up and crashes?

Of course, 13 seconds is such a small period that there is no telling what may be going on and I suppose that you may also be getting attacked from the outside as well.

I suggest:

1. You find out and resolve the issues with the local IPs that show in the log.

2. Do a network capture of the both the LAN and WAN interfaces and examine the traffic to see what else may be going on.

3. If possible disconnect the WAN (external) for a bit and see if the situation continues or if it gets better (I know this may not be possible).

4. I think you need to be certain that the PBX is not being overwhelmed by the number of requests/devices that may be messaging it to insure that the CPU is not being overloaded. Hackers can be relentless in their effort of trying to get in to the point that it becomes a DOS attack and at some point the system could crash. This would oftentimes also manifest itself in slow response times, long connect times or poor audio if using SIP. A WAN capture would show the hacker effort whereupon you could implement some firewall rules to mitigate. Keep in mind that while the device has some protective measures of its own, it has to spend the CPU cycles to do so, which in my opinion is better spent by a decent firewall so the PBX can do its job rather than having to defend itself.

5. Submit a ticket to Yeastar support. There may be something else that is not shown in your log which.

Riddhe M May 14, 2019 06:11

The PJSIP stack seems crashed at that moment. It's better to contact Yeastar support check further. It's critical. As I experienced, you'd better to offer the System Log enabled with debug option and report the time of issue. They would check the core file when crashed. Hope this might help.

S300 asterisk code behaves weird

2 comments