I have a main site in the US with two S8720s and an ESS site in the UK with two S8720s and two G650s with IPSIs connecting back to the US via an MPLS WAN.

I am having a problem getting the UK IPSIs to register and stay registered with the US servers. They are intermittently connecting then getting disconnected and going to ESS mode. When they lose connectivity I cannot ipsisession to them (even from the UK server) but I can ping them from anywhere.

I'm getting a ton of sanity failure messages:

[login to unmask email]> logc lm -t today |grep checkSlot | tail -12
20071205:024931982:8516616:pcd(22448):MED:[[0:0] checkSlot: sanity failure (1)]
20071205:024931982:8516617:pcd(22448):MED:[[0:1] checkSlot: sanity failure (2)]
20071205:024931982:8516618:pcd(22448):MED:[[0:1] checkSlot: too many standby sanity failures (2), active failures (1)]
20071205:024932002:8516623:pcd(22448):MED:[[1:0] checkSlot: sanity failure (1)]
20071205:024932002:8516624:pcd(22448):MED:[[1:1] checkSlot: sanity failure (2)]
20071205:024932002:8516625:pcd(22448):MED:[[1:1] checkSlot: too many standby sanity failures (2), active failures (1)]
20071205:024932982:8516646:pcd(22448):MED:[[0:0] checkSlot: sanity failure (2)]
20071205:024933002:8516647:pcd(22448):MED:[[1:0] checkSlot: sanity failure (2)]
20071205:024933982:8516650:pcd(22448):MED:[[0:0] checkSlot: too many sanity failures (3)]
20071205:024934002:8516655:pcd(22448):MED:[[1:0] checkSlot: too many sanity failures (3)]
20071205:024934982:8516676:pcd(22448):MED:[[0:1] checkSlot: sanity failure (1)]
20071205:024935002:8516677:pcd(22448):MED:[[1:1] checkSlot: sanity failure (1)]
[login to unmask email]>


I also cannot ipsisession to the IPSI boards from anywhere:
From UK S8720
[login to unmask email]> ipsisession -c 3a
System Error: socket operation(connect) timeout.

[login to unmask email]> ipsisession -c 3b
System Error: socket operation(connect) timeout.

From US S8720
[login to unmask email]> ipsisession -c 3a
System Error: socket operation(connect) timeout.

[login to unmask email]> ipsisession -c 3b
System Error: socket operation(connect) timeout.


But....and this is where it gets weird....I have what looks like layer 3 connectivity everywhere. IPSI 3b is 10.20.177.2

From UK S8720
[login to unmask email]> ping 10.20.177.2
PING 10.20.177.2 (10.20.177.2) 56(84) bytes of data.
64 bytes from 10.20.177.2: icmp_seq=0 ttl=64 time=0.312 ms
64 bytes from 10.20.177.2: icmp_seq=1 ttl=64 time=0.308 ms
64 bytes from 10.20.177.2: icmp_seq=2 ttl=64 time=0.311 ms

From US S8720
[login to unmask email]> ping 10.20.177.2
PING 10.20.177.2 (10.20.177.2) 56(84) bytes of data.
64 bytes from 10.20.177.2: icmp_seq=0 ttl=58 time=102 ms
64 bytes from 10.20.177.2: icmp_seq=1 ttl=58 time=113 ms
64 bytes from 10.20.177.2: icmp_seq=2 ttl=58 time=120 ms

The data switches at both sites are Cisco 3560s. If I reset one of the UK IPSI boards I get connectivity back to the US momentarily. At that time I can ipsisession successfully and that session will stay up as long as I stay connected. Once I exit out I cannot get back in.

This is all very new to me. Any help would be appreciated.

I know this is basic but, are the IPSIs and Cisco ports both locked to 100Mb Full Duplex. (Make sure you do not have a duplex mismatch)

A 'netstat -an | grep 5010' from the linux shell will show you if the sockets to the IPSIs are established or not.

I just thought of something else...

Are you running QOS across the WAN? The Cisco's should 'Trust' our IPSI QOS settings and the MPLS prioritize accordingly.

On CM in the 'ipserver-interface' form make sure you have 'Enable QOS' stroked to 'y' and set your Layer 2/3 values accordingly. Eg. 5 & 46. This will enable QOS from the server to the IPSI, however, it will NOT set it from the IPSI to the server, this must be done on the IPSI itself.

Login to the IPSIs and do a 'show qos' to see current values. If it does not match your customer's QOS policy, and if default it probably won't, then change to match..

Use 'set diffserv 46' as this is the norm and reset the IPSI. Log back in and make sure the values are correct.

Also do a 'show port 1' to make sure you are 'locked down'

Hope this helps..

Also if this connection is running through a firewall, make sure that the firewall isn't stripping the COS / QOS values when it pushes it to the MPLS network.

Looks like the sockets are established.

[login to unmask email]> netstat -an | grep 5010
tcp 0 40 10.20.206.1:34183 10.10.16.111:5010 ESTABLISHED
tcp 0 0 10.20.177.1:34033 10.20.177.4:5010 ESTABLISHED
tcp 0 40 10.20.206.1:34662 10.10.16.110:5010 ESTABLISHED
tcp 0 40 10.20.206.1:44996 10.10.16.106:5010 ESTABLISHED

I still can't "ipsisession" to them though. This is from the server on the local LAN so the MPLS network won't play a part here.

And we're locked down to 100/full.

As far as QoS goes - we're not doing it. We will in the future but this is a new site and out utilization is nill. We have zero traffic going over this link and I'm not getting any dropped packets.

What version of ACM are you running?? the distance and nature of MPLS are not optimal for IPSI traffic.. there are improvements in the communication in ACM Rel 4 and the IPSI sanity timer can be extended.. I recommend starting with 6 seconds.. but the max is 14..

DT

We're running 3.1. We had Avaya come in and increase our sanity timers to 8 seconds. This dropped our resets from 4 - 6 per day to 1 - 2 per day. Now I have zero connectivity.