Ghost Solution Suite

Since September, we have been using Ghost 11.5 upgraded from Ghost 8.2, ICMP is enabled on all our switches but when we do a Multicas session it seems to flood our network, and people elsewhere in the college at which I work lose network connectivity. If we stop the ghost session all goes back to normal.

 

When we do a unicast the flooding does not occur.

 

What our some suggestions we can try to help us troubleshoot this issue?

One thing you can do from the GhostCast server is limit the data throughput for a ghostcast session.  This would keep your ghost server from overwhelming your network with traffic.

---

ICMP is enabled on all our switches

---

I realise this is probably a typo, but just to be clear ICMP is the Internet Control Message Protocol, and has nothing to do with multicasting.

IGMP is the Internet Group Management Protocol, and is the one that controls multicasting. It's actually a routing protocol, and switches generally use a thing called "IGMP snooping"; the IGMP traffic they "snoop" requires a router (or a routing module in a switch) to be generating IGMP queries so the switches can monitor the IGMP protocol traffic. Multicast across subnets is a routing function.

If you are getting this flooding in a switched environment, it is always the switches which are responsible.

Most importantly, what specific switches are you using? Most low-end unmanaged switches do this kind of "flooding" because entry-level consumer equipment doesn't understand multicast at all and treat all multicast frames as broadcast. They can't be enabled for IGMP, and so just flood.

If you do have managed switches, it would help if you can describe exactly what settings you have applied to them, because there may be additional settings you need.

Hi Nigel, thanks for your reply. I work with RobJCW, and I'm the one responsible (inherited) for the hardware of the network.

 

So first off, yes, it was a type. We are using IGMP to manage the multicast groups.

 

Second, we are using a router--an HP ProCurve 9315 with the following modules; J4895A, J4894A, 2 J4885A--to generate the IGMP queries.

 

Third, we are definitely not using low-end consumer switches. All our switches are HP ProCurve switches, and we are using an Hierarchal architecture. 

 

From the router we go to an HP 5412zl switch. From there we go to our various wiring closets throughout the college. In the wiring closets, for the most part, we are using HP 4108gl or 4000m switches to distribute to the various labs. However, depending on the size of the labs or classrooms, we occassionaly use a 2524A or a 2650 in the wiring closet. Then from the wiring closet, we either go straight to the PCs, or many of our labs have switches in the labs themselves (all HP 2524A in the labs).

 

We are vlanned, and all these labs fall on our 172 vlan. IGMP is turned on for all the switches involved. If there's any more info you need let me know.

 

But ultimately what happens is that when we run a ghost session, our networked is saturated/flooded and brings our network to a crawl, and some areas even "lose" connectivity because the ghost is saturating their uplink port. We have gigabit connection between most of the main closets and our core switch. We even have trunked (link aggregation) the most critical path to 2 gigabit, and it still seems to make no difference. Are we missing a switch for Ghost? or is it some obsure setting on one of our switches?

 

Any help is appreciated.

 

---

we are using an Hierarchal architecture.

---

By this do you mean specifically that there are no redundant links between leaf switches of any kind (i.e., that you have physically configured things so that you do not need to employing any spanning-tree techniques?).

[ VLANs and spanning trees can interact in complex ways, such as described here. ]

At which level in this tree is this "flooding" occurring? Is this occurring at the root (i.e, is the 5412 sending tagged frames out down to every switch below it) or primarily at the leaves (i.e., underneath the leaf switches in each lab)?

What traffic inspection have you done? For instance, if you have an untagged ports configured as a member of this particular VLAN under the 5412 at the root, what do you see when you capture traffic on that port? Do you see the IGMP queries coming from the 9315 (which will have the 9315's IP address) and is the imaging traffic from the Ghost server being filtered or is that also being forwarded?

Have you also tried doing some packet capture at the leaves to see that only the 9315 is acting as querier on this VLAN and some other device has not managed to get itself elected as the querier?

---

We are vlanned, and all these labs fall on our 172 vlan

---

"172 VLAN", presumably you mean a VLAN on which you have configured IPs like 172.1.0.0/16 - and you configured the tag for this particular VLAN such that it is spanning your entire switch tree - including both the 9315 which is also acting as the router, and the 5412 which is at the root of the subtree under it which connects the labs?

---

IGMP is turned on for all the switches involved

---

By this do you specifically mean to say that IGMP snooping is on for this particular VLAN on every switch?

Bear in mind that each individual VLAN configured in each switch will likely need the IGMP snooping set for the specific VLAN ID on that switch; setting IGMP enabled for the default VLAN will generally not affect other VLANs. See for instance the guide to IGMP for the 5400 series.

It would be good to be able to see the result of "show ip igmp config" on every one of the switches to verify that a) snooping is in fact on for the VLAN id, and b) the state of IGMP querying in each to consider whether perhaps the IGMP querier election process is an issue.

---

But ultimately what happens is that when we run a ghost session, our networked is saturated/flooded and brings our network to a crawl, and some areas even "lose" connectivity because the ghost is saturating their uplink port.

---

Where in your tree of switches - specifically, to what port on what switch assigned to what VLAN id and with what IP - is the Ghost server located?

Then, what is the specific traffic being sent upstream that is saturating the links? Only the GhostCast server should be sending multicast frames, so the configuration of your switches should be such that this traffic would only be sent downwards from the root to the leaf switches, unless perhaps you have some kind of spanning-tree problem whereby the switches are configured with multiple links causing a loop which is causing them to circulate the traffic (i.e., the kind of problem that spanning-tree protocols are designed to solve).

It may also be a good idea to gather some of the IGMP-related statistics and counters from each of the switches: for instance there is a similar problem described here where it appears that due to querier election the switches are making bad decisions about which ports to forward traffic to.

hmmm... You've given me a lot to chew on, but let me quickly answer a few of your questions.

If I don't answer one of your questions immediately here, it's probably because I'm still thinking

about or I just don't know yet.

 

 

---

---

We are using an Hierarchal architecture.

---

By this do you mean specifically that there are no redundant links between leaf switches of any kind (i.e.., that you have physically configured things so that you do not need to employing any spanning-tree techniques?).

---

 

We are not using spanning tree anywhere in our network in any way. And no redundant links anywhere. So if something goes down... It's down.

 

---

---

We are vlanned, and all these labs fall on our 172 vlan

---

"172 VLAN", presumably you mean a VLAN on which you have configured IPs like 172.1.0.0/16 - and you configured the tag for this particular VLAN such that it is spanning your entire switch tree - including both the 9315 which is also acting as the router, and the 5412 which is at the root of the subtree under it which connects the labs?

---

Our 172 VLAN is configured to have the 10.172.0.0/16 network address. And yes, the tags have been configured from the router down to every single switch in the tree (Tagged on the ports it needs to be tagged and Untagged where it needs to be untagged)

 

 

---

---

IGMP is turned on for all the switches involved

---

By this do you specifically mean to say that IGMP snooping is on for this particular VLAN on every switch?

---

Yes. In fact this is the ONLY VLAN we have it enabled.

 

 

---

 It may also be a good idea to gather some of the IGMP-related statistics and counters from each of the switches: for instance there is a similar problem described here where it appears that due to querier election the switches are making bad decisions about which ports to forward traffic to.

---

This one intrigued me. The problem described in that HP forum sounds an awful like our problem. The area with the biggest issue of "flooding" was between 2 4108gl switches. I will be investigating this and I will let you know.

 

Thanks again for your help so far.

 

---

This one intrigued me. The problem described in that HP forum sounds an awful like our problem. The area with the biggest issue of "flooding" was between 2 4108gl switches. I will be investigating this and I will let you know.

---

Then let me just explain what was going on there, with the caveat that IGMP snooping is very much not a standard part of the IETF Internet architecture, and that this is all up to switch vendors.

IGMP querying - which is a basic part of IGMP - contains a specification for limiting the number of IGMP queriers on a network, on the basis that such queries are (in the standard IETF architecture) generated by routers. Multiple IGMP queriers would be redundant since if multiple routers are present, it is the nature of the IETF architecture that those routers will be enchanging route information directly amongst each other, so the election process for queriers is used to reduce the volume of query (and thus host-generated response) traffic.

IGMP snooping typically does two things: first, and most importantly, the switches inspect the query responses from hosts, and thereby dynamically learn associations between switch ports and multicast groups so that it need only distribute frames to ports on subscribing hosts.

The second thing, however, is that in the absence of any other configuration, an incoming IGMP query on a switch port may be used by the switch to infer that that port is the location of a multicast-aware router. Since any particular piece of multicast traffic may likely need routing, such a switch will treat that port as one to which it will always forward all multicast traffic that comes from any other port.

In the article which I linked, the customer describes a non-routed network. However, in order to snoop IGMP there needs to be a querier - and the switch that wins the IGMP query election process will by default be treated by other switches as a router, and will always receive all multicast traffic because of that.

If you have query generation enabled in a leaf switch and that leaf switch wins the election process, that leaf will end up as a sink for all the multicast traffic. For this reason, it is basic practice with IGMP snooping to ensure that the only devices capable of winning the standard IGMP querier election process are devices located and provisioned such that they can be a sink for all multicast traffic on the attached layer 2 subnet.