One of my past employers had a crazy simple networking philosophy. It killed me. He once had his provider shut the port facing his core switch down, due to STP BPDUs being transmitted. To remedy this, instead of disabling the feature on the switch, he replaced it with an unmanaged switch that doesnt speak STP.
That's not great, but it's better than the opposite. I've been in networking for ages and have observed that most networking people will, as a rule, make networks as complicated as possible.
Why have one layer of NAT when you can have four or five? Why not invent a bespoke addressing scheme? Why not cargo cult the documentation and/or scripts and config files from Stack Overflow or ChatGPT?
Under-engineering is an easier problem to solve than over-engineering.
He had a particular niche that had few requirements on complex networking, and a business plan that was so brain dead simple it could be achieved easily.
But yeah, really nice guy but unable to expand beyond that paradigm.
I once helped him deal with a fibre ring, where there were 5 sites in the ring, but only 2 internet services. So sites 3, 4 and 5 had to communicate via sites 1 and 2.
His team had put a sophos firewall, with NAT, and no routing, on every fibre connection and called me when it didnt work.
I’m still a bit unclear about how an IX is situated relative to the internet and to end users. Per the article, it’s not meant to have desktops or print queues plugged into it, but it’s also a LAN. What sort of computer is meant to participate in an IX?
The idea of an IX, or IX peering LAN is simple in concept. It is a LAN (a flat, layer2 network), to which multiple ISP's can plug in routers.
Like your home LAN might have 192.168.0.1 = router, 192.168.0.2 = laptop, 192.168.0.3 = phone etc, a peering LAN will have things like 195.66.224.21 = HurricaneElectric, 195.66.224.22 = NTLI, 195.66.224.31 = Akamai, 195.66.224.48 = Arelion etc ...
So instead of all these ISP's that want to exchange traffic with each other having to assign ports and run cables in a full mesh (which quickly would get out of control), everyone connects to the "big switch in the middle" with that peering LAN on it, and they use that.
Back in the day, that might have been an actual single big switch, or a stack of switches. Now IXP infrastructures are much more complex, but the presentation to the end user is usually still a cable (or bundle of cables) that goes into something that looks to them like a "big switch".
There is a LOT more to know about this space (Peering vs Transit, PNI's, L3 internet exchanges, what Google are doing by withdrawing from IXP's), but I wanted to write a comment that didn't turn into an essay.
An IX is the internet, or at least a small part of it. It's sort of a network where each device shares their "routes" with the others, and then propagates those routes back through their own network. They're exchanging their respective pieces of the Internet with each other.
The devices themselves are just routers, though much larger and more complex than what you'd see in a house or office. Instead of one route, "put all the traffic through eth0", they'll have hundreds, thousands, or millions of routes depending on their location relative to the "rest of the internet".
It's a big peering LAN (not dissimilar to a LAN party, but you know, with equipment costing more than a nice car), scaling from a switch in a rack somewhere (eg. FCIX [1]) up to hundreds of switches in dozen of locations not even in the same city anymore (eg. DECIX, AMSIX).
You connect to the IX either over a cross connect cable from some equipment nearby (in the same DC or same building) or across town via some leased line / dark fiber / lambda / etc. But usually what's at the other end of a connection to an IX is your edge router, on which you will then run BGP to all the other folks on the IX.
However, it's not always this simple - for example, you might have another switch-like device in between the router and the IX, to maintain some level of flexibility for your own services, or because you're actually being bundled together alongside multiple customers into IX access by some provider. You might actually go into a whole MPLS backbone first, because that's how your provider is selling you transport to the IX. Or you might've set up some peering LAN bridging on your router to set up some hot standby and then plugged it into some switch for convenience to run it across the office LAN to your desk and then...
What ends up happening, is with this amount of complex network devices along the way, and with how network equipment is generally provisioned (you SSH into it and then you do some mutable changes, then you remember the update the orga docs), mistakes happen.
In a critical case of misconfiguration (also stale configuration and VLAN identifier reuse) it's not unimaginable for an IX peering LAN to then get accidentally bridged into some VLAN that actually reaches an old Windows laptop that was at some point plagged into another VLAN for troubleshooting. This is especially likely for customers that co-locate their office equipment, AD servers, web servers and edge router in the same building across the same infrastructure.
Ideally, participants in an internet exchange purely exchange routing information, usually using the Border Gateway Protocol (BGP). This implies that devices connected to an internet exchange are routers. These devices receive routes for other autonomous systems (AS), and (selectively) publish their routes to other parties on the internet exchange.
(Internet exchanges typically offer a route server, such that every participant of the IX can easily publish routes for other participants, and simultaneously receive published routes of all other participants.)
The _effect_ of exchanging routing information is that IX-local participants know where to send traffic destined for certain IP ranges from other participants.
An autonomous system internally "knows" where each of their routers are located, and all these routers are typically connected with each other. When several routers of an AS are connected on different IXes, this means they can take informed decisions on where to send traffic destined for other ASes. It could be that AS 64496 is only present in IX-A, while AS 64510 is only present at IX-B. Suppose AS 64499 is connected to both IX-A and IX-B, traffic sourced from AS 64499 (e.g. endusers or "eyeballs") and destined for either 64496 or 64510 knows, through internally exchanged routes, where to send that traffic.
Scale this to even more autonomous systems and IXes in different geographic reasons, and you'll find it becomes a network of networks, or: the internet.
In an IX you have mostly routers, firewalls and servers. But sometimes people will need to connect a "personal" device to the internet or the internal servers network, and instead of using a NATed/firewalled zone, they will use a public internet port (mostly by mistake), and then that kind of stuff can happen.
I remember in the 2000s a large-ish Telco network in the US was running ospf on an IX. A few of us on IRC did the what if? And one of us brought up the adjacency and it worked.
Same network also had all their network links in MRTG public too with no auth - if you only knew the hostname/URL you could see it all (which their staff would sometimes drop in Noc communication when linking a graph and you attempted to go there to poke around).
One of my past employers had a crazy simple networking philosophy. It killed me. He once had his provider shut the port facing his core switch down, due to STP BPDUs being transmitted. To remedy this, instead of disabling the feature on the switch, he replaced it with an unmanaged switch that doesnt speak STP.
That's not great, but it's better than the opposite. I've been in networking for ages and have observed that most networking people will, as a rule, make networks as complicated as possible.
Why have one layer of NAT when you can have four or five? Why not invent a bespoke addressing scheme? Why not cargo cult the documentation and/or scripts and config files from Stack Overflow or ChatGPT?
Under-engineering is an easier problem to solve than over-engineering.
He had no business managing a network.
He had a particular niche that had few requirements on complex networking, and a business plan that was so brain dead simple it could be achieved easily.
But yeah, really nice guy but unable to expand beyond that paradigm.
I once helped him deal with a fibre ring, where there were 5 sites in the ring, but only 2 internet services. So sites 3, 4 and 5 had to communicate via sites 1 and 2.
His team had put a sophos firewall, with NAT, and no routing, on every fibre connection and called me when it didnt work.
I’m still a bit unclear about how an IX is situated relative to the internet and to end users. Per the article, it’s not meant to have desktops or print queues plugged into it, but it’s also a LAN. What sort of computer is meant to participate in an IX?
The idea of an IX, or IX peering LAN is simple in concept. It is a LAN (a flat, layer2 network), to which multiple ISP's can plug in routers.
Like your home LAN might have 192.168.0.1 = router, 192.168.0.2 = laptop, 192.168.0.3 = phone etc, a peering LAN will have things like 195.66.224.21 = HurricaneElectric, 195.66.224.22 = NTLI, 195.66.224.31 = Akamai, 195.66.224.48 = Arelion etc ...
So instead of all these ISP's that want to exchange traffic with each other having to assign ports and run cables in a full mesh (which quickly would get out of control), everyone connects to the "big switch in the middle" with that peering LAN on it, and they use that.
Back in the day, that might have been an actual single big switch, or a stack of switches. Now IXP infrastructures are much more complex, but the presentation to the end user is usually still a cable (or bundle of cables) that goes into something that looks to them like a "big switch".
There is a LOT more to know about this space (Peering vs Transit, PNI's, L3 internet exchanges, what Google are doing by withdrawing from IXP's), but I wanted to write a comment that didn't turn into an essay.
An IX is the internet, or at least a small part of it. It's sort of a network where each device shares their "routes" with the others, and then propagates those routes back through their own network. They're exchanging their respective pieces of the Internet with each other.
The devices themselves are just routers, though much larger and more complex than what you'd see in a house or office. Instead of one route, "put all the traffic through eth0", they'll have hundreds, thousands, or millions of routes depending on their location relative to the "rest of the internet".
It's a big peering LAN (not dissimilar to a LAN party, but you know, with equipment costing more than a nice car), scaling from a switch in a rack somewhere (eg. FCIX [1]) up to hundreds of switches in dozen of locations not even in the same city anymore (eg. DECIX, AMSIX).
You connect to the IX either over a cross connect cable from some equipment nearby (in the same DC or same building) or across town via some leased line / dark fiber / lambda / etc. But usually what's at the other end of a connection to an IX is your edge router, on which you will then run BGP to all the other folks on the IX.
However, it's not always this simple - for example, you might have another switch-like device in between the router and the IX, to maintain some level of flexibility for your own services, or because you're actually being bundled together alongside multiple customers into IX access by some provider. You might actually go into a whole MPLS backbone first, because that's how your provider is selling you transport to the IX. Or you might've set up some peering LAN bridging on your router to set up some hot standby and then plugged it into some switch for convenience to run it across the office LAN to your desk and then...
What ends up happening, is with this amount of complex network devices along the way, and with how network equipment is generally provisioned (you SSH into it and then you do some mutable changes, then you remember the update the orga docs), mistakes happen.
In a critical case of misconfiguration (also stale configuration and VLAN identifier reuse) it's not unimaginable for an IX peering LAN to then get accidentally bridged into some VLAN that actually reaches an old Windows laptop that was at some point plagged into another VLAN for troubleshooting. This is especially likely for customers that co-locate their office equipment, AD servers, web servers and edge router in the same building across the same infrastructure.
[1] - FCIX is also a good source of how this sort of stuff actually looks like IRL, instead of the bullshit marketing 'rack of neat racks' renders on providers' websites: https://pbs.twimg.com/media/FIsd-JsVgAQLUh9.jpg?name=orig https://pbs.twimg.com/media/FP72xaiWQAsyPUM.jpg?name=orig
Ideally, participants in an internet exchange purely exchange routing information, usually using the Border Gateway Protocol (BGP). This implies that devices connected to an internet exchange are routers. These devices receive routes for other autonomous systems (AS), and (selectively) publish their routes to other parties on the internet exchange.
(Internet exchanges typically offer a route server, such that every participant of the IX can easily publish routes for other participants, and simultaneously receive published routes of all other participants.)
The _effect_ of exchanging routing information is that IX-local participants know where to send traffic destined for certain IP ranges from other participants.
An autonomous system internally "knows" where each of their routers are located, and all these routers are typically connected with each other. When several routers of an AS are connected on different IXes, this means they can take informed decisions on where to send traffic destined for other ASes. It could be that AS 64496 is only present in IX-A, while AS 64510 is only present at IX-B. Suppose AS 64499 is connected to both IX-A and IX-B, traffic sourced from AS 64499 (e.g. endusers or "eyeballs") and destined for either 64496 or 64510 knows, through internally exchanged routes, where to send that traffic.
Scale this to even more autonomous systems and IXes in different geographic reasons, and you'll find it becomes a network of networks, or: the internet.
In an IX you have mostly routers, firewalls and servers. But sometimes people will need to connect a "personal" device to the internet or the internal servers network, and instead of using a NATed/firewalled zone, they will use a public internet port (mostly by mistake), and then that kind of stuff can happen.
https://en.wikipedia.org/wiki/Internet_exchange_point
it's a network for ISPs and organizations with big networks to hook together to let traffic flow more directly, mostly only routers talking BGP
I wish everyone used LLDP everywhere: it is harmless and immensely helps in finding the correct spaghetti in the plate.
I remember in the 2000s a large-ish Telco network in the US was running ospf on an IX. A few of us on IRC did the what if? And one of us brought up the adjacency and it worked.
Same network also had all their network links in MRTG public too with no auth - if you only knew the hostname/URL you could see it all (which their staff would sometimes drop in Noc communication when linking a graph and you attempted to go there to poke around).
I wish they didn’t defacto sort least-interesting-first :s