These are unedited transcripts and may contain errors.
EIX, 2nd session ?? 26 September 2012 ?? 11 a.m.
CHAIR: Good morning everybody, welcome back. Welcome to the newcomers. Okay. We have got a fairly full packed session for the second half and just a quick reminder if anybody wants to do a 30 second ISP update, please upload a single contact slide to the presentations archive.
So, first up, we have got a quick update on where we are with the IXP switch list and Harald from University of Vienna, is going to do that.
HARALD MICHL: Good morning everybody. I am Harold Michl. I work for the Vienna Internet Exchange and give you a quick update of what has been done and what may be could be on this switch list list.
The most important news is the new list is online. And you can see it here on the EIX Working Group web page. There is the current new wish list and the whole one is here as well but I think it will be removed soon because we should concentrate only to have one document that is maintained.
It went online on the 23rd August, which was quite a while after the RIPE meeting in Ljubljana, and currently we don't know exactly why it lasted so long, maybe just because it wasn't updated for seven years or so, it was burned on an E prom instead, but we now know the way to update it very quickly and there will be some changes that are based on responses I got after the RIPE Meeting in Ljubljana. Just some minor comments.
About what next, I asked myself the question what is this document for? And of course it's, it includes items that IXPs expect from vendors that they offer these kinds of functionality and this results to, if you are get in contact with a vendor, you say, okay, look at this list and tell me what you can and what you can't and various status regarding this list. This leads to a situation where you have no defined way of asking vendors and you get several different answers and it's hard to compare that.
That is a reason why I thought about this one here. What if we include kind of a spreadsheet or at the table also a document where we can send a standardized set of questions to vendors and they are able to answer these in a standardized way. This would make it easier for us to only maintain this document once and just send it to the vendors and for the vendors it should be easier, because if they have filled out this table once for a certain product line, they just can send it to all of those who are interested in this.
The format chosen for my opinion, should be a compatible format independent of operating systems you use, and it should be from the structures or from the order of the questions, it should be related to the switch wish list itself, whereas I think it makes just sense to include it within the document.
Including also the last point here, as soon as you have a number of files higher than one, you end up with a mess with different versions, so I searched for a solution where you can have still only one document but it includes whatever it interested and necessary to complete all the information. So, it's very easy to attach other files to a PDF, so this was me, my proposal to attach this kind of table to the already existing document, so if you download the PDF you get the table automatically and you can then take it edit it and for example, send it to the IXP of your choice and tell them what your product can do.
So, with this method, we would have always the appropriate table to the appropriate version of the wish list, because it's only one file.
The table itself could be just some questions here. The number of ports, power and whatsoever, so everything of the wish list could be covered and you send the empty list to the vendor and the vendor you like just answers it and fills out this right things here, maybe one colour maybe even more columns, the more product lines and if you want to compare afterwards what each vendor is able to do, you just have to copy this columns, copy them side by sigh and you are able to very easily compare this.
So, now, I need some feedback from the community because of course, creating the table is some kind of work and if nobody is intending to use, it I wouldn't do it. If you think that's a useful thing and I can make use of it, I will take it and send it to my vendor, I like, then it makes sense to make it, to what do you think? By the way I am standing here but I am not the only author this document, there is always mike and Martin who are responsible for the contents. So it's not only me.
So I guess no feedback means not useful? That's also an answer. Thank you.
(Applause)
CHAIR: Thank you Harald. Okay, next up is Bijal, and after that it will be Andre.
BIJAL SHANGHANI: Thanks Fergus. Right. So I'm not going to tell you all what EURO?IX is because you should all know. At least no one yet has asked me what locations I'm at or how much a 10 gig port costs.
So, we have the latest numbers that we have got 65 an affiliated IXPs, 51 are in the Europe region, 14 from the rest of the world. And our latest members to join are from Albania, Johannesburg and Mozambique.
We have 11 affilliated patrons as well.
So, a little bit about the past. The first IXPs appeared around the 1990s and mostly from academic and research institution. They still are today a lot of them are still from academic and research institutes. The first commercial IXPs we have seen around 2000 and 2001 and between 2001 and 2003 we saw a really big jump and the establishment of 42 IXPs.
So, this is the situation as we see it now in Europe. There is 103 known IXPs, and that's in 42 countries out of 50, which are in Europe, and 118 cities. The green dots are the ones that are already affiliated with EURO?IX and the red ones are the ones that I am chasing.
This is the EURO?IX membership traffic growth in gigabits and this is also, it's important to note here that this is actually the, offer the public peering LAN, there is lots of peering going on privately as well and that's not included in this graph.
So, as I mentioned we have a number of members from other parts of the world. In Africa we have 4 members. Three of them are twinned. EURO?IX runs a twinning programme which is there to help with IXPs, new IXPs in different parts of the world who need help, who need support, and we have twinning in Africa with TIX, that's in Tanzania, with DE?CIX and NetNod twins with Kenya Internet Exchange and the latest one is Mozambique.
In Asia we have five members and one of those is a twin and again that's with DECIX and that's with the Nepal Internet Exchange. We have two members from Latin America and the Caribbean and four members from the USA.
With the number of the different Internet Exchanges showing an interest in EURO?IX and the different regions felt that they wanted to go off and form their own Internet Exchange point association. So, for a couple of years now, there is been LAC?IX and AP?IX and since the meeting in Africa about a month ago I think, they formed AF?IX, which is the African Internet Exchange point association.
So, we, together, with LAC?IX and AP?IX at the stage, that's still very, very early, thought it would be a good idea to form a Federation of Internet Exchanges. So, we sent out the MoU to the membership and also to the other Internet Exchange Point Associations and we have got agreement from all of them and we will be signing the MoU in the next EURO?IX forum which is in Stockholm and that will be the beginning of the Internet Exchange point federation.
And the idea behind this is to have a single global database, and IXP database, to automate data collection and perhaps look at setting some standards within the Internet Exchange point.
So, a little bit about the IXPs around the world. This is the LAC?IX region. They have 42 known IXPs, there are Internet Exchanges that are out there that you know we are still discovering every day even in Europe surprisingly, but in other regions as well, it's slightly harder to detect but as we go around and we talk to people, we discover new ones all the time which is good. So these numbers are growing and up and down all the time. But we're trying to keep a directory and make sure that everything is kept updated.
So, in the Latin America, there are 42 known IXPs. In 14 countries, and that's in 37 cities. The blue dots are the ones that are affiliated already with EURO?IX and with LAC?IX. The yellow dots are the ones that are already affiliated with just LAC?IX and the red ones are the ones that are currently not affiliated at all.
This is the, again the known LAC?IX region traffic growth and in the last 12 months they have seen a 47 percent growth in traffic, which is quite a lot.
In Asia, so, this is the AP?IX region and we have 8 known IXPs, in 20 countries, which is 40 countries cover the Asia continent and that's in 86 cities. Again, the blue dots are EURO?IX affiliated. The orange runs are AP?IX affiliated and the red ones are still yet to be affiliated.
So what's interesting here is I'm not really sure what happened. I will need to speak to my Asia IXPs, but they have actually seen a slight dip in traffic over the last 12 months, which is unusual, because we are so used to seeing everything going up and to the right. So, I will need to do a little bit of research and find out what's going on in Asia, but there has been a slight decrease there.
And this is Africa. 25 known IXPs in Africa in 18 countries and that's in 24 cities. So, there is already quite a number of IXPs in Africa, but there is still some work to be done there. The green ones are the ones that are EURO?IX affiliated. The red ones again are not and there are a few purple dots in there as well because we are still trying to figure out if they exist or if they are operational, and actually just what's going on with at the moment.
And there is the growth in the traffic in Africa, which is also doubled in the last 12 months.
So what else does EURO?IX do? We have two forums a year, and this is really is the key thing with EURO?IX, at the last forum which was in Amsterdam we had 120 attendees and that was from 37 different IXPs, so, it's a real good opportunity for the Internet Exchanges to get together and share ideas and experience and talk about what's going on in the industry. We maintain a website, we have a database, and we have tools.
We have different mailing lists for the members, ranging from technical to regulatory ones.
We have Working Groups and taskforces, we have set up a data collection task force and the idea behind this is trying to, really, set some ?? well, trying to ?? first of all, automate how we collect data from the IXPs, but also trying to look at what information we want to collect from the IXPs, what is important, what is interesting, and the task force is working on things like that.
I have a board and I have four of my board members here in the audience, Andrej, Caras, John and Joab, three of them are not here, but you can talk to them about EURO?IX I'm sure.
I have a Programme Committee which helps put the programme together which is, which I couldn't do without them because, you know, like I said, the EURO?IX forums are really key to what EURO?IX does.
We also have a benchmarking club, which is done on a volunteer basis once a year, and we work on other projects as well, like, what Mirjam talked about earlier like the sports things, so, we like to have a little bit of fun in there too.
And we have an annual report which is on IXPs, and the one for 2011 was released about six weeks ago, and here are some of the results from there.
So what we look at in the EURO?IX database as well and this is just within the membership, is we look at what switches, routers, route systems, operating systems are being used within the membership. So, here we have got brocade and Cisco still have the large share, market share of the switch router vendors and for the first time actually we have got Juniper in there and that's due to the LINX migration last year.
So this is the different route server operating systems in use within the EURO?IX membership. And the route server, demons in use. Again we have seen a really big jump to bird, which Andre is going to talk about a bit later. And if you want to know more, take a look at the website, you'll find details of all the members, the affiliated IXPs, what we're doing around the world, information on the federation, lots of documentation, and of of course the tools. And you can also follow EURO?IX on Twitter.
Thank you. Are there any questions?
CHAIR: No questions? Okay. Thank you very much Bijal.
(Applause)
CHAIR: Next up is Ondrej who is going to give us an update on BIRD and then we have got an update on Quagga.
ONDREJ FILIP: Hello everyone, I am Ondrej Philip from NIX.CZ but.
IPv4 importantly one of the BIRD developers and I'd like give you a short update since what has happened since my last update, I think it was in Vienna. I have a couple of topics to cover.
First of all, we have some more deployments, we have some new exchange points that deploy BIRD and more importantly, we introduced some new BGP related features and I will talk just about the key features not just about the BGP related because you are probably not interested in the CPS stuff in this Working Group.
Which is protocol at the moment plates, R OA, import and export route limits and the biggest thing is the secondary route export feature Tore possibility.
In Vienna a talked about the extended community that was the version 1.34 and now we have 1.38. So we get, you know, for more versions.
So we have some more deployments, and that used to be my favourite page but now I have problems to fit all the logos in that, so, I think last largest IXP is apples items who implemented bird and we are one member from non IXP world which is NetFlix that uses BIRD.
So, and Bijal I think explained the situation in the field very well so you can check the EURO?IX report and see how well developed BIRD S
Now back to the new features. First of all the route origin support. It's, we tried to help people that are really willing to work on RPKI stuff, but that's not the only use of this. Because now you can define in the configuration file the ROA table, but this table can be dynamically filed using common line interface in case you have some row bot that collects this information and you want to fill online, with BIRD you can do that. That's quite handy. And it can be used for RPK but also for the IR db, it's up to you how you want to use this feature. This ROA table and then you have more than one can be matched with the function ROA check and that gives you three answers, whether the route is now in the ROA table Tore is valid according to the table or is invalid. So, how you decide to work with this stuff, it's up to you, and again, it can be used for RPK but also for the regular filtering and it's pretty quick and efficient. So. If you are interested, you might use it.
Another thing we introduced route limits. Now you can limit the number of routes that are imported to the protocol, so, which you receive from the peers. Also you can limit the number of ROAs that you are sending to appear which is probably some case that there is no, ma'am problem and you want to know about it. So that's why there are four times of reaction that is BIRD can take in this case. It can either send you a warning just to the syslog message and ask for your reaction. Or it can block the ROAs above the limit so, in this case, you have just the certain amount of routes you defined and nothing else, no more. Or it can restart the protocol or it can just disable it completely, and waits for the operators to either enable it or change the configuration.
So, that's another feature you may like.
And one which was requested by many people I think, even in this room, was about protocol templates. We introduced a system how to make at the moment plates for the protocols, so you defined some template for the group of protocols, and all the items in the template are default for the protocols real that inherited from this template. So, this is, for example, for AS112. DS Anycast engine, we improperty things like that. And then the protocol, you know, peering with a router in NIX.CZ inherits all these items, all these things, but it needs higher limit to route limit so this can be over written. So a very handy makes the configuration much nicer really, much simpler so, very good if you added those configuration files manually especially.
And now, one thing about the secondary options to the BGP protocol. Let me explain the problem with route servers.
In normal routing the filters match the best route only. Nothing else. So when the best route comes, the U DE filters are triggered and they decide whether the exit the route and continue sends to the routing protocol like BGP for example. If the best route is denied, no other exported, that's a problem. The problem is simple when the route server has two peers and behind those pro peers is a single customer. So, the router has two paths to the customer. And another peer of the route server decides that he accepts only ?? so to make it very easy, we have customer X and path to the route server through A and B through two ISPs and if I decide not to buy routes from ISP?A and I'm just peering with ISP?B but unfortunately the path through ISP?A is the best, I don't see this customer to it's unreachable to me I have to buy for example, from transit which is all a bit silly because now I know the path existence.
In BIRD we use a solution which is called multiple routing information basis so, we have very large routing table connected to other routing table with prior protocols because in those protocols all the routes are mentioned, not just the best ones and it consumes a lot of memory and it it is quite complex.
To give you a picture. This is a picture I usually use to introduce how a route server is implemented in BIRD. So as you can see we have peering partners connected to a routing, a route server. All the peering partners from the same AS have their own local, but local means also big ones because in this local is everyone. Local routing table. From this routing table it's filtered to the Master routing table and then exported back to those peers, what does it mean? It means that every route is duplicated in all the routing tables. So that's why the memory complexity is very, very huge. On the other hand, you have very good control what those guys are sending to you and what you have sending them. But unfortunately in some very large deployment in larger IXPs, this causes a really huge memory consumption, so that might be a problem for some of you.
So, what we did to solve this problem, we introduce add set of single RIB and we allowed to send the second best route to export second best route from the route server. This is caused by a very simple protocol operation secondary and there is one precondition, you have to use sorted routing table. Normally we used to have a routing table that we just had to point to best route and a unsorted list of other passes but now those passes are sorted by priority.
So, that means including any route takes a little bit more time, but that's really hugely compensated by the fact that the route is not replicated by many routing tables. So you don't have to use protocols. There is only one disadvantage, you can't check what exactly routes are filtered out from the peer because you don't see that in the local routing table, but that's probably not a big issue for many of us.
So, again, to show the picture. Now, those things are not part but just the import /export filters, we have just single master routing table and all the peers are connected through the filters directed to that.
This is how this looks like in the configuration, so again I have some at the moment plates for the route server. I have to sort the routing table and now I have an instance of some route server client and with the import and export filters. So, if you use this kind of setup, this might probably help you.
So, let's look at the real situation in NIX.CZ. We have two route servers, one running 1.3.8 with single RIB with Linux. Second is BIRD 1.3.5 with multiple RIBs on FreeBSD we have about 30,000 IPv4 and about 7 thousand IPv6 routes. And this set up decrease the memory consumption to 1.7th in IPv4 and to one quarted in the case of IPv6. And the problem is squared, so it means more larger deployment it says more memory, or higher fraction of the memory so, this is really significant for really large guys, so if you face such problems, this can help you a lot. And of course memory it means also CPU because there is a lot less copying in the memory.
So, that was about the new features and I have some forecasts for you. We are working on some new stuff for you which is looking glass, some people were ?? had problems that there are no good looking glass for BIRD so that's something we are working on. We have BETA, so I think you know, we'll announce this pretty soon, and I don't think we implement ?? we are implementing BGP ET path feature for those of of who are interested in
A little bit further we are working on IPv4 and IPv6 integration, you may know that BIRD has separated duty moon more IPv4 and IPv6, so we would like to unite it and one of the main drivers for was that the protocol ISIS, which we are working on. That's also going to come quite soon. And again as I always finish a presentation like this, your feedback is very important for us, so if you really would like to see some feature, please send us e?mail or talk on these forums and we will be happy to hear it and we'll be happy to help you.
So, that's probably all from my side. So first of all you can save a lot of the memories if you have trouble with the route servers, and we have some new features and we plan some new fancy stuff as well. So look forward to more. Thank you very much.
CHAIR: Thank you and Andre.
AUDIENCE SPEAKER: I am Howard. Could you go back to the network diagram where you have the multiple RIBs because we have implemented a solution where I think we have the big advantage of BIRD that you don't have to have a master routing table so we avoid using a master routing table and we create pipes between the small routing informations tables, and the pipe equals to a bilateral BGP peering in fact, so we don't have to have such much memory, and it works quite well in this case, at least at our exchange point quite good. So I think ?? there is ?? I note where you don't have to have a master routing table and redistribute all these things. That's one of the biggest advantage of BIRD and I really like the software, by the way.
ONDREJ FILIP: It would be interesting to compare the two setups. I might try it to show the results what's better and how it performs, it will be interesting, I will try to da that, thank you for that.
AUDIENCE SPEAKER: Kurtis. Maybe I missed this, but there is a feature request I have been asking for awhile for, ISIS, did you say that? I missed that, sorry, I wasn't paying attention. When?
ONDREJ FILIP: I would expect it by the beginning of next year probably. You know, it's development, you never know what you will be facing, but first of all we will unite the IPv4 and IPv6 which is going to be version 2.0 and on top of that we will implement or ISIS for both IPv4 and IPv6.
AUDIENCE SPEAKER: Will that be part of 2.0 then or is that going to be a separate development?
ONDREJ FILIP: No, that will be in the main branch.
KURT LINDQVIST: You are going to do all of that and then that's going to be 2.0?
ONDREJ FILIP: Yeah.
AUDIENCE SPEAKER: Hello, at even a from the RIPE NCC, I have a question from a remote participant called James Blessing. Question: With a single RIB function you said secondary, how many secondary routes can it hold?
ONDREJ FILIP: Well, it imports the best that matches the filters. So all the available routes from the same prefix are evaluated and the best that matches can go through, so that's the answer. So ?? and this is probably unlimited, because it's a sort the list so it depends on your memory.
CHAIR: Any more questions? Thanks a lot.
(Applause)
MARTIN WINTER: Good morning, I was just asked questioned to give a quick update on Quagga.
So I haven't done a presentation in this Working Group, just for the few people who are not familiar, I am Martin Winter, I am open source routing which is part of ISC. We are not directly like the Quagga community, we are just basically trying to help out Quagga people who have been fixing and we are funded by a few companies who like give us money.
So, if you haven't looked at Quagga recently, just to give you a very short overview before I go into detail where we're standing. We have the different features basically protocols, we have all the protocols in there. Quagga is more likely to be used as a real router replacement. These are similar from the ?? it is Cisco like CLI. The key missing features or limitations, which I probably, or you may not be using Quagga today: BGP efficient for route server, especially if you have many full feeds.
I was surprised before in the statistics percentagewise how many people are still using Quagga for route server but it's usually smaller exchange points.
So, as I mentioned before, from the users. It's not just the router which is like a very small part but Quagga has more important then growing to us like OpenFlow directions, there are a few smaller exchange points. ISPs using on Linux directly as a router, mainly OSPF and BGP. And quite a few large data centres who have highly customised Quagga versions are there too.
So, to give you an idea where Quagga standing, maybe you used Quagga in the past, you want to use it so the overview of the different protocols.
BGP, as I mentioned before, the main challenge there is performance part. There is a EURO?IX branch which you may be familiar with it, maybe not, which Chris Hall worked on it, that one tried to fix a lot of these things. It is successful in performancewise in fixing it.
OSPF version 2 is like basically running quite well. There are a few issues if you are really doing regular large topologies there.
OSPF v3, not great because soft am data structures or the code which some of the fixes from v2 didn't go into v3.
ISIS, as I was asked before, we actually have ISIS for IPv4. We got the last changes in which basically came from Google who basically fixed a lot. We just had David, by the way, who has, sits over there too, is one of the active people in Quagga, he pushed a more fixes from ISIS in the GIT. If you compile it from GIT, you have IPv4, usable or maybe even good enough for your network ISIS.
RIP is basically not a real issue.
But anyway, you are exchanges so I assume you want to know more about BGP. So, I recently ran a quick test. BGP, the main line, which is the one you find on Quagga.net and also the EURO?IX branch, Chris hall, was the last thing I went through the BGP clients, RFC errors, what issues we see. We are looking at it, actually the first one ?? I ?? the bad missing notification that's actually fixed in the URX branch, that's wrong in the main line. If you want to note exact details there is a URL at the very top, if you download the slides, if you go there it gives you the exact details from the different test runs, which tests are failing, which RFC like they are matching to it and the section of the RFC.
So, there are a few things on, like, in error message handling there will be wrong notification or no notification on certain bad attributes receiving. The NIX branch, it has a few cases when the BGP update has like minor errors where based on RFC it should be closing the connection and it close it is.
In the open, we have coalation detection, version check, that's not serious, which is basically wrong.
More essential maybe if you are doing some real deployment and use it for forwarding, the route recursion in BGP, that's the BGP route recursion, is kind of broken, in basically both codes, so if you have a BGP route which depends on next hop normally in BGP you do, but in Quagga it will go wrong.
The route decision also, the lowest peer tie breaker doesn't work and one of the largest things if you work with BGP aggregation that's seriously broken, a lot of the part in it, it does a lot of the part wrong.
So just as a last summary to give you an idea from open source routing, as I mentioned before, NIX with Chris Hall, they sponsor the work quite a bit on BGP work. We from open source routing, we focus currently less on the BGP. We focus more part of Quagga as a real router forwarding so our focus is more on ISIS, OSPF, trying to get that in good shape so it can use Quagga for forwarding. We are working on some internal data structures and changes that should speed up also I hope some of the speedups will help BGP as well. And the API to the Zebra, which is kind of interesting, that comes more from the world of OpenFlow or those who want to build their own forwarding engine below and they want to use Linux so they need a reliable way to get forwarding table out from Zebra.
That's it for the quick update.
Any questions?
AUDIENCE SPEAKER: Maksym. What's the problem to join two branches, main BGP and EURO?IX BGP?
MARTIN WINTER: Actually, we had a discussion this week on that. The challenge is obviously in a community people want to see like nice commits and small breakdown. A lot of the code which gets ?? he basically wrote it on his own basically without much community input. By now is about 170,000 lines and would I say BGP, like more or less, everything changed a lot of the library, things changed below so I think the idea is to see if we can get the library issues like merged in there together and we may actually try to see if we can offer the EURO?IX as a separate like BGP as a user you can pick which one and merge it together. We are still looking at it but it's obviously a lot of work to clean it up, have the code reviewed and so on and test it a bit better too. There is also the testing things which I haven't mentioned if you go to our website, we did some protocol, like the coding testing, there are at least one for sure like a denial of service attack which I found which Chris Hall is working on, it but that's basically some bad formatted packets which crashes some BGP demon. So...
AUDIENCE SPEAKER: A quick question also. Gunter Van de Velde, Cisco. I'm just trying to understand. So, are there like any things here which also relate words to security of BGP as such which are like new features? Like something you can actually offer there?
MARTIN WINTER: So BGP on new features, there are a few things. I mean, right now I would say a lot of of the features, I mean there are ?? it doesn't have all the specialised feature which BIRD has in there which does a lot of special things specifically like for route reflector, route server things. They may have come later from ?? a lot of the things we focus more from getting it stable, that which we do a lot of extensive testing there and at this time, as I say I focus more getting the code quality up and less on adding features.
AUDIENCE SPEAKER: Are you doing anything like RPKI or that sort of environment?
MARTIN WINTER: I'm not sure I understand your question correctly. Are you asking specifically security related feature in BGP?
AUDIENCE SPEAKER: No, just going into the normal RPKI kind of stuff just to secure the routing table as such.
AUDIENCE SPEAKER: I can take that one. I am David Lamparter, I am actually the maintainer of Quagga. There is a branch, I think it's managed by LACNIC, and they are implemented RPKI in Quagga and we will, I guess, merge that at some point but it's not in any new releasable shape yet. So it's upcoming.
CHAIR: Any more questions? Thank you, Martin.
(Applause)
CHAIR: Next up is Toshi, who is going to talk to us about IXP, caching with ISPs.
TOSHINORI ISHII: Hello. My name is Toshinori Ishii from NTT Communications. I'd like to share the information about our trial case.
Small IXP also wanted to reduce their transit traffic, but cannot gather enough traffic because they are so small ISPs. And how can IXPs help those small ISPs?
This is a very small IXP configuration. AS CDN is AS for Content Delivery System.
Small IXP, why small IXPs built their own IXP named Echigo?IX and they put CDN cache in Echigo?IX and. And we can see some merit. It's QOE improvement, low latency and less jitter. And transit traffic decrease, it's about 12?14 percent off. But, we can see some but point, for example, access line traffic was increased, it's about 10?20%. One reason, I think unreason is QOE improfit.
And the next challenge we tried to implement transparent cache technology on the that IXP. It has some technical difficulty especially BGP routing.
Thank you.
Any comments?
CHAIR: Any questions? Thank you.
(Applause)
CHAIR: Next up is Zaid Ali from Linked In. He doesn't often come to RIPE meetings so it seemed like an opportunity to get him to talk to us and he is going to chat about remote peering from Linked In's perspective.
ZAID ALI: Fergus met me in Malta and roped me into this.
Remote peering is something pretty interesting. I have been working actually with Bill on a paper that he wrote, so I'm kind of going to use a little bit of the data that he produced that we kind of talked about in the content community and in general about, you know, what people are doing, what peering, what are the challenges and there is a little bit of a debate so I have some questions that I hope that people can kind of comment on.
So, just observations, Bijal was saying earlier we have so many IXPs that are growing, this is kind of the trend. We got peering density add IXPs that are growing a lot more, people are peering. Transit prices are dropping. These are all individual topics that people can talk about, but I'm just going to kind of bring it up as an observation. I notice that people are building sound cases for peering in various places I go to I see that there are numbered associated and things like that, which is a good thing. There is more love between content and eyeballs, the more there is that, peering will be good. High adoption of remote peering. This is kind of what I'm going to talk more about, but I noticed that talking to people in Europe, it's really popular in the US it's not so really popular, I don't know why, but anyway. Let's just talk about what this is.
So, traditional peering model, as everybody knows, it's, you go to your favourite IXP and you put in a router and then you connect to a switch and there is a bunch of routers that are connected just like you in your neighbours. You pay the transport into your IXP. You are there, or you are down the street, you pay the colocation fee, you pay for the peering port, this the the standard way people do things.
What are the challenges? So challenges are that you know, the cost is really fixed right, so the cost of putting a router, paying for the port, that doesn't really go down that much. It goes down a little bit with competition, it goes a little bit as more exchanges come around. There is just market push, but it doesn't really go down as much as transit does. The transit has been dropping a lot, lot more. And those routers and port costs, they don't really go down.
So, if you want to look at it. The cost of peering, all the time is going down but the transit price is really really gone down and keeps going down. In Europe it's really really cheap also. So you really have to push a lot more of your traffic into peering, right. And we have large traffic volume it absolutely makes sense for you, you have got justification but what about the other folks who are trying to make other things like I just want to peer for performance? Now, do I pay so much for actually doing that? What's the benefit for that?
So, remote peering becomes very interesting play here. Remote peering model. You kind of go to your favourite L2 MPLS provider and say hey, I want to buy a transport service from you and then you get me to this various exchanges that I want to go to. And so you don't have to invest in all the routing and switching equipment. You start at point A and you don't have a lot of capital expense, you don't have any deployment and you can actually bring up these exchange, you know, pretty fast like within a week.
Router, configurewise, very, very simple. Your favourite exchange provider will give you a transport. You plug it into you're ethernet, you create some interfaces, apply the right rules and you're done. So very, very easy way to join with the remote peering.
So, people are kind of throwing some numbers around. Bill and I walked and talked around with a bunch of people in the content community and in general, and these are numbers that we kind of came up with. These are not one specific case but we took the high and the low end and we found that people were making arguments that you know, it costs a lot for me to build like say four POPs in different exchanges. On the high end if I'm building like a backbone it will cost me maybe a million dollars can, at 275,000 per POP, may if you like. You have got very expensive line cut and stuff. Some people may argue hey you can do it a lot cheaper. These great if you are just putting in one router. That's T then you have got to think about all the circuit costs, which you have to pay for. With remote peering you don't have to pay for all that. You can connect with something as low as roughly 6k a month. For just your circuit costs, but really ?? sorry, that's actually connecting POPs. If you add all the stuff on your left side it's really really expensive, but really you are looking at you know, something like $1200 a month, kind of a ballpark figure I put it there. So I don't want to favour any vendor. So anyway, you can go at really low cost, 1 gig, let's say you are starting out, so that's actually a very good way to get in and actually helps your argument of getting into a POP very quickly. Or increasing a peering that is.
So, our experience and advice. We have a hybrid model exchange. What I kind of made this slide after I heard the first presenter talk about remote peering. And what I want to say is that you probably don't want to do remote peering across continents where latency is not in your favour. So don't do that. That is a very bad model. You will fail, the whole model will fail and that's not good. What you want to do is try to make the hybrid approach where you sort of blend your traditional IXP setup with remote peering. So like, for example, if you are in Europe, you could ?? and you are trying to get, you are trying to extend your backbone or doing something in that nature, then you pick a location where you actually build out your POP in the traditional model and then kind of expand to get close to your eyeballs from there. So this way you are not hauling all your traffic across another continent but you still keeping it localised but you are getting closer to eyeballs and quickly because you don't have to pay for all those POPs.
So, as I was saying, remote peering is a great way to get closer to eyeballs. This is a good way to sort of grow your peering while, if you are building out a backbone and takes time to do this stuff, it can take a year to build a very good decent backbone, but you can kind of save that time and get closer to eyeballs by doing remote peering.
One thing I find with remote peering is that IXPs treat you the same even if you come through a partner. This is a really good thing, so I encourage IXPs to keep doing that because that will really help the adoption of remote peering.
So you are not treated like in sometimes when you are resell our models. People who resell circuits they don't necessarily give the same quality of service to a partner as they would do to a serve a direct customer, so that's my point.
So the debate. So this is kind of what I want to get to at some point. The community has kind of talked about you know, hey, L2 service as more complexity. It's hard to monitor. It's complex to debug, compared to L1. It adds latency. Remote peering can lead to routing niche see. Breaks the model of peering. Keeps the idea of keeping traffic local. Latency benefits could disappear. High adoption of remote peering could lead to routing problems or anomalies. People will say that you might not be doing good engineering with this. So, the other one is the dropping bits on floor while waiting for BGP timers. Now you have an L2 service which means you have to wait for timers for your BGP session to actually expire, whereas if you have an L1 service it's pretty fast.
So, someone may actually counter argue with that and say how it's different from peering across multiple. At some point during questions I'd like to hear your opinion.
Some people say commitment issues. Right, not physically present means that you are not really serious about peering in that region.
So, I usually say that it's really about choices. So, I like to think of a transit like a bus, right. Where you know it's completely full, it may not take you there in time, you might not be able to get on it, so, it's a mess, right. Peering is about choice. You can directly peer to people and move traffic across, I kind of think of it like a car. Remote thinking is kind of like a zip car, like a taxi service because you don't want to pay for the car, but you kind of want to just get somewhere very, very quickly and move and get to eyeballs quickly so I would use that as an example.
Okay. So, Bill and I worked on, I really want to give thanks to Bill Norton because we kind of talked a lot of about this and he developed a white paper which you can get, about remote peering and I have some comments on that paper as well and some people here, I think from AMS?IX as well. So I would say that I can kind of bring it back here and open it up for questions and see what people's opinions are. I'd like to hear that.
AUDIENCE SPEAKER: Mark, you highlight the attention point of latency. Linked In, I guess, is content eyeball traffic mostly. Can you highlight a little bit what the multiple BHP sessions mean in your latency traffic? I guess your screens are being built newspaper multiple sessions.
ZAID ALI: We think it's just like extending to a POP right somewhere like New York and London and I have another POP in Amsterdam, I am still going to build a transport layer, right so. I have that amount of latency still. It doesn't ?? unless I am doing some kind of local caching if I need some presence there where actually I am serving content of that place, then latency will matter and it won't actually help me, right, so, in some places in the world, that's I didn't said the hybrid approach is nice, because that will remote entirely on remote peering because that won't help me but in some cases where I just want eye balance traffic to come quickly to me, that's another key thing, right, I want control and that's why we do that.
AUDIENCE SPEAKER: I am Nina from CDC. I am just wondering two things. This is very much from ?? I am a content holder perspective that you have. And I am kind of wondering if the same benefit would apply to a network like mine where I have like eyeballs. One thing that I do see that I think I would like you to elaborate a little bit on, is the routing inefficiencies. So I wonder, do you use traffic engineer where you send your traffic on which ports according to the distance of your remote peering connection, so you try to get as close to somebody's eyeballs, and what have ?? you have several sessions with the same network or with the different IXs that you reach. How do you choose which port to send the traffic to? Because if you have all the ports on the same router you actually have only one port as far as I understood your presentation. How do you choose between the different sessions and make sure that the traffic gets most efficiently to the end users? I see that's an increased complexity that you haven't even mentioned in the presentation.
ZAID ALI: Yeah, that's a good question. So that is part of the debate on the routing complexity, right.
AUDIENCE SPEAKER: I am asking you you do, because this is your experience right?
ZAID ALI: We do a traffic engineering, we don't ?? well, we try to peer with people in multiple locations. So, we actually do have policies to actually do that. It's a little bit tricky. We actually don't ?? we try to let BGP do its work as best as possible, unless we actually feel that ??
AUDIENCE SPEAKER: That actually means it's completely random which port you use.
ZAID ALI: It is, yes. For the most part unless some cases but we can talk off line about it if you want.
AUDIENCE SPEAKER: Will Hargrave. Going back to, you asked for comments on your section, your presentation dropping bits on the floor when you have a layer 2 outage. This sort of like apples and oranges so when you have any decent multi?site IX has a very good resilience protocol in its layer 2, so, any outages is under ?? some, it will be very short. When you have this sort of carrier arrangement, it's actually very difficult to do some inter provider layer 2 resilience which kind of means we are building a bit more of that fragile IX. If you imagine an IX that only has remote participants, and you can think about, you know, where outages are likely to occur, they are much more likely to occur where you have multiple parties involved because you can't assure the resilients, so I think they are very different things, a multi?site IX versus your members connecting using carriers.
ZAID ALI: Networks that's a very good question and that's actually a big point of debate. A lot of people have said, even internally in my team, discussed and argued a lot about this because we actually prefer L1 service, we like to be at the exchange but sometimes the cost isn't justified, to you would say well, you know, can we just go with this model and live with it? So it's kind of ?? it's really a choice, right, so where you want your RIS to be. And yes, I think the biggest challenge in that is really up to these transport providers, who are in this business to make sure if they actually provide you a good L2 service, and give you that level of service that doesn't cause outages every time and all that, that's the success of remote peering, right. Otherwise then you are right, the fragility will take over and people won't do this any more.
AUDIENCE SPEAKER: Here is the thing here is that a lot of Internet Exchanges are obviously looking for increased reach, they're partnering with people to do this. And maybe we as Internet Exchange Operators need to think carefully about the way that we engineer resilience with our partners and carriers with the service that you guys buy, because I wonder whether that is good enough right now?
ZAID ALI: Good point.
CHAIR: Mike has got something to follow up on that.
AUDIENCE SPEAKER: Mike Hughes. Just to follow?on from Will's point, are with the thing about L2 fail over and waiting about BGP session to say die. That's going to happen if you are over on an IX with shared media regardless of whether you are doing remote peering or whether you have got an L1 connection if the peer goes away on the far end T doesn't matter whether it's one IX switch or it's multiple IX switches F there is a particular peer that you care so much about that you need instantaneous detection of that peer going away. You need to P N I with them or you need to use some other sort of way to check liveness. I think that's the thing and obviously when you get to somebody that you care about that much that you want to P N I with them then you have got to be local them, you can't just say well do that. I see where you're coming from with the hybrid approach and I think that's great. I think too many people have gone down the road of being remote only or very heavily remote. And having this hybrid approach has been to the way, it's a good cheap way of putting your toe in the water and seeing what the peering market is like in new exchanges without committing yourself. I think it's good for both of exchanges as well. Those are just my comments to follow?on from what Will said. Particularly your L2 thing. It doesn't matter whether it's multiple exchanges or remote peering if the far end goes away, you still have got to wait for timers.
CHAIR: I think Will's got a followup fpr that.
AUDIENCE SPEAKER: I disagree with you, Mike, what matters here is not just the fact that an outage occurs and howing fromly is occurs and lasts. If there is a sort of, if there is more fragile elements in the network, then by its nature, there is a greater RBKI of happening and the impact is greater. It is relevant to think about this. Yes, this is a problem with Internet Exchanges, but I think we need to think about it in the whole context of the whole network and the whole path.
AUDIENCE SPEAKER: Mike Hughes: I think you're right. I'm not a million miles apart. Like I say if there is somebody you care about that and you need immediate notification you need to take the fragility away, like you are saying the fragility is a bad thing.
CHAIR: After that double act...
AUDIENCE SPEAKER: Blake, withheld 33, just a purely hypothetical question out of curiosity, have you had any service providers propose you a virtualised router service at any exchange like where they lease you for example a Juniper logical system?
ZAID ALI: No.
CHAIR: Any more questions, any more double acts? Okay. Thank you.
(Applause)
CHAIR: Okay. Next up is puss let owe is going to talk to us about the new exchange at Lesotho. And then after that, we are going to do the 30 second IXP updates and if everybody who is doing one could could think gate by the AV area, we'll whip you through the slides quickly at the end.
PUSELETSO PUTSOANE: Hi, everyone. I am from Lesotho. I currently work at Vodacom, but I volunteered the exchange.
So, this is where we are, this is Lesotho. It's a very small country, a dot in South Africa. And we are around 30,000 kilometres squared, so it's over 2 million, just a little bit over 2 million of population.
The IX was launched or was established last year on the 26th August. The initiative was from the Lesotho regulator, they worked together with the internet society to get it up. And at first there was supposed to be at least 5 ISPs, which are the ISPs in the country at the moment, but only two are peering. The other three have the infrastructure at the IX but haven't ?? are not connected to the switch yet because of some reasons that will come later.
This is graphs that were taken from one of the peering. We reached a peak of 1.3 Meg, I don't know what was happening there, but that's the highest we got so far.
Like I said, we have two, only two peers, two people peering. One of them being Vodacom, which is where I work, and the reasons why some are not peering is that the cost of back haul to the exchange is a bit expensive for some he have them and one of the mains reasons is that they are also resellers of the ISPs that are already peering. So, there is no point. And lack of local content is another issue, people still use the area who have g?mail accounts and they still host their website online and the number of Internet users within the country is also limited, bandwidth Internet is still very expensive.
And it's a struggle to get ?? it is a struggle to get people to peer and another most important thing is there is no dedicated staff at the exchange but there is a big work around on that by the Lesotho regulator, so there is a lot of ?? one of the main things that's happening at the moment we are moving from the current building of the exchange to another, where there is proper infrastructure, proper power source and stuff like that.
The benefits of the exchange: A lot of engineers in Lesotho got exposure and training by the help of ISOC. There was AFPIF and there is RIPE today, so there was a lot of things came out of the establishment of the exchange. We are also in the process of getting the local DNS for the ccTLD, which is .ls, being hosted in the country, it was initially hosted in the neighbouring country, South Africa. And since the establishment of the exchange, a lot of engineers now know each other. There is a lot of exchange on experiences and any issues that we have, we share. And we are working hard to get it up and running, the other exchanges. In the next 24 months we would like to have the other peers also working, finalise policies, address the human resource issues to get real staff working there, and finish with the ccTLD hosting and the transit for the exchange we don't have transit at the exchange. And NTP also maybe to get route servers.
How we can be helped: We are not very experienced because it's the first exchange in the country, we are trying to get a business plan, and finish with the hosting of .LS and probably host the route server.
So that's it. Any questions?
CHAIR: Thank you. Any questions?
AUDIENCE SPEAKER: It's not a question, it's just a comment. This is mike from Google, and we are very pleased to see this development in Lesotho (comment) we are very pleased today's the internet providers in Lesotho coming together to exchange traffic locally regard than internationally where traffic ?? how much does transit cost in Lesotho?
PUSELETSO PUTSOANE: I think, mike, it's $1,000 I think, I'm not sure, but it's very expensive. It is.
CHAIR: I think it's time for Remco to date us on how much does transit presentation.
PUSELETSO PUTSOANE: I forgot to mention we are getting Google caches, I think the infrastructure is already in one of the peering companies and it will be interesting to see how the curves change after the Google caches are up and running.
AUDIENCE SPEAKER: There may only be 1.5 Meg on traffic today but that's states 1 and a half thousand dollars per month that's with only two people on the exchange and without any local content so it will be really interesting to see how that grows. Thank you.
CHAIR: Any more questions? Okay. If we could have the 30 second presenters up. And this time, in case you wonder when your time is going to run out. Andy is going to ring the bell.
So, your presentations are in order of first name, so if you can like show your badge to Marco, he will then be able to find it and put it up on the screen.
Seconds out, round 1...
BEN HEDGES: I am Ben Heges from LIX, we spoke to you in Vienna about we were starting a region national initiative in the UK, look to go put some Is out into the regions and we launched the first one in Manchester in April. We have now got 28 peers up there. We are having a local meeting on the 3rd October, so if anyone is in the UK in around Manchester come along there. It's going to be a couple of hours of quick update in Manchester. At the cube and then a social afterwards, and then we are having a consultation meeting in Nottingham about a week later about an exchange in the east of England. So again if you are interested in coming there come along.
CHAIR: One thing Ben forgot to say is it's birthday today. We could all sing happy birthday but we'll save him the embarrassment till later.
Round 2...
My name is Akio Sugeno from Telehouse. I'd lik to give you an update on NYIIX. The number of parties continue to grow, now we are getting again, getting a lot of customers from all offer the countries, the like of Brazil. We opened a new location, 32 Avenue of Americans. This is gives us two sites, the hub, meet me room in the core site. Also thanks to mar item from the Hurricane Electric, now we support jumbo frame.
(Applause)
Round 3...
SPEAKER: My name is Ben Gordon, I am from Stockholm and there is a new EIX there, actually it's an old ex, sort of old, 2006, and it's SDAH IX, and it's, it consists of 1 gigabit ethernets and gigabits 10 gigabit ethernets and it has 8 POPs.
The next one with the contact page. There is a presentation, you can download it.
CHAIR: Anybody what wants the contact details. It's on the archive.
CIARA MASCINI: What I wanted to tell you is that we have a new website. As mark already mentioned. And we are very happy with it. It has a lot of visuals. Something that our old site lacked at times. Also we have a lot of new stuff, you can see the details on the statistics there. We have a very ?? a nice visual on where all of the remote peering is available. Well you might have noticed we launched AMS?IX Hong Kong earlier this year. We are going to rebrand the crib yen Internet Exchange to AMS?IX crib yen and we have started a recent programme with C come and activity with CIX P in Kenya.
We will operate our switch to brocade when we will have more ports available for the customers. We are started recently the route server services, we are running BIRD, so it can be added to the list. And also, we are building our new data centre for our infrastructure, so we will free some space, space and power for the customers and we can host more customers. Thanks.
(Applause)
KURT LINDQVIST: Hi, I am from NetNod. We are on the exchanges in Stockholm, Gothenberg, Sweden. With have a reseller programme for remote peering, there is three current partners you can go talk to. They are all three here. We are work on 100 gig launch. We have a NetNod customer meeting on 9 and 10 of October, open to anyone. You are all more than welcome. You can go to find pricing and how contact information and who is connected and the traffic stats on the URL there, and you can send us an e?mail and you can also peer with us for the I root server. Was that quick enough?
(Applause)
ONDREJ FILIP: Hello, Andre Philip from NIX.CZ. We have five locations in Prague and we have news to share with you. First of all we introduced a new 100 gigabit connection. It will be available very soon and we have a first customer who announced to join with 100 gigabit and we also plan a significant, very significant price reduction for next year E we are waiting for approval, but the promise is to be about 20% lower, so it might be interesting for you. Thank you very much.
CHAIR: Thanks. Last up is Marty from TorEx, Toronto Internet Exchange, whose slides will be available later if you want to grab them.
Martin Hannigan: I didn't do any slides. I am on the board of TorEx, where 151 in Toronto. By the way with the closest exchange point of Europe. 127 participants. 92 gigabits peak. Non?profit. All volunteer, 1 gig, 20 gig ports. A little bit of a cost benefit over New York as well. Distance, neutral forth most part. Contact us at peering at TorEx dot CA. Thank you.
CHAIR: Thank you, Marty.
Okay. I think that's everything on the agenda. There anything else anyone would like to talk about or raise as an issue or something?
Well, thank you all very much for coming. A big thank you to Amanda who has to do all the work now typing up all the minutes and doing far more work than the rest of us put together. Thank you to our other scribes, the jabber scribes, the AV guys from both the RIPE NCC and the hotel, Andy my co?chair, and all the speakers, that was a great programme, and the stenographer from lovely Dublin. Thanks again and we'll see you all in Dublin in about six months time, stay off the Guinness. See you then.
(Applause)
