These are unedited transcripts and may contain errors.
Plenary Session
25 September 2012
11 a.m.
CHAIR: Welcome to the next Plenary Session. Sander and myself will keep you entertained during this session and also our great speakers, and the first speaker is Jan Linkova from Google, she will talk about what's up there on the screen. I believe Google is quite a complex network. Thank you.
JEN LINKOVA: Hello. My name is Jan, I am working for Google Network Operations and my work in Network Operations is looking after backbone and basically making sure when you are watching YouTube or reading g?mail, all those packets actually reach your computers.
As a result of that, I actually do like to know if everything is okay with the network and especially when something goes wrong. And what are my options.
First option. I can use what we call white?box monitoring. I can set up a system, system goes through all devices asking any problem? Link down, and BGP session, whatever.
Output drops, also we know QoS and Internet does not work and any other kind of problems. The problem is that it's good, but not good enough. Because if the device reports some problem, yeah, I need to follow up, but if device does not report anything and actually does not mean all good, because it might be we cannot trust the box or the box could not report it or it's some unknown issues. And actually, I do not care so much about the router state; I care much more about the ability to do the job, which means to forward packets from you to other services and from services to you.
So, what can I do to test my network? I can use black?box monitoring. I can install a lot of boxes in my network. By the way, one important comment: in this presentation, I am going to tell you about our backbone, which does not use ?? the word I know you'll like ?? OpenFlow. This backbone is using more traditional routing protocols. Just to make some people happy in this room.
So I can set up a lot of boxes. Sending packets to each other. And get an alert if some packets could not go through. Yes, it's actually good because I can see if any packet loss, if any quality of service misclassification or any other strange problem. For example, latency variation, but there are some issues which still need to be addressed. First of all, when I am sending packets from the box to some destination, most likely it will be following the best path, or best paths if you have more than one, and if this system reports some issue, it's sometimes pretty hard for me to find out where exactly I can see packet loss. Just imagine I have Xboxes sending packets to Xboxes, so basically I have a matrix and when it started looking like a Christmas tree, I need to spend a ridiculous amount of time trying to find out what's going on.
So let me tell you a story, why I am not mystified with just black?box monitoring. Once upon a time in one small data centre, three engineers were doing a new router. They erected, put all these nice line cards out, and powered it up, plugged all the fibers. Even configured it. The BGP up, ISI, ISP is up, probably my IPv6 is up. And okay. Now, we are ready to press a big green button and put g?mail traffic on it. Oh, no, it wasn't Google, sorry, it was some other company. And they pressed big green button and at the moment all monitoring system became red, pages started to melt and so on. They were scared, pressed big red button to remove traffic from that box and started to inst gate and apparently they found that they forgot to put switch in fabric in that box. What can I say? I can say how can I prevent this? How can I test links, boxes, interfaces, before I put real traffic on it? Because it sounds strange. On the one hand, I am using ISI overload, high matrix to remove all traffic from the book. And on the other hand, I really want to send my monitoring traffic through that box.
What can I do? We started to think about it a little. Let's look at the problem. How can I test everything even if those components are not in service yet?
Oh, yeah, I can go to each box and start sending ping packets. I can use whatever vendors call IP SLA and ask every router send packets to every neighbours. But the problem is, a while ago I spent three days of my time trying to find out what's wrong, and finally we found that I have a line card and on this line card one interface just black?holing all packets of particular size which go in transit from all other interfaces through this interfaces out. And I was able to ping through that interface from that router. It happened only to transit packets.
So, basically sending some problems packets from this router to its neighbours actually don't tell me much. I still need something more than that.
Okay, so if I want to test every single link and even better, every single forwarding path, I mean every path my packets could take in the network, I definitely need to tell the packet which path to take. Yeah, I'd be source routing. I'll going to say horrible thing. I am going to tell you that right now, I missing very, very much or ?? in IPv6, because it would do what I need. But we don't have it. But we have, we have MPLS. Awesome. I can configure LSPs with strict zeros and say this LSP is going from router A to neighbouring router B to router C and so on. And I can configure LSPs which cover all my network, all forwarding paths or links even more than once. Amazing. I am pretty sure someone is going to ask me, what I am going to do with aggregated links when I have more than one physical path, but single layers of link. Yes, it's a problem. I am not aware of any reliable way to send packets through particular member of the bandwidth. If you do know, please come and tell me. I'd appreciate it.
So we have to use statistics. We are going to send a lot of flows through that link. If you send enough flows, algorithms could probably cover all members of the band link. It's not 100% reliable, but my practice, from my practice I can say it's good enough. But again, I would be really, really happy if I can instruct the router to send this packet through this particular bundle member.
So, now, I am actually able to test all foreign paths in my network. I am able to test that every router is able to forward packets from every egress interface to every aggregated interface. Good. And as I mentioned, at least three times I think, that I can look at my network and make sure I am covering all possible forwarding paths. Pretty good.
Now, I have a lot of LSPs, I have a lot of probe packets going through those LSPs. Some of them even able to reach their destination. So, it's very brief diagram showing how it looks like. It actually looks much, much nicer in real life. So, as you might guess, now I have a lot of signal. Some packets going through, some packets couldn't go through. Some LSPs up and down. What will I do with this information? I still want to know where something is broken. Yes, let's say our network operation falls and we can not ping 8.8.8.8. You may say it's stupid example because I should have used something else so I could find where exactly packet is lost. So even traceroute, even MTR could not tell me if the packet was lost on the way there or on the way back. We don't know actually what causes the packet loss, if it's a faulty router, faulty lean, faulty line card, faulty fibre path, we still don't know. And actually, I'm sitting doing some real work or drinking coffee, I got paid, packet loss. I am at my desk writing MTR, packet loss disappeared. I don't know if something changed, if link went down and packets now taken another path. What exactly changed in the network and what was the state of the network when we saw the problem.
I can actually tell you another story which fortunately did not happen to me, it happened to a friend of mine in another company. They have a line card which, in particular situations where a drop in particular packets, particular sizing if I remember correctly, and the packet loss was about 0.3% of something, and probably no one would have noticed but unfortunately they had a customer and the customer was using some video call that was extremely intolerant to the packet loss. So they spent a lot of time trying to track it down to particular line card in their network and it's not happening all the time. It was happening like in some special situation. Such situation is very hard to troubleshoot here, and I am very lazy, I don't want to spend my life on this stuff, but I spend my time on making v6 work.
So, what can I do? Yes, I have a lot of similar, but, as I mention, all my LSPs have strict zero. I know all the time which path this packet, missing packet actually was taken. So, I can look in my packet loss data. I can look in the packet loss data for every LSP and look for correlation. Is there anything in common between all LSPs which show in packet loss. Any routers in common, any links, any line cards, any fibres? Whatever...
If I can see all that LSPs showing me a packet loss going through one particular link, I can suspect that probably I should look deeper and look at the link.
Actually, pretty good. At least I am satisfied. But we still have a few issues which we'd like to address.
This system creates a lot of states. Creates a lot of LSPs with strict zero. So let's say we have router 6 interface, I have to create about 120, it depends actually on configuration, of the parameters, LSPs covering only this router and you have a lot of routers in the network. So, those LSPs are not so bad as normal CSF LSPs because they don't move. If the link goes down, those SLPs don't move, but this deal can see resources on the router and we still need to manually check our network and create this set of LSPs. So basically periodically our system looks in the network, okay, it's my network, now I have to collate the set of LSPs so cover all links, at least three times. Good, then we have to go and configure all those LSPs on the network. Yes, indeed I'm not going and manually, I am not configuring them manually but I'm doing a kind of configuration push but I still hate touching production network. I don't like pages, I don't like red monitoring, and so on.
So, we are looking into how can we reduce number of straight in this system. And ?? and by the way another problem which this LSPs could not address. I'd like to create loops. I'd like to send packet from router A to router B, C, D, back to B and then back to A, and RSVP could not do that for me. That's understandable. Nobody likes routing loops only me when I am monitoring the network. So, again, we cannot use source routing, unfortunately. We cannot use RSVP but we probably can still use MPLS. That we create statistical LSPs. On each router, if I receive a packet with label X, I am popping the label, sending packet to the interface X. Pretty simple. Static configuration on the routers. I don't need to change it unless I am making new interfaces on the router. Okay. Now, let's say I want to send traffic from router A through the network and back to router A. I am crafting a packet, which actually has a stack of labels 1.2, 2, 1, 1, end of stack. One router A receives this packet. Okay packet with the label 1 on top of the stack. POP label sends is back to interface 1 back to router B. Router receives label 2 on top of the stack. Okay. POP the label, send to interface 2. And so on. So basically, by the end of it, router B receives a packet with label 1, which is actually the last label in the stack. POP in the second interface to router A, auditor A receives pure IP packet forwarded back to its destination. All good. Our probers are happy. Packets received. All good.
So, we actually already have in production the system with LSPs with strict zero. We are now doing a kind of pilot with statistical LSP stuff. You might ask me what have we learned and does it actually work. And I am happy to say that it does work. It actually works surprisingly well. Because, again, I am IP ratings at network, I am using this system on a day?to?day basis.
We can find very interesting situation, we can find silent packet loss, I mean packet loss for which you could not find any indirect signals, like any counters on the router increasing, no log centres, just silent drops. We can find pretty low level of packet loss and we actually we are able to find and detect and trouble?shoot packet loss which can happen for five minutes. You can find where it happens, probably look at why it happened.
And the most important thing, it speeds up the trouble shooting process. Real life example: I was configuring a new router, I was going to put production traffic on, it but I look in the system, oh, my God, I have two links on that router, two aggregate links and those aggregate links showing me 50% packet loss, only two of them. I looked at the router and I can see a line card and half of bundle members for both those links are on that line card. I shut down that line card, packet loss is happier, I have fallen out with vendor, please send me an error. It took me less than three minutes. Including the time I had, it took to log into the router. And you have plenty of such examples. When you can very quick detect and correlate packet loss to the particular network company.
Interesting, and again, we can test companies which are not in production. We are looking into this in this system and this dashboard before actually putting any new components in production. And I'm going to put traffic on new line card, no link, new router, new fibre. I am looking if this system is telling me it's good. Awesome. If not, we have to troubleshoot. Because again, I definitely don't want to use your traffic, your packets as a monitoring of my network.
This is this has the most interesting side effect. Originally our intent was to monitor forwarding plane, but we managed to find a couple of very interesting bugs on the contra plane. We found some situation when RSVP misbehaves because LSPs, or using RSVP so sometimes you can see something wrong with the LSPs, you trouble?shoot. And you find another bug with RSVP. It's pretty hard to find those bugs in production because it could move quite fast. And actually, you know, that's it from me.
I was ten minutes before the schedule. But, I hope you guys have plenty of questions.
CHAIR: So, complex network calls for complex solutions.
JEN LINKOVA: Not complex, simple, beautiful.
CHAIR: Yes, I see the queue on the mic.
AUDIENCE SPEAKER: Hello, David Freedman. Did you consider using BFD Echo mode?
JEN LINKOVA: I don't think it could tell you that I can forward packets from A to B.
DAVID FREEDMAN: No, but for every link connected to a router you could send an Echo mode packet around that link to the forwarding ??
JEN LINKOVA: Again, it would be from this router to its neighbour and I have seen this situation more than once in my life where you could ping through that link, but cannot forward transit traffic through that box.
DAVID FREEDMAN: Okay.
JEN LINKOVA: It's actually exactly what we would like to detect.
DAVID FREEDMAN: So following on from this, then, with your static LSP, how do you cope with multiple links in a bundle, an aggregate link?
JEN LINKOVA: It's exactly what we are working on right now because statistical LSP is kind of lab testing, but LSP with strict arrow in PRODUCTION, and, as I mention, we are taking a lot of flows so basically I look at how many members I have in the bundle and I have created ?? like, four, I think, or eight flows per a link, hoping that will be hashed across.
DAVID FREEDMAN: So I take it you are doing that aggregation at layer 2, you are not doing any layer 3 ping, you don't have a label per bundle member.
JEN LINKOVA: If I have LS3 link I am able to send LSP through that link. And if I have another link and I have somehow to another LSP cover in this particular link... that's why I really need IP address here on every link so that's why I don't like in v6 having backbone only with link local addresses. I know RSVP don't work over v6 now, but in future.
DAVID FREEDMAN: Thank you.
AUDIENCE SPEAKER: Hi, Will Hargrave from LONAP. Your description of the monitoring aggregated links problem is quite interesting for me as an Internet Exchange operator because there has been various instances where you have a big multi?way bundle and one member that have bundle goes down and the only in an Internet Exchange the way, that you are assuring life nest is by having a BGP session a sing sell TCP session is going over one lack bundle member. So actually my thinking on this is actually it would be nice if we go back to the bad old days of simply round?robbing across all the links because then at least we know when something breaks. The associated control plane traffic will die as well. And I wondered if you had any thoughts along that way?
JEN LINKOVA: So, for Internet it's probably better because you have LACP, because, again, I have seen the situation when small packets are going through, fine. LACP is okay. You start to send large packets through that link, packets disappearing.
DAVID FREEDMAN: I have lost a lot of faith in LACP anyway.
JEN LINKOVA: Again contra plane is good but not good enough. I want to make sure packets are forwarded.
CHAIR: Okay.
AUDIENCE SPEAKER: Just a small question on the 'it works' slide. Just prior to that, you were describing the static labels and here you are back to RSVP, so are these the results from the previous ??
JEN LINKOVA: Yes, initially this slide was in the middle but then I thought that probably it should be kind of conclusion slide. So again, RSVP system is in production. We are testing what we can do with static LSPs because we still need to make sure all vendors supported and they are supported exact leak as we want. Not as they think we might want.
AUDIENCE SPEAKER: So actually what are saying in the previous system when we had all this mash of tunnels we discovered a number RSVP boxes.
JEN LINKOVA: Yes we still do. We still discovered RSVP box, unfortunately.
CHAIR: We still have time for discussion.
AUDIENCE SPEAKER: Gert Döring, Internet operator. Thanks for bringing this here. This is cool. This is extremely way cool. I have been toying around with doing something like that in our network and never found time, so this brings it back on my to do list to really get going, because this is sort of what I had always been worried about. Some of my backup links might not work when I need them and I cannot properly test them in advance. What I'm really hoping for is that all my upstream providers that use some sort of weird traffic engineering MPLS in their net would actually do this and notice if they have problems.
AUDIENCE SPEAKER: Job Snijders. I was curious, do you also alter Mac addresses, for instance, when you create flows, because I recently had a fun issue on foundary equipment where packets towards a destination Mac address started with a 6 were load?balanced in an awkward way, so I was curious in your network ??
JEN LINKOVA: No, because what we do is we have probers connected to basically our network, and as soon as packets enters our network all Mac addresses should be the Mac addresses of routers on the link, yeah? So I'm actually testing exactly what happened to my production traffic. So I'm not testing local no ??
AUDIENCE SPEAKER: This was Mac addresses words to 6, that start with a 6 inside pseudo wires.
JEN LINKOVA: Oh, we don't use it. So basically, we test exactly what we see, yeah, we send in packets through MPLS, LSPs, we are not testing something we are not using.
SANDER STEFFAN: Thank you for your presentation.
(Applause)
CHAIR: And our next speaker is Job Snijders, he has had some debugging issues in the past and he is going to show us how he solved that.
So, while we are waiting for Job to get his microphone, please rate the previous talk ?? it's not uploaded ?? ah, can the tech team please make sure that the presentation supply loaded to the RIPE 65 website.
JOB SNIJDERS: Hi guys. I want to talk about the NLNOG ring. I'll first introduce myself and talk a little bit about NLNOG and then we'll dive to the cool stuff, which is the ring.
This is me, I am Job Snijders. I have worked for quite a bunch of companies. Mostly in the Netherlands, and my hobbies include IP routing and MPLS, v6, LISP and the Ring.
The NLNOG group is not a former organisation like UKNOF or what the Germans do. It's a loosely?connected group of operators, there is a mailing list which is dormant and there is a very active and distracting IRC channel. Nevertheless, this is where the ring originally started.
What is this ring? I'd say the ring is a platform for advanced debugging on an inter?domain level. The foundation of the ring is based on trust. I trust you with access to my resources, as you trust me with access to your resources. This seems to work pretty well for BGP, and, so far in the last two years, it has worked well for the ring.
I'll explain how the ring came to be, what the ring currently is. It's global domination. Some examples of the CLI interface, the web interface, and that.
In 2010, in December, I remember this fondly, a friend of mine had a problem on a certain well?known Amsterdam Internet Exchange. The issue was that from random sources to random destination IP addresses on the other side of the exchange, there was packet loss. And this was very hard to replicate. So, he asked other guys like me on IRC, can you do a ping sweep of this /24? He e?mailed people: can you provide me with traceroutes and see if you can reach that certain prefix properly and can you alter your source addresses. He even had to phone?call people. And as you can see, this took a long time before he had collected enough data.
I included a 'Lord of the Rings' image, because this situation proved that debugging networks was not trivial. And I thought to myself, wouldn't it be much easier if he could just go to those networks himself and perform those tests, and bypass that whole communication delay of asking people for information. Also, quite often, if you ask a person for information, they will send you back a trace route but without a time stamp.
So this is how the ring started. Now, we gathered that it would be easy to have boxes in all these networks and give each other access to the boxes. And now you can just log in with SSH and do tests.
It turned out in this particular case that there was a line card which had a few ports in an receiver channel set up and one of those dropped packets to the floor in weird circumstances. So what used to take like three or four days, now, with the ring, takes hours.
The ring has grown quite a lot since 2010. There are now 135 nodes, because we added one yesterday. They are spread all over the world in 27 countries. So, it has grown considerably since its original Dutch roots.
A random selection of ring participants. You can see even the Germans are participating. You will see there are companies from the whole ecosystem, they are content providers, transit providers, access providers, even a certain registry has joined and we have a domain named registry. So, the ring is for all network operators. You can even see that competing companies have joined the ring, because my grandmother used to say even in the case of unidirectional packet loss, nobody wins.
I'll now run you through a few examples how you can use the ring. This is the CLI interface, which is a fancy word for SSH, and it just means you type stuff in and things happen. We have developed a tool called ring?ping and what ring?ping, in essence, does, is it logs into every box. It will execute a demand and send back the output to the original place. I have ?? imagine that I am a transit provider and a customer of mine complains that he cannot reach a certain website. With ring?ping, I can quickly introduce is it customer at fault or am I at fault or is something else going on? And in this particular example, we see that over IPv4, the Internet initiative Japan website is very reachable, only [OKIDE] cannot reach it because this is by design because [OKIDE] is an IPv6?only node. IT gets more interesting if we execute an IPv6 ping. We see that latency is slightly higher on average compared to IPv4, and there are two boxes that cannot reach it. I know that from jump 01, which is in London somewhere, I believe, that IPv6 currently is broken. But let's zoom in on Inotel, because I'm not aware of any IPv6 issues with them. I can just SSH to the Inotel RING node, execute a traceroute 6 command words to the target, and I see that it dies after the default gateway. So, this could mean that INOTEL for some reason does not have a full routing table or one of their upstream providers is black?holing traffic. I don't know. In it particular example I'd send an e?mail to them and ask them why can't we reach this well?known website?
Other users for the RING are, for instance, checking your name servers. Every RING node runs its own resolving package unbounced and if you use GO IP to deliver your services, the RING can actually help you to test how you look from different countries and you can all do this in realtime. You can traceroute from all these nodes around the world. We have made some tools that you can just type RING trace and get a nice overview. And this gives you very good insight in how you reach a target. Or how your competitor is doing.
MTU testing, we'll get back to that later. MTU testing used to be quite hard because you'd need some cooperation, some server on the other side so handshake. With the RING, this has become much easier. You can validate your ASCLs, look from the outside because now you have shell boxes on the outside of your network. You can test your firewalls ? are they doing what they should do? And the main reason why the RING started, buck layer 2 or layer 3 load balancing issues. You can outer source ports. You can try from a lot of angles, you can quite fast pinpoint where load balancing issues are arising. Because you have shell access, you can script your own tests. We have installed a gazillion tools on all the RING nodes, but you can just write your own thing and deploy that on the RING. It's all quite easy. Another cool example is a script all RING trace, written by ?? in this particular, we asked the script to pick 10 random nodes, perform a traceroute to the target, the J route instance, and visualise that in an appealing way. You can look at a high?resolution version if you download the slides. To be honest, from a practical point of view, I don't really use this method, I more like the other presentation. But this is a good script to convince your managers that you should join the RING because it generates pretty images.
Now, I want to tell you about something that is new. This is not really been published before but the RING now has a web interface. And it's beautiful. The web [] better face is in [] bat beta phase. The software is not opt am and we are the first to deploy the software on such a large scale. So bear with us if sufficient doesn't work as advertised.
Ample is the web interface it's called. It stands for active measurement project. It's developed by the WAND guys. They are also made scamper and they have created a lot of very cool open source tools, and I'd like to thank Brendan Jones personally, because he is one of the developers on the project, when I e?mailed him I won a something for the RING, his first reply was awesome, let's do this, I'll help you. So we actually have some form of support for this web interface.
Which do any to any monitoring. I have drawn a few notes, 1, 2, 3, up to the last one. And all the nodes will monitor all other nodes with either pings or for instance, they do traceroutes every 15 minutes, these traceroutes are recorded in an archive so you can go back quite a while. And MTU is measured, packet loss, jitter, basically anything that we can convince the software to do
We have some puppet masters that configure all the nodes and the puppet masters will push words to the nodes what the targets are and what the schedule should be like for such measurements. The nodes are run a demon, the demon performs these measurements and they are reported to a collector. The collector also runs the web interface and it will store the data for the time to come. And what it results in is, in, for instance, if we phase off Germany on this side and England on that side, we get very nice over views of how networks are doing. And as you can see, this is IPv4 latency graph. This is all pretty good. What's red is stuff that has deviated from the average of the hour before, so, in case latency is unexpectedly increased, it will turn red?ish.
We can show the same thing for IPv6. I love ample because it has full feature parity between v4 and v6 and every monitoring system that does not support v4 as well as v6 in a seamless matter should get off the market.
If we look at v6, the images is very different. There is jump, which has no v6 currently, and we see multi?play, and it can reach some nodes but not all of them. I can hoover hover my mouse on top of one of these cells and it will show ?? I see it's barely readable, but here you see the average for the last hour, last 24 hours and last seven days, and packet loss has been a hundred percent for the last seven days. Which is not really good news. Now, if I would click the cell, you get graphs like this, which is the sources of the tests listed on the left side. This is time, and the colours indicate how they are doing.
So I checked with these guys and it seems that their BGP announcements aren't making to everybody on the Internet. I don't know the exact reason, but this is stuff that is much easier to see with the RING because you have instant visibility instead of five complaining customers.
The ample interface provides with you a more classical approach, such as in this case we have the mean, the maximum, the minimum, the jitter, which is a small green line here, and this is all graphed and stored in quite a lossless way.
So, with RRD you would lose the data as you move forward into the future with this approach we do not lose such data, so we can have very accurate views of what happened in the past.
This is also a fun overview. This is any to any Paris UDF traceroutes with path MTU path enabled. This is all matrix 1,500. For v6 you can see that we are doing not quite as well. I am unsure why we get this results. I am not here to dive into the exact analysis of the situation, but it's worth for RING participants to have this information, and actually try and solve path MTU discovery issues before all the customers are v6 enabled.
As I said, traceroutes are recorded. In this case we have a traceroute from soft layer words to your, and there is a nice button here and if I press, it I'll see the reverse. So if the mouse clicks, I'll get a good overview.
We also run a BGP looking glass and this is not really a fancy thing. It's very similar to route views. The only difference is that we have a pretty web interface and route views does not. Currently we are taking 30 IPv4 full tables and 31 IPv6 full tables and any RING participants can set up a multi?hall BGP session to our collector.
In the web interface, you can just type host names like www.ripe.net and it will resolve this and show routes and this list goes way beyond the slide. There is a small button here, view BGP map. And it will show something like this if you press it and it will use graphs to visualise how things are connected.
Let's talk a little bit about RING governance. Because NLNOG is nothing, it's just a name you can use if you think you have a cool project, and the RING is, there is no foundation. There is no corporation behind this. It's just a bunch of guys that got together and thought it was a good idea.
Currently, we have four RING administrators of which yours truly is one and our task is to do the daily operations of the RING. So, we configure all systems through our automated deployment stuff. We update, we patch for security issues if they might arise, we do user administration.
Furthermore, if we want to decide on things the RING should do or not do, I asked the mailing list, I propose something and the mailing list can give me feedback and we just use kind of rough consensus bottle to reach a conclusion.
We have a very active community. There are a bunch of guys that are fining very cool scripts or inventing new ways to RING platform to debug their networks. All the equipment that we use, for instance, those ample collectors, that's all sponsored by companies that love the RING.
The RING is a community effort. It is built by us and for us network operators.
Now, let's talk about how you can join the RING. And it's real easy.
All it requires is that you provide a single machine, well you can provide multiple machines, so, for instance, if you have six major hubs around the world, you can provide six RING nodes. The machine must have one IPv4 and one IPv6 address. But if you have no IPv4 space, because RIPE has run out, one IPv6 address is also enough. The v6 address is required, if you do not have v6 you cannot join the RING.
We use Ubuntu 12.04. We standardised a distribution that makes management easier. And you must be present in the default Freezone and have your own autonomous system. You can go to ring.nl.net and I have created a very ugly button on the front page which states application form. You can press that and fill in the form.
I recommend to all of you: Please join the RING. How many of you, by the way, have joined the RING? Can you raise your hands? Excellent...
So, if you could now move to the right, and if your neighbour did not raise his hand, give him a hug and tell him it will be okay.
I am almost done with the presentation. There is one last thing I wanted to make you aware of. Firstly, at 6 p.m. in the room next to this one, we have a small get?together where we can talk about the RING, exchange ideas and drink beer, because Robert from Lease Web is sponsoring, so, be there on first name.
Right. I have reached the end of what I wanted to tell you guys. Do you have questions? Please come to the microphone.
AUDIENCE SPEAKER: [Evango] de Marco, Google. Can you please elaborate on what sort of provisions you have to preclude potential competitors of using data collected by the system for, say, political reasons?
JOB SNIJDERS: What I understood from the question is you were asking whether we have mechanisms to prevent certain sensitive information from leaking to a competitor, is that correct?
AUDIENCE SPEAKER: Well, you can use the system to have information about, say, reliability or latency in network of one of the participants so, if marketing department of one of the members will try to push engineering department to provide information about how specific participant is providing service in some region, how you would preclude this from happening.
JOB SNIJDERS: I wouldn't. I view the RING as ?? you could go to your competitor and just rent a dedicated server or you could buy an ADSL connection from them and build a similar system to have monitoring like this. What we did with the RING was just make this process much easier. But this, in principle, is nothing new. I'd consider RING nodes to be just colocation customers.
AUDIENCE SPEAKER: Okay. Thank you.
AUDIENCE SPEAKER: Hello, this is Andreas Polyrakis from the Greek Research Technology Network.
JOB SNIJDERS: When will you join?
AUDIENCE SPEAKER: Well, this is quite an interesting presentation, so, most probably you have warned us. Probably you can give me some beer in the afternoon.
The question is can you back to the slide with the RING?ping? So, I wonder, supposing you have a lot of people joining. How well does this scale if you have, let's say 1,000 or a few thousand servers here and could this be a tool for a DDoS attack?
JOB SNIJDERS: So you have two questions. Let's address the first one. Scaleability: I firmly believe that the RING itself can expand to quite a large size and that the true scaleability issues would be the user interface words to all this data. So, I mentioned that in a situation of RING?ping you can specify with a flag on the come outline only Europe nodes, or only nodes with an upstream that is peering with AS 286 or, you know, smaller refinements like that to make the data more accessible.
So that's what I would focus on in terms of scaleability. By the way this is an open source script and you can welcome to help us modify it so it's better in the future.
AUDIENCE SPEAKER: So this is something that would help filtering the results, right?
JOB SNIJDERS: Yes. About DDoS. As I said, the RING nodes are like colocation customers so, you you should have your regular protection mechanisms in place. I don't mind if you put the RING node on a small link to prevent it from using a lot of traffic. We have a zero?tolerance approach words to abuse. I believe in name and shame. So if I ever catch somebody that that is misbehaving and abusing the resources given to him, they wouldn't have heard the last from it. But so far in the last two years, we have not had a single case of abuse.
AUDIENCE SPEAKER: Yes, but first of all, you shouldn't only consider malicious abuser. For example, if I want to have a ping to one of my peers, let's say in the ?? I ran a RING?ping to the website and, by mistake, or without having thought much about it, I ran a ping with let's say 2,000 servers connected to, 2,000 servers in the RING I mean and each of them, and I do a ping with a big MTU probably, then by accident, this is a lot of traffic that the web server that I have randomly chosen from on another network which on I do not have knowledge about, this might cause some problem somewhere, right?
JOB SNIJDERS: That's a good example. I think it's like with a hammer, if you hit your finger, it hurts. So, you should be careful with the tools that have been given to you. Although, in this particular example, 134 single messages are sent, so if your web servers dies because of that the web server probably needs upgrading. In the future it probably is wise, indeed, to try and limit ??
AUDIENCE SPEAKER: But the problem is that I will not have hurt my own finger, I will hurt someone else's finger and most probably by accident, so... but this is still very interesting, I mean...
JOB SNIJDERS: Thank you for your questions.
CHAIR: I want to close the microphones after these last two questions.
AUDIENCE SPEAKER: It's actually not a question. Gert Döring. We have a RING node since the beginning of this year and we had some discussions about abuse handling and how we are going to trust all these weird people on our network. And in the end we decide that had this is just a colocation customer and that's it. So, it's sitting on its own VLAN. It has no privileges of anything. It's monitored like any other customers and there hasn't been any case of abuse coming from that box. Which is very unlike all the other customers. So... as long as you keep in mind, this is not a trusted knock box but this is a box where other people have access and it should be siting in its own VLAN, then there is no problem there.
JOB SNIJDERS: Thank you for your contribution. The last question?
AUDIENCE SPEAKER: Hi. I was just wondering, basically decentralising all the servers between everybody's own AS numbers, did you consider to focus to forego Internet Exchanges on boxes over there that people can peer directly tots own demeanor form those boxing within the Internet Exchange directly to its network?
JOB SNIJDERS: So, did I gather correctly, you propose connecting ??
AUDIENCE SPEAKER: Folks like the servers are being placed now that people on an Internet Exchange can go to the Internet Exchange, give them access from the Internet Exchange directly into their networks instead of building systems which are separated to centralise them.
JOB SNIJDERS: I have the feeling that Arien wants to reply to this.
ARIEN VIJN: Several years ago we actually asked people, do you want to have a service like this? And the answer was plain no.
JOB SNIJDERS: Times have changed, apparently. I'd like to thank you for your attention.
AUDIENCE SPEAKER: As a former Internet Exchange, it's very useful to have this insight on AS, but on an Internet Exchange, you look at all the different networks. You are interested in what is happening through your packets inside in other complicated networks.
JOB SNIJDERS: Thank you for your time.
(Applause)
CHAIR: So, it's time for lightning talks. And I would like to invite Geoff Huston, for a ten minute lightning talk. He will talk about his DNSSEC measurements. Thank you.
GEOFF HUSTON: Seven minutes and questions. Not a problem.
Who is using DNSSEC? You are all lying. Because I know.
We decided to measure DNSSEC a couple of weeks ago. Because an if you other folk are getting interested in terms what is out there with DNSSEC? So, in this talk I want to answer three questions:
What proportion, not the count because I can't count, it but what proportion of the DNS resolvers are actually doing DNSSEC validation? So, of all the DNS resolvers out there, what proportion are DNSSEC?capable. But that's all about you, not resolvers. So what proportion of users are actually using these DNSSEC validating resolvers, and, last but not least, where are you?
So, the easiest way to test a lot of people quickly is with an ad. Because everybody receives ads. And the beauty about ads is that the advertisers demanded dynamic ads, which is Flash. And you can do a huge amount in Flash, including generating random URLs and then getting the user to fetch it. So has anyone been watching YouTube lately? Well, all of you have and enough that the ad which is embedded in YouTube basically goes and fetches those two URLs, and I won't tell you what the ad is because if you flick on, it I pay. I don't like that.
They look very similar except that one of these has an invalid DNS, that one. The other one has a valid chain and those numbers there are randomised. Every time you do the experiment you get a different number. So I em bed this in our flash code. I paid Google a very small amount of money to advertise it everywhere because as long as no one clicks on, it Google get very, very little money and everyone is happy.
Including, me. So, we ran this for exactly one week from the 10th to the 17th September. And we got a lot of answers.
So, the first thing is how many resolvers did I see if I make the assuming that each unique IP address is a resolver? Now I deliberately set DNS records in v4 only. V6 is another test and that's another lightning talk.
So, how many unique addresses did I see? How many unique resolvers? And of those, how many fetched the DNS KEY records in order to do DNSSEC validation. So, not bad. 57,268 and of those, 2,316 did DNS KEY requests. So, around 4 percent of all the resolvers that are visible appear to be doing DNSSEC validation. So, appear to be doing that. That's pretty cool. But I actually noticed a huge number of resolvers seemed to be very, very close to a small number of folk, clients.
So I split them out. How many resolvers served just one or two clients? 40,446, across quite a lot. How many of those did DNSSEC validation? Do you do DNSSEC at home? No. Because only 2.8 percent of folk actually, of the small scale resolvers, do DNSSEC validation. Interesting. What about the rest?
Well maybe DNSSEC is all about the big people. The big boys and girls. Because of the ones that are left, 16,822, only 1,180 do it, that's a shopping great 7 percent. Maybe the professionals do DNSSEC. So then I got the largest resolvers out there that served the most number of people. 8.8.8 does and does not. The ones that do, did about 36,000 clients and the ones that did not, did about 20,000. So I left them off the list. So Google does DNSSEC half of the time. Ripper. What about career telecom? That's a huge resolver. It doesn't. Nor does the next, nor the next, nor the next, nor do the Greeks or the Emirates, Pakistan, China. You can read that as well as I can. Isn't that amazing that the really big folk don't do it with one exception. That's bizarre.
So, let's flick our attention to clients. How many folk did I see who were using resolvers one way or another? How many participated and how many actually did DNS KEY?
Google is amazing. For a very, very small amount of money you can plant this code on 770,936 eyeballs in a week and spend 1,000 bulks. That's amazing. It's really, really cheap. Of those, 69,560 queried for a DNS KEY. 9 percent. Now, that's not the folk who actually apply. If you have a number of resolvers in your etc. Resolve dotcom F and only one of them does DNSSEC validation, you are still going to fetch the crap URL. That's your problem. But all I'm saying is 9 percent of folk were covered by resolvers and those appeared to be doing DNSSEC validation. So where are they on the planet?
Maybe it's a longitudinal thing, and that maybe Sweden and Libya have something in common, but, beyond DNSSEC, it's got me beat. But bizarrely, number one in the country stake for Libya is if the ?? is Libya; number 2, Czech Republic; Slovenia, the occupied Palestine territory, Azerbaijan, Djibouti, that is a really weird list. If you do GDP against this, it just makes no sense whatsoever, but to the good folk of Kurdistan, well done. Quite right too...
But if you name them, you got to shame them. So, Costa Rica, Uruguay, Croatia, France, but they are French ?? Spain, Australia is in the middle, the Netherlands might be doing fine but their colony over there in the Caribbean isn't, and of course Greece, you have to mention Greece, and there it is. But down there again in the major economies, Mexico, I am thinking what's going on? They are the bottom of the list. I don't understand that.
So, now I did the top ASs, who is actually doing DNSSEC at AS level, there is Serbia, Ukraine, Italy, Sweden and so on. And then I thought you know, I'm in the right region, aren't I? So I know all the country codes. So, there are you are. Sweden, Czech Republic, Slovenia. Well done. Qatar, San Marino and, yes, 18 people from the Faro Islands got the ad and did the test. Why, thank you. None of them did DNSSEC but by God its good to see them.
Thank you very much.
CHAIR: Thank you, Geoff, for this really lightning talk and it was right on time. We have ?? do we have any questions? Come on... Olaf, thank you.
AUDIENCE SPEAKER: Olaf Kolkman, NLnet Labs. So, what is your major take?away?
GEOFF HUSTON: My major take away is there is actually a huge amount of DNSSEC actually being used out there. But interestingly, it seems to be folk who have sort of bought the package, and some of the folk who have bought the package appear to come from countries that really want to see what people are doing. And some of them have come from countries that really don't want to see what people are doing and it's quite of DNSSEC is being used in both contexts at the same time. For those who are actually fighting the good fight about DNSSEC, this is really good news. That's an extraordinary amount of coverage in just two years from signing the root. You know, this stuff is working the resolvers are really doing a great job. However, a lot of folk use multiple entries in resolver dotcom. And while 9 percent of users use some resolvers that are used by DNSSEC in another lightning talk in another day, we'll actually talk about the folk who refuse to download an object which is DNSSEC invalid and unfortunately, that's a little smaller.
AUDIENCE SPEAKER: I was actually hoping for a different answer. I was actually hoping for the point you made yesterday in the hallway, that there is no excuse to not run DNSSEC validation because you can actually run it on fairly big resolvers.
GEOFF HUSTON: Certainly that's true. If Google can do this for a massive collection of folk, configure 8.8.8.8. Anybody in the room who says, oh, we don't DNSSEC because our resolver is going to melt has no idea and no clue, because some folk can do it and some folk are doing it. So, yeah, it's perfectly possible to have a massive client set and run DNSSEC capable resolving. Go do it. There is no excuse left. Thank you.
CHAIR: Thank you, Geoff.
(Applause)
Okay, next speaker is Lorenzo. He is going to talk about the world IPv6 launch event this year.
LORENZO COLITTI: I recognise that I am standing between you and lunch. So, I'll try to be brief.
So, really, we ?? I didn't see anything on the agenda about what happened on world IPv6. I thought at least we have solve something here, so I put in a lightning talk.
So this is from strictly from Google's perspective. We have a lot of data historically so it might be useful to share it.
What did we do as part of the run?up to world IPv6 launch? As you know we run at option measurements on daily basis we have got quite a bit of data now. We used this data to help power the tool that would allow the committee to decide who could participate and who was going to be listed on the web page and that was in order to ensure that networks with no IPv6 would not appear on the list. That was part of the bargain.
And so we provided data. We also provided data for the measurements the public wants that you eventually see in world IPv6 launch.org/measurements.
IPv6 is growing by about two?and?a?half times a year. This is on top of regular IPv4 growth. It will get to 50% in maybe five years. This is a smooth graph by the way. I just uploaded this right now.
It's interesting to see that if you smooth is on exactly seven days you get a much better looking graph. Because we see a weekday weekend pattern. That's a cubic curve, it's manually fitted, it's cubic for now. So...
I don't know what it really is.
We also measure IPv6 breakage. In the run?up to the day, there were certain very large networks that were deploying IPv6 that also had significant issues in their user race. For these networks we actually helped them by telling them exactly where the broken, the breakage was, and that wasn't scaleable but it was necessary. Now, of course that everyone is seeing AAAAs from major websites that's not so much of an issue. If you break IPv6 in your network today you do software and people notice. Before IPv6 launch that wasn't the case. In some cases, we actually were ?? these measurements were critical in successful participation of networks to the event.
The users with connectivity problems, we publish a list of networks where we see problems and where we disable IPv6 for transparency reasons and so that those networks can actually notice. We found paradoxically we found that it's more useful to put somebody's resolver on a list than to, and disable IPv6 for them, than to cause breakage. Because in one case, when the resolver is on the list the knock notices. If the users are experiencing trouble, nobody notices. Which is, you know, take that as you will.
Some people have reacted to IPv6 launch by telling their resolvers to drop all AAAA responses. This is a bad plan, because it doesn't resolve the underlying problem which is broken IPv6 connectivity on clients, and also, it completely, it makes it impossible to measure the problem. So, you don't know how to turn the filtering off. Of course there is a lot of policy and sensorship issue that I won't go into here, but you know, please don't do this.
You know, some networks have completely disabled IPv6 for all their users because they only have one set of resolvers and to you know, save the ones that are broken they also disabled v6 for the ones that are working.
This mostly happens in Japan, not quite. I suspect that one of the major ISPs in Greece may be talking about this later on and I look forward to their talk on what happened here. I think this is a bit too clean to be a software roll?out. It's probably some filter somewhere.
What happened on launch day? We ?? you can see this ?? we one of the measurements that our reliability team likes to look at it the number of queries per second that users do and you can see 75 percent increase there. You can see that the trough, the new trough was higher than the previous peak, so that's good. And we saw some pretty impressive rolouts. This is RCS, RDS from Romania. I think they'll be talking later on, they got v6, 23 percent of their users. They use, I believe, PPP. AT&T using 6 RD, they have gotten to 7 percent of their users, they published they might reach 5 million by the end of the year, this is XS 4 All, you can see at the time of the launch they started enabling v6 for more users an they are up to 10% now. This is [Rise] on wireless. This is LTE dual stack phones. Almost 20% of their phones using v6 on a daily ways I say. This is really impressive really. A lot of that runs Android but a few days ago, something was shipped and those devices are using IPv6 as well now. Although not very much, because they do happy eyeballs in a very ?? in a both ruthless and biased way, they bias against v6 and then they take the fastest connection which leads them to prefer v4 most of the time.
So, in conclusion, I think we sort of achieved what we wanted to do which was to try and get the Internet moving words to IPv6. I think we actually had real impact. We got thousands of websites, you know, tens, dozens of ISPs and even home vendors to leave it on. If you deploy IPv6 to your access network, 40% of your traffic to v6 users will be v6 traffic and so we believe that that's you know, it's certainly better incentive than it was to deploying IPv6, okay. And we have seen real deployment everywhere around the world on virtually every access technology so the next time somebody says oh you can't do IPv6 on LTE or you can't do IPv6 on DSL because the DSL don't support it, look at one of these operators and, you know, they will prove you wrong.
That's really all I had. So, hopefully we can go to lunch on time.
CHAIR: Any questions? I can't believe no questions.
LORENZO COLITTI: We launched it and it worked.
(Applause)
CHAIR: Thank you, Lorenzo. And now for our last speaker, Todd will tell you how hard we have to work at the Programme Committee to select all these talks for you throughout the week and, Todd is the Chair of the Programme Committee and he will explain everything to you in a minute.
TODD UNDERWOOD: So I think I am going to speak very slowly and use up every second I have to prevent you eating lunch. I am going...
STENOGRAPHER: No, please go slowly...
TODD UNDERWOOD: So part of this is wholesale text support. So we're recruiting a lot of new members to the Programme Committee here, because there are five slots coming up so if you are interested, there will be something about that for new this presentation. We are also recruiting a lot of talks for the next RIPE here, so if you are interested, there is something for you in this presentation and we are getting a lot of same questions from lots of same people, so we decided ? well, we didn't decide; the rest of the Programme Committee decided that I should give a presentation about this.
So, yes. There may have been ?? anyway, I didn't finish. So, I was going to fix, but this ?? but I put in the British spelling, the unnecessary ME at the end to make you guys comfortable.
How I write Plenary is born. So when a presenter and a conference love each other very, very much ?? networks we are not going to talk about that.
So, look, a RIPE Plenary programme is built, it's built by the Programme Committee selecting talks that you, the members of the community submit in an effort to support the policy and engineering work done in the Working Groups. In short our goal is ensure that there is sufficient technical discussion and education about the important issues that the Working Groups are working on so that people can make informed policy decisions. That doesn't always happen. Sometimes it doesn't happen because you lot don't present interesting, or clueful or cogent things, sometimes it doesn't happen because we are lazy and we pick the wrong things. You can fix all of that. This presentation is about demystifying both the Programme Committee and how it works and the process of submitting talks.
Yes, there will be information on the current location of Elvis presented before the end of the presentation.
Programme Committee, where forth are thou, Programme Committee? We are, we are there primarily to recruit and select talks. The recruiting part turns out to be the more important part of that. There is a reason for that. If we do not spend time recruiting talks, the entire Plenary programme will be filled by talks produced by the marketing departments of your equipment vendors. Because, those are the people who are paid to produce presentations. And they are not bad presentations. They are really not and we do accept some those when they have legitimate technical content. But, the rest of you who are here who don't work for equipment vendors or don't work for service provider marketing departments who do some sort of engineering or operations for a living actually have awesome things to talk about. It's just that you are not paid to talk about them. You are paid to do them. So our job is to use this sort of emotional, mental, professional Jiu?Jitsu to transform your experiences into pixels on a screen while you stand here and talk about them. This turns out to be fairly difficult because many of you don't want to talk about the things you do. But we are going to fix that.
So, look, Plenary content should appeal to network operators, anyone in the RIPE constituency base, it should coordinate with the engineering and policy content. I said all this. It should educate, entertain, I should be wearing a tutu, I'm not.
Features and benefits, so the requirements: I am getting a lot of questions about this from people who think me might want to join the Programme Committee but are afraid they might have to do something if they did that. So, it is true, you will have to do something. It's true that the burden is not enormous, so, you should recruit one to three talks per RIPE, five would be better, none would be disappointing. You will have to read and write about 30 presentations of enormously varying quality, occasionally, and I'll talk about the enormously varying quality in a second.
You will have to attend three conference calls, and I have a special note for Americans, people who live in anything in the sort of more than six hours off of UTC, the calls are convenient for Europeans. That's fine. If you live on the US west coast and you don't want to either get up extremely early or stay up extremely late, this may not be the Programme Committee for you. We did a conference call at 5 a.m. my time one time and I lived on the east coast. So, calculate your math accordingly. The Programme Committee controls when it does its own calls and there is a chance if you live in California, and you were extraordinarily persuasive you could convince your European colleagues to have a call in the afternoon but there are people in UTC plus 2 and 3 who may not be interested in that either. So plan accordingly.
And there is an enormous amount of communication overhead primarily with presenters. What's wrong with my presentation? Here is some feedback, can we work on that? Here is a panel, oh, I can't come because it's a Tuesday and I have a religious objection to working on Tuesday, etc.. so you have to work through that.
There are some benefits. You get free lunch at all RIPE meetings you attend. You have to pay to go to the meeting, but do you get free lunch once you're here to that's really special for us.
The main benefits are, you get some visibility, I'm not actually sure what good that is it's never gotten me a job or any money, it might do something for you and you get to contribute to the community and I think one of the things I have heard from people at RIPE is when the content of the Plenary is not very good, people are upset. People want a better programme. And if you have ever complained about the content of a conference you have been to, if you have ever complained about the content of RIPE. If you are complaining right now about the content of this RIPE, kick yourself for not having submitted a better lightning talk.
Okay. So now we have gotten to get Todd off the stage and join the right PC slide. This is really straightforward. Send mail to pc [at] ripe [dot] net. Send a bio and a statement of interest. Those will be published. So it should basically say who you are, why you want to do this and why anyone should let do you this. It's not more complicated than that. The picture is useful if ?? if you had conversations with people that didn't go well, you might not want to include a picture, but if you have had conversations with people here and they think they know who you are and you are not useless, then you should include a picture, because then they will identify that as awe. You have to register for a RIPE NCC access account. Then you'll vote on Friday and it's going to be fantastic.
So let's say you want to present at RIPE. This is straightforward. You rate an account in this meeting system thing, which is a totally different account because authentication is hard and merging these systems is hard and this is all sort of slapped together. But it will get better eventually. So you submit a presentation with slides and I have two comments about that in a second. You practice your presentation in front of a mirror and you read Sander's wonderful slides about how to do that. Then you wear a snazzy T?shirt to impress your friends and stand up here and mutter a little bit. A word about draft mentions. We have got a fair amount of feedback from presenters who say gosh, the conference is a long way away and I can't be bothered to put together some slides but accept my presentation anyway.
And here is what invariably happens. Either you are one of the cool kids who always gets to talk at RIPE in which case people will say I am sure it will be fine and accept it, which I regard as anti?democratic and unfair and also providing mediocre content sometimes. Or we spend, we figure out what the presentation is about and inevitable write back and say could you please submit some slides because we don't get it, so submit some slides. What do the slides need to contain? They don't need to contain results. They don't need to contain sort of fancy pictures, they don't need to be enough to fill a reasonable amount of presentation. They could contain unicorns and/or ponies and/or rainbows, but they don't have to. Although they are always appreciated. But they have to tell us a story. So, here is a reasonable set of slides. DNS is evil and it should be abolished. Title slide. Slide number one, I am convince that had DNS is evil for a variety of reasons. List reasons here. But you don't include the reasons. That's it. Results slide one, results slide 2, DNS is bad for networks. Results slide 3 and 4, DNS is bad for society. Results slide 5 and 6, I conclude is DNS is evil and I hope you agree with me. Questions.
Okay. Like, as a PC member, I may be unconvinced by your argument, I might be convinced but I at least know what you are trying to present on. That's enough like there. I will submit that presentation actually now that I mention it. Because DNS has got to be evil, especially that rude DNS sexy version that, sounds all of.
Okay. So improving the right Plenary. Even if you don't want to join the PC and you don't want to submit a Plenary presentation, there are several things you can do. You can rate the talks as some of you have been doing now that we are bribing you to do so. You can send us e?mails with suggestions about things that went well and poorly like I didn't like that guy as T?shirt or you know, so and so muttered and can you teach people how to not mutter. Whatever the feedback you have. We are here in person and we have like this PC dot thingy and there is people around over here and so, you can say Hi to us in person and say something to us in person you are not willing to say in e?mail. You can suggest presentation topics or specific presenters and this is enormously useful because it's our experience if someone comes up and says this person is willing to present on this topic and I know this they're good, then that is 80% of the way words to cajoling that person into presenting at the next RIPE. So identify presenters and topics for us. Consider joining this PC and consider presenting and specifically consider presenting lightning talks on Friday because we still have three open slots and only a couple of presentations.
Any questions?
You guys knew that Elvis is dead. He is buried in Graceland. It's in this meditation garden in Graceland, I don't know why people think he is alive. He would be really fat if he was still alive too.
CHAIR: We are still standing between you and lunch, so any questions?
TODD UNDERWOOD: Thank you very much.
(Applause)
TODD UNDERWOOD: Did you guys want to announce the winner of the gift cards or ??
SANDER STEFFAN: I think it was after lunch. So see you back at two then.
(Lunch)
