These are unedited transcripts and may contain errors.

MAT Working Group ?? 27 September 2012 ?? 4 p.m.:

CHAIR: Welcome everybody. Good afternoon. Please come in and take your seats. Settle down. This is the measurement analysis and tools Working Group. My name is Richard Barnes. I'll be chairing today together with Christian and Ian. I'd like to thank you our transcriptionsist and Mirjam for scribing for us. We have a nice agenda with some talks from the NCC on current activities and some discussion about how to take event of this stuff and some work that Martin's been doing in mapping out inter AS topology at the country level. I have been asked to pass us over to Brian for a minute with a non MAT announcement of more general interest.

BRIAN NISBET: Hello. With my Programme Committee hat on, very briefly, the nominations for the Programme Committee have closed. We have ten nominees. Their bios and pictures are available on the RIPE 65 website, under Programme Committee, you can look at them there. And the voting will take place in ?? should be towards the end of the first Plenary Session tomorrow and we'll explain voting systems then. It's simple. Look at the nominees, we have five slots open and please turn up tomorrow and vote. Thank you.

We said that a number of times. I'll repeat it again. You do need a RIPE connect account to vote. It's ?? a RIPE NCC access account to vote and because you know we are obviously harvesting, or the NCC are harvesting all your information for nefarious purposes. But if you don't have an account you can't vote. So thank you very much.

RICHARD BARNES: Everyone please remember to vote for the Programme Committee. I think that's the ?? all I have for intro. Martin, are you here to get us started? Martin Levy, come on down.

MARTIN LEVY: I realise this is the last one of the day, so... a light crowd.

I forget they write that down.

Hello. I am Martin Levey, Hurricane Electric, the most important thing this talk is not about v6.

So, this talk is about pretty diagrams and it's a work in progress that is based upon all the collection of data that we have been doing on our BGP dot site so. I'm going to run through some stuff, just the methodology of what we're doing, some pretty pictures, nice and easy forth end of the day and then just a quick summary and then you can decide whether this is a good idea or not.

So the idea is to visualise in some useful manner global routing. And this caveats, this is really where you already start by apologising. BGP is not the be all and end all of trying to look at networks. But, it's what we have got and what we have been playing with and this data is from that source. We use a lot of data that gets collected from RIPE RIS, from the project and we are thankful and we have been generating and I have talked about before these types of diagrams, these simple BGP propagation diagrams. They aren't a hundred percent complete because they are based upon the routes that are actually installed and used and therefore propagated, there may be other routes you just can't see.

But one AS connects to the next AS connection to the next one, etc. That's not the part I want to talk about. The question we had was could we use another of this data in some aggregated manner to look at whether we could do in country or regional diagrams and actually see where people are connecting and see something that makes sense. So, the goal was to do everything on a per country or a per region basis as opposed to a per AS basis.

I said it before, I'll say it again, we are very thankful to the RIPE RIS community, we are very thankful to the Oregon route views guys because that data is key for this and that's what we have been processing, normalising for many years now, and what we use for this BGP dot database.

What you see online is first step, what I'm going to try and show you is what we want to add to it.

So, methodology:
Take the data on a per country basis, take each AS that we find in a country and decide whether that AS connects inside the country or outside the country. Very simple. So there are two pages just to use sort of a visual approach to that statement. The first one is, look at all the ASs listed in a country. We do this today. There is app simple page for every country and every active AS can be ordered by some useful manner but basically listed. So here is Malaysia and then if you take a particular backbone, you can then look at all of the interconnects and because we can look at the country code that we believe the AS is associated with, we can decide whether that particular network connects inside a country or outside a country.

Now, before I go any further. The UK is not a country code. You have to sort of write and make sure that you don't get things like looking at the ?? at England and realise you have got to do Great Britain. So what can you see does this look like? You go take ?? you take a listing for the UK or Great Britain in this case and you actually start looking at all the adjacents and you sort things out and then you say, okay, fine, I'm going to take that listing and I am going to generate something on a per network basis.

Now, just to run through this. What we were showing were these types of diagrams. Basic, simple diagrams for each of the ASs that existed, very bothering, what we wanted to do was something on a per country basis. What you end up with is this. I hope that's all readable to you. But actually, these diagrams are kind of cool and you have to squint at them. So, right in the middle here, we had the rest of the world. This is every AS number that we could find that wasn't registered in the UK in GB, and then from that, we started to look at all of the connections to ASs that had international connectivity. There is a bunch of red ovals here, I'll zoom in on this in a moment, and they have all got some form of international connectivity. Around here we have got some orange ones that we couldn't find any connectivity within the UK but we could find it to international ASs and I'll talk about what that is in a moment. Finally at the green here we have ASs that only connect or we see only connecting with inside the country.

Now, what's interesting is you see these sort of hot spots here where there are different networks and sure enough they turn out to be the classic wholesale backbones or the incumbent in some form or other.

Well, what's also interesting about this diagram, I'll talk about the colour scheme because that comes up quite a bit. A blue line means we see a v4 connectivity. A red line means we see v6 connectivity. And a magenta line says that we actually see v4 and v6 between those two ASs. Zoom in: You still can't read, it I know this, the file is just way too big. But the reality is that we get to see this sort of relationship. We see somebody ?? this so happens to be, I can read it from here ?? this says COLT ?? if you get online, you can read this ?? this is Virgin Media, so these are the sort of large networks and they have lots of connections and you end up with this nice diagram, but remember I like to look at v4 and v6. So the first attempt to do this was this.

This is only v4. This is only v6. Simple statement: This is sparse err than this. There is less v4 interconnect ?? there is less v6 interconnectivity than v4 and no one is surprised at this statement. It may annoy people but no one is surprised. A simple squint test without being able to look at this says they don't match but we had to look at a better way of drawing this after a while.

Let's look at some other countries because it's more interesting. This is hungry and in fact actually we start to see one or two or three, maybe four main players, here is the rest of the world depicted here, and a lot of the networks in hungry are not connected to other ASs that are global. Now, this makes sense. We have gone from the UK where there is an extremely witch Internet peering infrastructure with international carriers from umpteen countries to hungry that has a really cool peering point in Budapest, but has a much smaller base of countries in there. So, this sort of again, A, B comparison seems to make sense.

Zooming in a little bit here, yes, you can sort of, this makes sense when you show this to, A, to people that are local there, it makes sense, but there is a caveat and there is a caveat on everyone of these before you come up to the microphone. These diagrams with not complete. They can actually never be complete because you can't collect enough route information, whether it be via RIPE RIS or or a gone or anything to see all the relations ships between all of the networks. But it seems to make sense.

Here is Austria, slightly more dense, got slightly more peering. So, we sort of see more networks connected internationally than we do just connected in country. But, not too bad.

Here is the Czech Republic which has a very active exchange point. I did not put the diagrams up for the Netherlands with AMS?IX with Germany with DCix, it's pretty big. I want to go back and talk about one key point.

This is v4 versus v6, reorganised so that the diagrams are identical and the colour schemes are identical and sure enough you see around the edges here, you see v4 connectivity between some countries and then you see other places where you just don't see anything, and if you look at these on the website, the PDF file you can actually read what these are. But you'll see things like Albania, you'll see Serbia connected to it down here, you see Serbia, etc., etc.. this is totally unconnected. It's some random island north of Norway whose name I have forgotten.

Zooming in on this, zooming in on this I want to show you that blindly accepting data is a problem. I mean, we're here talking in a MAT group so this is actually the meet of the problem. Pretty pictures are great but it turns out this is a problem. Well, actually ?? yeah I know ?? this is a problem ?? let's not go there.

So, the European Union, the country code for the EU two letter sequence is used with inside the RIPE database to define a whole bunch of attributes, IP address blocks or ASNs. Well, it turns out there is about 223 ASNs marked with EU. And if you look at them you sort of realise, okay, some of them are pretty universal around Europe, maybe not EU but Europe and then others just are left over mistakes or requested that way. And it turns out this sort of ruins the diagrams a little bit and it ruins the diagrams not only in the core but also on the edges, because earn countries get left in the dust. So here is a couple of examples of the ones that, when you actually look at this. So we found a whole bunch of very early ASs, Austria was very much front and centre early on in the world of Internet access here. And there is a bunch of them that just simply sit in the RIPE database with the EU country code, I can't use the word country code, EU code. Here is a bunch in Finland, these guys really got into it including the prime Minister's office. But they are all marked as EU. Everybody should be online.

Here is some ones that are much later in the game, but again they are marked as EU as opposed to SI for Slovenia. Here is a couple in Italy, wind may actually want to consider itself as a European player, but I sort of said no, maybe IT. So, and then the GARR, Italian Academic Netowork. So again this is historical reasons. So be careful when you use data, you realise that it actually messes around with the diagrams. There is many more by the way, there is ASs that have yet to be transferred to be AfriNIC, for Tunisia for example, to it looks like Tunisia has nothing when you look through AfriNIC's database or look for Tunisia as a code. It turns out it's there. Again, this is the problem that squinting on the screen is not too useful but the bottom line is it changes the v4 /v6 diagram. If we zoom in it turns out the European Union truly is a smaller bubble and everything else gets more interesting. Displaying this much data, really hard. This is not meant to be brilliantly presented. It is just that there is a lot of data there, but you can see it. But this is only part of what we wanted to do. We wanted to look at places that were more interesting. So this is looking at the African continent. This diagram now takes the rest of the world and breaks it into different country codes. By the way, you'll see, here is South Africa, very well contacted because there is quite a few players that come back out to Europe in some way, you see there is Kenya in here, fairly well connected. And then each one of these black boxes was a different country that was the upstream or the peer of that particular, of a collection of ASs in that country. So that's have a look at what we get.

What we get is European colonialism, shown on BGP tables. Here is Cameroon. French provider. Here is Malawi and Sudan with Belgium and here is Angola with Portuguese upstream. This is history. This is every kid should learn this history, who knew we needed to know it for BGP?

You can play around in different regions. Africa shows this particular feature more than anything else. But the net?net of this is that you can take some of this data, somewhat with a grain of salt, definitely with a lot of edits to make sure that you actually have the right data and to generate something that gives you an idea, both maybe politically and simply network operation wise, what's going on.

All right. So, here is the question: Can you really do this with just BGP data? And answer is, sort of. It turns out if you went to the NLNOG ring talks and looked at that data you sort of go, oh, there is much more data you could overlap because a BGP adjacentey is a BGP adjacent see but it doesn't mean in fact actually that there is a direct path between networks in country A versus networks in country B. In fact actually you can quite easily see what at first hand makes sense. South Africa has a bunch of ISPs and you see peering with a few ISPs out of Egypt, and you go, I know why. There is an undersea cable that comes up the east coast of Africa and goes through the Suez canal, Egypt, and then goes off into Europe. It makes perfect sense. Well it turns out that isn't the case. In both cases those ISPs end up in Europe, they peer in London or Amsterdam and the traffic trombones, just like we were discussing with Asian traffic 12, 13, 14 years ago, European traffic 15 years ago. It's the same story that we have seen, so BGP makes for interesting diagrams, but not much more.

And, you have got to be careful, so you actually want overlaying this with traceroute data would make some of it more sane.

That's it, it's a bunch of pretty pictures, it's a nice idea. It will go life sometime soon and we can all squint at different pictures for different countries. And thank you, thank you for the time.


RICHARD BARNES: Thank you, Martin. Are there questions for Martin?

AUDIENCE SPEAKER: Daniel Karrenberg, doing measurements at the RIPE NCC. Thank you, very nice pictures. A couple of suggestions, since you are giving suggestions about looking at the data quality. Definitely the first two letters of RegIDs are not a country code so that explains why cannot see it because they are not. But they are better data sets, we have worked offer the last two years to make something called RIPE Stat, which you might have heard of, and it has a number of data sets that, if you prudently use them, will give you much better results with some euristics and what's more, all these data sets have an API, a data call that you can actually use dynamically to query so, just to say that, and it's also third party data like what comes from MaxMind and so on.

Now for the question of course, there has to be a question. You said even if you use route views and the RIS, which is a humongous amount of BGP data, you do not see every adjacent see that there is, could you, if I were to ask you how can I make RIS better in order to catch more of this, what is the strategy to follow? Where are the gaps? What kind of peers do I need and things like that?

And the second thing is, you said you would enrich and verify your work with the use of actual packet data, like we would have from Atlas, any ideas about how that kind of merge could work. But I am much more interested in the first question. The second is maybe one we can discuss in the hallways.

MARTIN LEVY: I'll answer the question about the country code data. Very quickly. We are actually using the country code data that gets exported and delegated by all the RIRs, which is nearly the point you made is that you can't exactly use that data. It has a pretty good analogy. The algorithm we use forward fixing the EU data was to turn around and say, if we saw an AS number with EU, and yet all the IP space announced from it, v4 and v6, was from a very specific country code, only one country code, then we knew that the EU could be replace with that country code. And there is a few other things we can do but that's a really raw piece of code and I'll hit the rest it have off line with you. But the two points: Could you get better data in RIS or in organ route use, the answer is absolutely. All you need is to take 39,000, 39,000 active AS numbers and send them an e?mail and say I don't have you on a route RIS feed, can you go to the peering page and set up a session. Technically that's all you'd need to do. But ??

DANIEL KARRENBERG: My question was more in which sequence? Given limited resources, in which sequence or by which strategy to maybe attack the first 2,000 of them?

MARTIN LEVY: There are a couple, if you even look at the ranking of ASs, large ASs downwards, you know, anybody that's missing from that point will help, but that's still only gives awe transit or a feed style look. It doesn't give you much. What we found was taking certain countries and getting a well connected AS in country to convince them to feed even on a remote feed into Oregon or into RIS would suddenly explode all the connectivity inside that country. So we did this Mozambique and we could not find any data until there was a feed out of there and all of a sudden we could see all the connections. That's a special case. You'd have to repeat that many times, it's non trivial. But exchanges that have the usual ?? it's where all the ASs seem to end up, the good ones in the country, means that you want to have an active collector, actively going after the peers in Kenya, South Africa, in south east Asia, in Latin America, and this is slowly mapping, but maybe many more collectors. The traceroute thing I'll take off line with you, because that's a much ?? it's cool but it's an off line subject.

AUDIENCE SPEAKER: Wilfried Woeber, Database Working Group co?chair. Just a little bit of background regarding the country codes in the registry data. If I try to remember all the potential sources for these "Wrong" things, there are at least two of them, probably more. One of those backgrounds is the early registration transfer where quite a big bunch of those pieces of resources from ARIN database were moved to the RIPE database and as nobody had any better ?? well this is going to the EU RIPE database, we just put EU. We can, together with the IPRAs, we can more or less easily get that corrected, updated, made more correct. There is another thing and that was actually a discussion that we had, I don't know, how many years ago, three or four probably, pondering the question sort of, would it be prudent to do away with the country code attribute at all in the first place, and we did receive feedback from people doing statistics and all that sort of things that you are doing, that no, please keep, it we know that it is not complete, it's not correct, it's not completely credible, but it's good enough for the moment. But I seem to be hearing from your end is that we should eventually revisit that, asking the question: Is this data quality good enough for you? And I think that should be an open discussion.

MARTIN LEVY: Yes. Two things. Don't remove it. It is useful for people doing statistics. Don't remove it because it's a resource and quite frankly, it should be something that should be tied to some geographic and the more geographic data the better, but some is good. A caveat can obviously exist. The third part is at the moment, a resource holder of an ASN cannot edit their country code and that actually is ?? I know why. I now know why. They can if they have an IP address, but they cannot with an ASN. They can through a help e?mail to RIPE themselves. Now, the only thing I would suggest, but it will break a lot of people's software that looks for EU and therefore tries and corrects it, it is that EU is just the wrong code, but we are 20 years too late for that discussion and let it go. I mean, even I will say that. But keep up ?? keep the good work up, it is useful. There are clearly ASs that are not bound to countries and there are clearly ASs that are bound to countries only, there is some in the middle of course.

WILIFRIED WOEBER: I take it to be one of the proposed agenda items for the next meeting and maybe we can even before that start a discussion on the Database Working Group.

MARTIN LEVY: And I'll say that by visualising the data on a per country basis, the brain has this amazing ability to say that shouldn't be there. Whereas looking at a list of 38,000 ASs, numerically, alphabetically, whatever would be useless. This has a rather interesting method of finding things that are obviously missing. It's the brain using visuals.


RICHARD BARNES: Any other questions? Martin, is this going to go up to

MARTIN LEVY: That's the plan. We keep dumping more servers in front of BGP.AG. This will take some more servers and the algorithms. So, it goes back to this whole thing. We don't want to put stuff up unless we feel happy with it. There are definitely still a few hiccups in the graphs but they are getting there and they are way better than they were a month ago. They'll be probably better in a month's time. I am trying to aim for a month or so, and and then you can click on any country and any region and have fun diagrams. Regulators love these diagrams, so that's why actually we are very careful about getting them right.

RICHARD BARNES: So once you get that up I might suggest you send a link out to the MAT list and possibly the database list as well, so that you know people can start putting their brains to work, seeing ??

MARTIN LEVY: Hurricane Electric has never been shy about telling people about stuff.

RICHARD BARNES: Thanks a lot Martin.


RICHARD BARNES: I think it's Vesna's time to give us an update on all the NCCs projects, all the different measurements they have got going on.

VESNA MANOJLIVIC: So, I have four talks, it will be Vesna's hour or not, if the Working Group Chairs stop me in time. I want to give, first, the strategy talk, then the RIPE Stat, then the Atlas update and then the Atlas anchors.

So before the slides go up, my name is Vesna. I am a community builder for measurements and tools at the RIPE NCC, and one of my tasks is to inform the community about the features that we have in our tools and measurements and the other one is to collect feedback from the users and bring it back to developers. So, we don't have much time right now for the feedback, so, if you want to tell me something, please see me after the MAT Working Group.

The first talk is high level strategy for this year and next. And I want you to meet my colleagues. Most of us are actually here today, so you can try to match the photos with the faces here in the audience, except for a few that couldn't make it. One of them is Robert, the other one is Anthony, and I haven't seen Bert, but maybe he is somewhere around. And some of them you also saw on Monday when we had the BoF and on Tuesday when we had the BoF. The little acronyms on the top stand Tore measurements community building, global information infrastructure, global Internet infrastructure, I always get confused there, and research and development.

So, the highest, highest level of our strategy is to consolidate. In the measurements area, we only want to end up with RIPE Atlas and RIS and with data presentation we want to have everything in RIPE Stat.

Our two main target audiences are the community and the next one is hike membership. This is the coverage. This is currently where we have Atlas probes distributed and we implemented two more measurements for hosts who have these probes, so the hosts can actually perform currently four different kinds of customised measurements. I'll get to more details later when I have a presentation about Atlas. So if you are a RIPE NCC member, you can do the one click test your IPv6 measurement, and then you get these beautiful drawings, diagrams, which are interactive, you can click around and it shows you the traceroutes that were completed towards the target destination and the traceroutes that were not completed and you can then use that for trouble shooting.

The newest project that we have is called Atlas Anchors and we'll have more talks about it today, and we also have beautiful tattoos that some of you already received and you can ask for them later also. So we have plans to deploy about 50 of these anchors by the end of next year.

Now, moving on to RIPE Stat. The RIPE Stat is one stop shop for all the information that you want to have about IP addresses and about AS numbers. You can query RIPE Stat for the data that RIPE NCC is actually authority for, but we also host a lot of third party information. So, this is ?? most of this is totally open for anyone to query, so this is the major benefit for the community. And one of the newest features that we implemented is the prefix side distribution which was just presented in the Routing Working Group.

If you are a RIPE NCC member, you get some additional services and this was just a demo, we also presented at this RIPE Meeting, you can see the history of the allocations. In RIPE Stat, you get a lot of beautiful visualisations, we build it in a modular way so there are widgets that you can imbed in your web pages and there are also data calls that you can use to make your own scripts.

The plans for next year: We will make the data presentation easier to actually digest because we realise we want to show you everything. We have a lot of data and people sometimes get confused by looking at it and then they give up so we want to make it easier to look at it.

Routing information service: As we saw from the previous presentation, it gives a lot of raw data to the researchers. However, we want to show the pretty pictures to the people who don't want to write their own script, so we are moving the user interfaces more and more into the RIPE Stat.

And these are the services that we are going to consolidate into Atlas and RIPE Stat. One of them is test traffic measurement. We announced already several RIPE meetings ago that we want to move away from the D TM and most of the functionality will be carried by Atlas Anchors.

If you have some questions about this, you can also see us after the session. And DNSMON currently is just operational service. However, our plans are to also move this into Atlas and Atlas Anchors plan to have functionality of DNSMON built on to them.

The data presentation will also be in RIPE Stat.

So, if you want to learn more about this, we publish a lot of research and a lot of our plans and all the descriptions of the new features on RIPE Labs. It looks quite colourful and we inform you on Twitter about it and we also take comments on the RIPE Labs, you can each us on Twitter, you can find us on many conferences, so we are open to the feedback.

Any questions?

RICHARD BARNES: So, questions for Vesna on the overall strategy? Okay. Thanks.

VESNA MANOJLIVIC: Then I'll move onto the next one. The next one is RIPE Stat. So on Monday we had a whole hour of the interactive demos and discussions about the features of RIPE Stat and now I am going to do a short recap and then we have also a surprise.

So, as I mentioned, the RIPE Stat is a web interface and a data repository that you can query using data API and we want to develop it further based on your feedback, so please let us know what do you find useful and what would you like to see in there.

What did we develop since the last meeting? I will try to use the laser here. So, as I said, we publish everything on RIPE Labs, so here are the links to the articles. You can see the registration history in the object browser. Object browser is quite an nifty tool. I was happy to hear Martin say how our brains can actually deal with visual information much better than the textual information, or at least some people see the visual information in a different way. So we have tried to show the objects in the RIPE database in a sort of Lego trademark way, so you see little colourful blocks running around and you can click on them and browse through them and now you can ?? we have also added a history there. So it's like a time travel into the past and you can see the details of the objects how they used to be.

Then we are making another interface towards the WHOIS database abuse finder function, so, currently you can query RIPE Stat to receive abuse information. We asked for feedback in the Anti?Abuse Working Group and we are going to work further on this. We also introduced this RIR prefix size distribution as an interactive widget so you can query for any prefix and it will tell you what are the sizes of the prefixes that are registered for that specific address range, or you can go to a page which is generated once a day for all of the RIR delegated prep fixes.

Then we also introduced another one, ASN neighbours history, and we showed most of them in the demo.

So address some the concerns that were raised in the Database Working Group, the privacy controls for the historical information. So, if you are a member of the RIPE NCC, you can see the historical information but without personal details. And there is the rate limiting, you can only do 1,000 queries per day. And this rate limiting is implemented based on the RIPE NCC access account. So, only members of the RIPE NCC can authorise their employees to query the history of the objects, and again they can only see non personal data. If you only create an SSO account yourself, there is a rate limiting, and if you logged in without any credentials, you don't see personal data at all.

So we added many many more features. And some of them are documented on RIPE Labs and some you need to research for yourself.

Now for the plans for the future. This year we want to focus on the user interface. I already mention it had too many widgets, we are going to split them into different categories. It's coming up on the next slide. For different kinds of people who don't like graphical interfaces and don't like widgets at all we want to develop the command line interface so that you can do your own scripting.

And people like to see information about their countries, yes, the regulators, but also people who want to see the competition between countries, so we are going to create special pages about the country's specific data.

For next year, we are going to include a lot of smaller tools that were developed over years by the RIPE NCC, and that we can not support any more. We are going to fold them into RIPE Stat and make the RIPE Stat even richer.

The more and more of the results of the RIPE Atlas measurements will be visible in RIPE Stat and in addition to the i?phones, we are also going to make the support for Android.

So these are our plans. Again, we are listening to your feedback. We will watch all the reruns from all the sessions and try to hear what you told told us and we'll base our plans on that.

This is a sneak preview. It is already available on RIPE Stat, not on the main page, but if you get the normal results you can also click somewhere away and see the new version of the interface and so you get first at a glance, a view with very few crucial widgets and then you get different categories, and then next step is we will make the personalised choice so you can choose whatever is important for you to see in your tabs. And those are the plans for the next two months. And then we'll see when we introduce this, what do you say is the most important thing to work on next.

These are some of the channels, how you can reach us. The slides will be online. So, I want to move onto the fun part. We announced a quiz on Monday, and now we want to announce the winner. Christian is coming with the present and with the choice of the people who actually answered the quiz.

CHRISTIAN: I think you all know about the quiz and about the prize that you can win, it's a bottle of Saki, and we'll we got some people actually sending in some kind of solutions for the quiz. In the end we decided to take all the solutions into account that are at least 60% true. I mean, for the first question that was about ?? it would involve WHOIS information; it was the net name for a prefix. Well, that was quite easy. The second question was about using the tier location widget, or basically tier location data. We were asking for a prefix that was not in Indonesia and was also not in Netherland, and there was another country mentioned for this specific prefix and that was India, and so that was also quite easy. The third question was a bit more difficult, and it was not ?? well, not everyone got it right. So, we just tried to get all the questions, or the solutions that got the first two right into account and we have four people, and Vesna, please draw a winner.

It is iney grow. It's totally random, our favourite user, come up here.

CHRISTIAN: Well, congratulations, you were actually under the two people that got the answer completely right, so, yeah... well done... and I hope we can enjoy it afterwards.

VESNA MANOJLIVIC: This is teach you that it actually pays off to use RIPE Stat. So use it more in the future and maybe you get a bottle of Saki next.

RICHARD BARNES: Any questions or comments for Vesna on RIPE Stat, things you love, things you hate, things you think should be different or better. I think we had a BoF on that. Unfortunately we can't offer more Saki in exchange for comments.

VESNA MANOJLIVIC: Moving on to RIPE Atlas update.

Since I have all these presentings there will be some repetition to again I am going to say what RIPE Atlas is. We have deployed a huge number of probes that are doing active measurements 24 /7 from around the world and we collect this measurements information and make it available to the research community to the hosts of the probes and to the members of the RIPE NCC and we are creating the biggest active measurements network in the world with your help.

We want to make this a truly community effort, so we are publishing updates and road maps on RIPE Labs, on RIPE meetings every time and we are asking you for feedback and then we are reacting on what you told us. So, let me give you a status update first.

So, we have more than 1850 probes up and running. This is the number from last week so it's actually increasing all the time and we have distributed more than 2,000 probes but they are in transit and we are waiting for them to actually come up and to be visible. So, if you apply for a probe right now, you will have to wait less than a week for us to send you the probe. If you are actually want one while you are at the RIPE database meeting you can come see me or Alistair after the MAT Working Group or you can see me tomorrow and we have some probes here and we can give it to you.

So, as I said, data is available to everyone. The user?defined measurements are only available to the probe hosts and to the RIPE NCC members.

If you want to host a probe, it's very simple. We made the procedure even easier, and you need to have the RIPE NCC access account, and so, there is about more than 3,000 users of Atlas, and 1,300 of them are from the RIPE NCC membership. So if you are an LIR, you can receive extra credit for performing customised measurements. You just have to ask us and then we'll give you 1 million credits.

Without those credits, you can still perform this very specific measurement which is called: Test your IPv6 connectivity. And you only have to type in the destination and we are going to grant you enough credits to perform traceroutes from more than 600 v6 capable probes towards the destination and then you can download the results and the graphical interactive visualisation of that. And again, that is described in the labs article. And also, if you are a member of the RIPE NCC, you can express your wish to become a pilot host of the Atlas Anchor, but more about that later.

So, since the last road map update, we have implemented several things. One of them is two additional UDM measurements, customised measurements. We have made it possible for you to share the results of your measurements either about your colleagues or with the world by generating a specific key that then enables you to share one measurement or all of them, all the results with whoever you share that key with.

We are working on a new map. I was told it was published but I'm not sure any more. Is it already published? Yes it is published, I just don't have the URL on the slides but we will put it in the labs article when we describe this, and we are working also on other results that actually not directly visible but they do improve resilience and stability of the whole system.

So we also had on Tuesday a little bash. It was during the BoF slot, so it's kind of confusing, but anyway we brought some Atlas beer and it got quite cheerful by the end of it. So these are the sponsors. All the sponsors that I had in one picture previously and then the two new ones that I kind of added to this slide. So thank you very much for sponsoring Atlas and the sponsored benefits are you get promoted like this on the presentations that we make, you get extra credits, and you help build Atlas for the benefit of the community at large, so thank you very much.

Moving onto the future. So, the very detailed plans for next month. We have lined up all these things that we want to work on, and some of them are kind of done already, but we didn't publish them yet, so we are generating monthly credit reports, so you can see how many credits your probes have earned, how many credits you have used up, and you can also transfer the credits from one user to the other so you can actually see all that in these reports. It will be possible to download on the 1st October and then the 1st of every month.

Then we are going to start reminding people and chasing them to actually plug in their probe and make it active. And currently, we do it just once, but we will implement something called Three Strikes, which is three reminders until we actually kind of give up on you and say, okay, this probe is never going to show up, let's not count it any more and we will, like, write it off.

So, we will also put more effort in putting more probes out, and we recommend you to order one as a Christmas present soon, because the shipping and the wrapping it up will take sometime, so do it soon and you can surprise your girlfriend or your parents with a little cool hardware device which they don't know what is it for, but yeah, it's kind of cool and they'll just plug it in and forget about it, but we will then have more data to play with.

We are working on the new firmware release. We are going to start at Atlas Anchors, we will come to that later, and we are going up to date some of the graphs and explain how can you actually read the information. We publish a lot of information but sometimes isn't totally clear what does it represent.

Then for the rest of the year, the plans are a little bit less detailed, because we want to actually work on something that we heard about at this RIPE Meeting in November and December, so, more membership specific services but we don't know exactly which ones. Internet traffic maps. Again, we can't really say specifically which ones. Still making the number of active probes higher. Removing the limits. Currently you can only use a certain number of probes, and we want to make it possible to actually perform any kind of measurement as long as you have enough credits.

So, the Atlas data is currently on but the visitors of the can't always find it so we want to make it simplified so it's not only for geeks and researchers and people in the MAT Working Group, but the visitors that go to the and to make them more familiar what what can Atlas do for them.

We work on the hardware devices for the next year and Atlas Anchors again.

And so next year, as I said, we will be moving DNSMON functionality to Atlas and Atlas Anchors. We will represent Atlas data in RIPE Stat. We have big plans for expansion, so 4000 more probes and 40 more Atlas Anchors. And many for features that people have been asking for and again, these are quite general plans and we will refine them as we go. We are planning to publish the monthly roadmap updates based on your feedback and based on how many of these things we actually manage to squeeze into developers' time.

So, let us know. What do you think is the most important? What do you want us to focus on. These are our contact details.

RICHARD BARNES: So there is lots of channels for feedback. We had a really good BoF on Tuesday night. Any further comments at the mike right now? All right. You have one more topic to talk about. The anchors. Oh, Martin.

AUDIENCE SPEAKER: Martin Levy. Your three strikes comment. After three, why don't you just insist that they return the probe? It's worth a try?

VESNA MANOJLIVIC: It is, yeah, we will do that. We can not guarantee success.

AUDIENCE SPEAKER: Actually, what we are going to ask is either return it to us or give it to someone else who will plug it in.

VESNA MANOJLIVIC: And finally some puppies.

The model on these slides is called Shiska, just so that you know.

I already presented these slides on Tuesday, so some of you actually know about it but these are are the plans for Atlas Anchors. So currently we are in the pilot phase. Last month we announced the pilot. There was a labs article and we asked for the volunteers, who would like to take part in this. We were aiming at about ten organisations. Now, by the beginning of the RIPE Meeting, there were 32 people interested in this, and so, we have been asking for more and there is a beautiful map hanging there in the hallway where you can actually pin your card, and we are going to get back to you and put you on the mailing list.

So, there is a lot of interest in this pilot phase.

So, this year we are planning to look into the virtualisation and in parallel with that, we are going to install about 10 of these boxes to start with the pilot, and also to practice how is this whole procedure going to work for next year when we want to make it a production service. So by the end of this year, we should have about 10 of these Atlas Anchors. In the second phase of the pilot, which is the first three months of next year, we want to install 10 more, but by the beginning of next year, we have, we are going to learn something and maybe we will have different hardware specs, different terms and conditions, so based on the feedback and the experiences, we are going to start the second phase of the pilot, and by April next year, it should become a production service.

So, what are these boxes actually going to do? Which services are they going to have?

Well, they will start with installing them as a target for the measurements. So there will be 100 to 200 to 500 probes nearby the anchor that are going to do baseline measurements and as a host of Atlas Anchor you will be able to perform user defined measurements with the credit that you earn, so either towards the anchor or to any other destination from all the other probes that you want. Then the next step will be to add the vizlisation to the v6 traceroutes and the next one will be to install the probe software on these anchors and because they are more powerful than the small hardware devices that are used for probes, they will be able to perform more measurements as probes.

So, additional services that we kind of thought about already are DNSMON, maybe K?root if the virtualisation turns out to be possible, maybe RPKI repository and all kinds of ideas that we will get from you during the pilot phase. We also want to collect the measurement traffic towards these anchors and so on. So we have some ideas and we are going to learn much more. And we also plan to install 10 of these boxes every quarter of the year.

So, for the actual interactive session, I had some questions. For you it's only, do you want to be a host or not? If you want, see me afterwards and if you don't, you can still follow the development on the mailing list. For those who already applied, I am going to send a survey with very detailed questions where we want to learn from you, are you still interested? When do you want to take part? Where do you want to install it and all kind of details and we are going to make a choice and inform you and we are going to start the pilot very soon, in next week or in two weeks.

And this is the end of my slides. I think I am in time.

RICHARD BARNES: Thank you, Vesna, for your sequence of talks.


Any further questions from the audience? I think lots of interesting work going on, lots of new stuff coming up in the next if you months so I'll look forward to more exciting things. Everyone who is interested please sign up for the anchor pilot and we'll look forward to new interesting things in the next few months.

So, on a related topic now, Benno, I worked for some students that presented in the Plenary on how they had used RIPE Atlas to do some experimentation so we invited him to talk to MAT about more general thoughts in using Atlas and doing experiments that way, and how you build an experimental methodology based on this sort of data set.

BENNO OVEREINDER: It's a bit ad hoc presentation, or last?minute presentation, it's with Philip Homburg, so we will give two examples of Internet measurements, and we used Atlas but it can be any other measurement platform, we want to just give examples and figure discussion, the discussion is important here.

So, again, so, we just give two examples of how we did our measurements using RIPE Atlas. But, most importantly, we want to trigger some discussion, some feedback, get some feedback or create some community, that's our motivation.

So, people get involved, start sharing ideas, sharing results and maybe, and that's later on in one of the discussion points, share a code.

For the first example. During the Plenary two of our students presented their work on IPv6 or actually ICMP packet to pick filtering fragment dropping. I won't go into their results but I want to just dive into the method they used, how did they organise and implemented their experiments. So the question actually came from a problem, actually people going from tunnelled IPv6, or not native IPv6 to native IPv6 and they couldn't reach our website and some ?? the only thing we could do was traceroute and somewhere in between there was a problem. But we were just interested, can we do some kind of triangulation and figure out which network it is giving us problem because we lower our MTU on the firewall, all the problems were solved.

So, there is something both things going on. It's either ICP packet too big dropping or there is fragment filtering, we really didn't know. So the problem is you can dissect the problem into two separate problems and make it easier to study and find out what's the reason what's going wrong.

Doing IPv6 measurements, actually the baseline, operational baseline is of course IPv4, so if you do something in IPv6, it should be as good as or even better, we should strive to be as good as IPv4.

So thinking about IPv6 measurements, we should also include IPv4 measurements. So, starting out with one idea, we should do. We ended up with four experiments we should do in IPv6 ICP packet too big filtering and fragments. And IPv4 the same. It gives us a kind of experiment matrix. Originally we wanted to do the thing on the left side here, a kind of triangulation. So we have all these measured points. And again, do all kinds of traceroutes or any other experiment, and find by triangulation which network is giving us trouble. But by the nature of the problem, actually with the current Atlas probes, you can do traceroutes and MTU detection from the packet nodes ?? or were the probes. But we want also to have explain what's going on and fine what's the problem. So you want to have this back scattering. You want to have the traffic on the line and that's not possible with Atlas probes so we decided for another approach, the second best approach, which is a combination of these two. So we have a measurement box and all these targets, which are here a target and here actually the source of an experiment. And we can measure both ways and find back scattering from packet filtering or from fragment dropping, etc.

So what does it mean? With the experimental setup, we have these probes everywhere, Atlas probes. We have at our lab two machines, just the default MTU 1,500, and here we have an MTU we can play with, 1,500 or 1280. So, we have ?? and we do some TCP dump on these lines so we can see all the traffic that's going in and out. I show you the probe, another probe, it's default probes, but we get some support from Philip, from RIPE NCC. Behind that there is some special code. We couldn't run these experiments directly with user defined measurements so we get some support and with these support, we could do some extended measurements towards our network and see what's going on.

So, again, I won't go into the details of the results. But this is the data gathering, it's very easy. The probes they published a data in their database, or every now and then. It's available on the Json format, we use Python script, Json, Python, it's a good marriage. It works very well and our own site servers, we run a TCP dump. We got loads of data. Then you need of course a number of scripts. We had to filter and correlate these events because of course the probes were generating the first descent and our TCP dump was just generating all this data and we have to pick out which Atlas probe event matched which TCP dump data. And then we correlated the data, we are running some statistics, we generated histogrammes and finally we did some analysis of this data. For example, we had a question, are the problems, are they generated or the problems, are they occurring in the edge, at the edge of the network or in the core? And we did some path enumeration. So, these are the probes where problems appeared and in this way we tried to find out if this is a large collection of probes which have a problem and do they share one router, for example, one network and then we can say well probably the network, that transit generates this problem.

Okay, this is just our experimental implementation. I want to hand over to Philip.

PHILIP HOMBURG: So, in talking to Benno about, well, saying something about methodology, I realised that in, well, the recent graphs we produced for DNS data, that there was a very interesting variety in how you can measure stuff. So if we look at the first map, then what actually happens there is that continuously the Atlas system will always save the last result you get per probe per measurement into a database and then if you say I want to map, then it just gets the latest data out of the database and then displace T so it gives essentially a realtime update view of what the world now looks like. And that's ?? it has one disadvantage, because it's sensitive to trenchant errors, you just get whatever is the latest data and if the probe just happens to have a problem just when you're looking, then you see that but you don't have any historical data to see whether that's normal or not.

If we then go to the second graph, which is essentially plotting the exact same data, there it's completely different approach because there what is done is it actually takes all the data from one day and then it says, well this is how it looks like statistically if you look at it over one day, and also the access mechanism is different because we are trying to store all our data in edge space and this is sort of ?? but we are not there yet, we have a lot of issues really getting it to perform and this is one of the experiments to see can we get useful data out of it.

Now, the other dimension that's interesting is that this is an experiment that just continues to run. And so, instead of having experiments that always run, the other thing is you can also have experiments that just run when you need them.

I have two graphs for that. The first one is that we started plotting the K?root data, the K?root TCP data and then there was some weird stuff and now K?root is all the RIPE Atlas so it is Atlas was wrong. So then I completely reimplemented sort of the same measurements using normal Linux and using Dick as a tool for querying K?root and it turns out that gave the same data as RIPE Atlas, so that confirmed that there was an issue with K?root. But it shows another interesting thing, because if you look to the left side of the first graph, then there is some really interesting noise there and that's just because of that vantage point and then if you use Atlas instead of just a single machine, then you can usually look at nearby probes and you say well, is it really this particular probe, this measuring device or is it a more common problem? So there is definitely a dimension, do you have one vantage point or do you have multiple vantage points?

And then finally, the last graph was also really interesting because of sort of the way we dealt with data, because we took the normal user interface except that if you are an administrator, then you can run experiments with slightly different parameter settings, but then we actually shipped all the data to Peter Koch who was interested in it, so that's sort of the collaboration part. If you have an interesting research question but that's also for Benno, and we are also interested, then we make sure you get access to the system, he did the processing and then he gave us the results back because we have the scripts to visualise, it which he doesn't. So, this, I think, gives very nice description of all the diversity in how you can do measurements. Over over so, actually this is the main thing we want to discuss.

What are the next steps to create such a community, and these are two examples, we have run some experiments. We want to share our results. But we also want to share our scripts, to make it easier for other people to run experiments and they don't have to do all the scripting.

So the publishing of results, I think ?? well, it's all for discussion. These are just suggestion but I think publishing at RIPE Labs is already going on and I think quite successful. People do publish their stuff. But also it creates an index of the tools or the scripts people have been using and it can be just a web page with links to personal web page where I publish to my own repository or we can set up something like a get up or, and people contribute, people can make their clone and work on it. But something that kind of a tool box for analysis is generated. It's built by the community. An of course make use of the e?mail list, exchange and ask questions, there are no stupid questions, there are only dumb answers, and, well, get some work done together. I think there is a lot of interest at least as a consumer of these results, but I think that also there are many people here interested in generating results and contributing. So, that's actually my last slide and that's the discussion. How can we get further on and get something going on, a community effort in building such a tool box or sharing results?

RICHARD BARNES: Thanks for getting us started with that, Benno. Does anyone have thoughts on these questions, suggestions that Benno is making? I think, you know, sharing some tools and scripts is probably a good idea. It's sort of a ?? we have done a pretty good job here with the M and A in this group and not so much with the T for tools.

LORENZO COLITTI: A question for Philip, really. I was impressed by the speed with which you were able to answer the question that I posed to you about IPv6 filtering, right. Is that because you had some special NCC powers or, you know, what kind of access would have been necessary and how could you have allowed a user of the system to do the same?

PHILIP HOMBURG: So, what I did, so that the interesting question, well, in your part for me was: The country code stuff because I have never done anything with countries and I had a fake idea that we would have it. So what I did was I set an experiment on all probes, at the time Atlas users cannot do that but as Vesna said we want to lift that. I have no idea how practical that is going to be because it also consumes a huge amount of credits and I let the expert just run through the night and my guess is that it may just consume a year worth of credit or something like that. So, we have to look into that.

But, apart from that, I used standard interfaces, so, yeah, in the future that if you have any credits, then you should be able to do it.

LORENZO COLITTI: I guess the question is how would I have known that I could just run dig on the probes? I thought I could only run ping and traceroute.

PHILIP HOMBURG: You can also ?? just read documentation.


PHILIP HOMBURG: No, no. The DNS and SSL measurements are open for everybody who has credit on the Atlas system. If you say that we have to publish more about what you can do, then that's I guess a definitely a hint for Vesna.

LORENZO COLITTI: Maybe you have and I haven't read it, I'm not saying ??

RICHARD BARNES: Are you on the Atlas users mailing list because I think there was some discussion there.

LORENZO COLITTI: Networks I should join. I should definitely join.

RICHARD BARNES: Maybe one upshot of your question is we should kind of merge the two lists because a lot of stuff here is related to Atlas and maybe there is not such a huge distinction between the two.


AUDIENCE SPEAKER: Daniel Karrenberg. And joining that mailing list is certainly a good idea but that was also published on RIPE Labs so if you follow RIPE Labs you could have seen it.

I think this is great. Benno coming up and saying, hey, you know, let's build some community and tools around this because that's really what it's all about. So, I really like this to happen. And I think it should also be in a forum that is what software developers usually do, so I don't know, I am a manager now, but what I hear is that if it's not on GIT up, it doesn't exist. Anyway, the more serious remark is that what I see at this meeting, and what's coming up in the Plenary and everywhere is that the research community loves this stuff, and that's really good, you know, I like it. You know, after all my title is chief scientist but we have to be very careful that to realise that the people who are paying for this are the ISPs. So, if we are building a community, the worst thing we can do is build a community just for the academics and researchers. I think we can ?? it would be really, really good if we could engage some of the operators as well and make some tools that are interesting for them in their day?to?day operations or in the questions that they have that are a little bit more longer term, but if we let the thing diverge into two different communities, one for the people who publish and the others who do the packet pushing, I think that would be really, really, really bad because then we might lose the support from the people who are actually, the packet pushers who actually in the end pay for all of this.

BENNO OVEREINDER: Yes, I agree. I was also thinking this tool box it should be really building blocks, so it could be more easily, to lower the barrier to run an experiment, so you have these building blocks kind of ID and you create your, or you build your measurement experiment and it should lower the barrier and for me lowering the barrier was to get two students doing the work. Okay, I am in the position, luckily enough to do that, to have this opportunity. But for many situations, I understand. Just short of time you want to do ?? you are interested in the measurement and experiment, but you don't have the time. And we should lower that barrier around I think such a tool box or ?? could help us out.

RICHARD BARNES: One I think I think might be interesting to discuss maybe in the BoF session afterwards, anyones the NLNOG ring is Atlas similar in its distribution but it's more operator focusnd an driven. I wonder you could build a version of ring?ping and build on top of Atlas. I wonder if there is tools like that or we could figure out the intersection of intention that come out of the Internet operators community.

LORENZO COLITTI: I think if you want to involve operators, well, he can read the transcript. I think time is the key, right, I mean, it could be nice if, you know, say some LIR has a website and you want to measure a performance from 1,800 probes to the website on an ongoing basis, what you need to build there is you need to build something that can tell them what's happening and that can generate pretty graphs and alerts, right, and then a lot of operators will use that because I mean, think about MO, TG, right, it's relatively easy to set up and everybody use it is, so if you can get to the point where it's that easy to build something that's really useful in daily operations but not just that, take traceroute alerts, oh, my path is suddenly 20 hops long, something like that, that's useful to operators. But to Daniel's point, there is a natural difference, I mean, researchers tend to want to do stuff that's not been done before and they can publish it and the operators mostly have to solve similar problems. So, not that it couldn't be one community, but I think there needs to be a few more building blocks before an operators could have the return of investment on the time that they have to use it.

BENNO OVEREINDER: I am aware what Daniel said. Because our ?? the costs are paid by the operators, and that's important. I was also thinking, well, lower the barrier ?? the operators should get access to these ?? well, to the opportunity, get the opportunity to run these experiments and make it available.

AUDIENCE SPEAKER: Job Snijders. As mentioned before the NLNOG consisted of 99% operators so it could be of interest to reach out the NLNOG mailing list if you are an academic and you want to learn what operators might find interesting. As Lorenzo pointed out, I think the time frame in which things happen is of essence, for instance, as an operators, I want to know what is going on right now. I need realtime data, so any experiments that are running should be continuous and I should have continuous access to that. I must admit that although I find it fascinating to know that in the summer of 2010, the average AS path was longer than in the summer of 2011, it is not really relevant to me on my daily ?? on my day job. So, I'd be interested in new tools and the topology that is run continuously and give continuous feedback so that as an operators you have some kind of baseline based on cool novel matrix and the moment it goes to the red, you start working and debugging. So, please contact the NLNOG if you need inspiration as a researcher. And you said, ring?ping on Atlas, I believe that's already possible, although it's not realtime. But you could correct me if I am wrong.

PHILIP HOMBURG: So one of the goals also, I don't know the exact time frame, at the moment the Atlas system is optimized for doing continuous measurements, you set them, you let them run for, well a while and then you analyse the results. And one of the things that we also scheduled to add is you start the measurement now and say within half a minute, you issue have the first results. But that's not a feature that we have at the moment. If you ?? I think that if you would do a ping now, then the fastest you get results is probably six minutes. So that's sort of the current time frame.

RICHARD BARNES: Daniel, did you have anything else?

AUDIENCE SPEAKER: Daniel: I just wanted to answer a little bit to Lorenzo. I realise astutely that the interests of the research community and the operators community are distinct. And I wasn't suggesting to make them the same. What I was suggesting is to keep talking in the same room, and to use the tools that was shall ?? build a community around building tools and I wouldn't like to see the research community makes their tools over, on the left and the operators community makes their tools over on the right, or doesn't, but to see where the overlap is, and to keep talking because what's research today is probably something operational tomorrow. So, my only worry was let's keep ?? if we are building a community, let's be ?? realise the fact that they are two large groups here, research and operations, and let's keep talking in the same room and let's also realise, and it does me really good to hear that people reflect that, that the operators are actually funding this whole exercise.

RICHARD BARNES: Thank you, Daniel, and thanks to Benno and Philip for getting this conversation started.


So that brings us to the end of our scheduled agenda. I think we are one item of AOB that Ian has proposed so I'll hand it over to him.

IAN: I just wanted to announce that I have decided to resign as one of the chairs of the MAT Working Group after ten meetings and nearly five years. Back when I became the Chair, at RIPE 55, it was known as the test traffic Working Group and there was just two chairs, myself and Hank. We managed to move it from being the test traffic Working Group to being the measurement ?? this is why I am leaving, I can't even remember the title of the Working Group, measurement analysis and tools Working Group. I actually wanted to call it Internet measurement Working Group, we recruited two very capable chairs in Christian and Richard and we have moved it from being something that was always stuck in the small room to being one of the bigger draws at the RIPE Meeting. I'm leaving it in the capable hands of Christian and Richard. I leave it up to them if they feel that I will or indeed can be replaced and I'd like to finish by thanking you as members of the community, thanking Richard and Christian for their support and all the staff at the RIPE NCC past and present who have made it such a success and I am looking forward to attending without having to worry about getting enough content and making it run to time without crashing the BoF which I think for once we have managed to do. Thank you.


RICHARD BARNES: Yes, thank you very much, Ian, for your service here. If we have no other items for any other business, then we are adjourned. Thank you all for coming. And I think in this room, is it scheduled to start at 6, is the NLNOG BoF. So we'll be back for more measurement action in a little less than 30 minutes.