Anyone in the Portland area interested in a fun and creative event is invited to the 1st Timbertalkers Powerpoint® Karoke contest on Tuesday 11/24 at noon.

Meeting location is: 9403-B SW Nimbus Ave., Beaverton, Oregon

If you have never done PPTK, here are the rules:

• Topic is draw from set of 30 topics. Probably 10 to 15 slides
  
• Speaker will have 2 to 3 minutes
  
• Prizes awarded
  

In spirit of open source, it will really be a OpenOffice Impress contest, and the slides will be drawn from Creative Commons licensed decks.

Ubuntu has never been the easiest distribution to do kernel development, but it looks like with 9.10 it has made things too painful. I need to build and install kernels all the time, and usually just update grub menu manually. But now with grub 2 in Ubuntu 9.10 they have wrapped the grub menu in grub-mkconfig. Why?

It would be great if the system was setup so just doing 'make install' in the kernel source put in the kernel and updated the grub.cfg, but no that would make too much sense.

P.s: they managed to break the sky2 driver somehow, the connection won't come up and negotiates the wrong speed. It turned out not to be a kernel problem; wiring issue (speed), combined with some Network Manager changes

I am giving three talks: 1) routing performance, 2) staging drivers, 3) Vyatta CLI.
So if you are attending JLS please stop by and give me support.

It will be a busy week. The network developer's are getting together at Netconf over the weekend,
then LinuxCon followed by Linux Plumber's Conference. Hope the weather holds out, Portland has a tendency to rain when ever there is a big event.

Nice. Microsoft has released the Hyper-V drivers as GPLv2. I know was a hard step for Microsoft to take, since it means acknowledging GPL and respecting the Linux community. The releasing of the drivers is good news for users, developers, and in the end Microsoft as well. Like most GPL related actions, a lot of work was done behind the scenes to get the offending company into compliance.

This saga started when one of the user's on the Vyatta forum inquired about supporting Hyper-V network driver in the Vyatta kernel. A little googling found the necessary drivers, but on closer examination there was a problem. The driver had both open-source components which were under GPL, and statically linked to several binary parts. The GPL does not permit mixing of closed and open source parts, so this was an obvious violation of the license. Rather than creating noise, my goal was to resolve the problem, so I turned to Greg Kroah-Hartman. Since Novell has a (too) close association with Microsoft, my expectation was that Greg could prod the right people to get the issue resolved.

It took longer than expected, but finally Microsoft decided to do the right thing and release the drivers.

Hey kernel developers, more proposals related to networking submitted for the Linux Plumbers Conference. This is the chance to have in-person discussions about future proposals like receive packet steering, RCU netfilter optimization, unified flow cache, and all those other topics that need need more brainstorming and discussion.

The Netconf 2009 is also being planned to occur before LPC.

The Linux networking receive performance has been mostly single threaded until the advent of MSI-X and multiqueue receive hardware. Now with many cards, it is possible to be processing packets on multiple CPU's and cores at once. All this is great, and improves performance for the simple case.

But most users don't just use simple networking. They use useful features like netfilter to do firewalling, NAT, connection tracking and all other forms of wierd and wonderful things. The netfilter code has been tuned over the years, but there are still several hot locks in the receive path. Most of these are reader-writer locks which are actually the worst kind, much worse than a simple spin lock. The problem with locks on modern CPU's is that even for the uncontested case, a lock operation means a full-stop cache miss.

With the help of Eric Duzmet, Rick Jones, Martin Josefsson and others, it looks like there is a solution to most of these. I am excited to see how it all pans out but it could mean a big performance increase for any kind of netfilter packet intensive processing. Stay tuned.

Many times I hear about GPL violations in vendors software, especially it seems in embedded routers. There are two cases which hit me in my home.

The first is our FIOS router which is an Actionec MI424-WR which runs Linux inside. You can even get to a telnet prompt. The problem is that it has a crappy DHCP server and always seems to assign different IP addresses even to the same MAC address. This breaks ssh and other services which do strong man-in-the-middle prevention. It seem the vendor hasn't fixed the problem, but as a result of a GPL violations suit the some source is available but the DHCP code is not included probably because it is BSD licensed so they don't have to. Given this I'll just punt and do the lazy solution and just turn it into an dumb Ethernet bridge and use something better like Vyatta V514 test box or Linksys WR54TG, both of which are repairable.

The second is the Asus P6T motherboard which has a SplashVM feature. This allows booting to a lightweight desktop in less than a minute (the BIOS is still slow to get its hardware setup). The desktop is based on Linux with standard kernel and browser. It is kind of a toy, but good for checking gmail etc. Since SplashVM is using GPL, if the vendor was following the GPL license I should be able to find the source on their website. It is possible to find some pieces on the Splashtop vendor website, but it is the responsibility of the system vendor not the subcontractor to make available the source for the actual firmware they are shipping. In this case, it matters to me for a couple of reasons. I wrote the driver for the Marvell Yukon-2 EC Ultra NIC's on this motherboard and would like to know if 1) the vendor fixed some bugs 2) the vendor still has some bugs that other users will pester me about. As copyright holder for this driver, I may have to go nasty to find out; stay tuned.

At netfilter workshop, Patrick McHardy described an exciting new feature implementation of netfilter firewalling called nftables. This has the promise of reducing 100's of netfilter modules down to a smaller kernel footprint, and allow for optimization of rulesets. Eric Leblond's blog has more information.

I really should stop reading the Oregonian, they do such a poor job of covering high tech and the business section is especially weak. The recent piece about OSCON moving to Silly Valley overlooked so many obvious things like the Linux Plumber's Conference next week, the Kernel Summit not to mention the Open Source technology center, Oracle office in Portland, Portland State, and Free Geek. So the loss of one conference which is mostly attended by out of town people is really no impact on the local open source infrastructure.

Sunday 8/31 user friendly cartoon is great.
Do these people look like aliens?
Guess I'll have to give up on trying to setup selinux.

In order to implement router style semantics, Vyatta allows setting many different configuration variables and then applying them all at once with a commit command. Currently, this is implemented by a combination of shell magic and unionfs. The problem is that keeping unionfs up to date and fixing the resulting crashes is major pain.

There must be better alternatives, current options include:

• Replace unionfs with aufs which has less users yelling at it and more developers.
• Use a filesystem like btrfs which has snapshots. This changes the model and makes api's like "what changed?" hard to implement.
• Move to a pure userspace model using git. The problem here is that git as currently written is meant for users not transactions.
  
• Use combination of copy, bind mount, and rsync.
• Use a database for configuration. This is easier for general queries but is the most work. Conversion from existing format would be a pain.
  

Looks like a fun/hard problem. Don't expect any resolution soon.

Added CLI support for TCP MD5 (via Quagga) to the upcoming Vyatta release. It worked fine under testing (VM) but wouldn't operate with IOS. Reduced the problem down by making some useful utilities:

• Patch for Netcat to support MD5
• Standalone using libpcap to validate MD5 option in capture file

It turned out that the sender was generating wrong MD5 option after the initial SYN handshake. When data is finally sent, the problem is that the data in the kernel is fragmented because the underlying device supports scatter/gather but the md5_calc doesn't do scatter gather.

I have high hopes for the first Linux Plumbers Conference. Unlike an academic conference with papers, or an un-conference with no agenda; the plumbers conference is using a mini-conference format to break down by topic. There is even a Call For Speakers to get speakers in topic areas.

First time conferences have a different feel, more rough edges, but more passion and fun. So I hope it works out. There is no particular networking track, mostly because the other areas seemed to need more work.

For the next release of Vyatta, I wanted to enable the Trie algorithm for routing in our kernel. Since FIB Trie is compatible with the previous hash, I expected no change. Well the day after enabling it caused an immediate failure in the regression test. The regression test plays back a full BGP input stream into the router and polls for the result. The problem was that the Trie to a long time, ... a really long time, to dump the routes. For the full 163395 routes, the dump was taking 20 seconds vs 1/2 sec for FIB_HASH.

As expected the problem turned out to be an N^2 lookup. The code was basically:
walk tree to find a route, and put it in buffer; if buffer is full, then give up, then go
back and walk to the last location. Since the trie has nice fast lookup function the change to just record the last route dumped (rather than offset), then use the lookup to find the location. This dropped the dump down to under 3/10 sec.

Finding this took several days, mostly because of looking at the profile; see "nextleaf" is the hotspot, so let's look at that. The real breakthrough came when I realized there were other operations that were walking the tree, like collecting stats, but they were fast. The next diversion was figuring out all the other suboptimal behaviour (ip route flush calls fflush for each route), which although slow weren't the real issue.

Started out for first day at Vyatta. There is a lot of overlap between what I know from doing iproute2 utilities so hopefully the learning curve won't be too steep.