Version 1.4, which includes benchmark results for FreeBSD's in-kernel PPPoE implementation. Version 1.4 was published on December 7th 2002. Version 1.2 added benchmark results for OpenBSD and an editor's note regarding FreeBSD in-kernel PPPoE support. Version 1.2 was published on December 2nd 2002. Version 1.0 was published on November 29th 2002. Changelog is available.
Point-to-Point Protocol over Ethernet (PPPoE) is used by many Internet Service Providers (ISP) because it eases the administration on the server side. However, on the client side it can be a pain because PPPoE adds to CPU overhead if a userland implementation is used and not many operating systems support kernel-side PPPoE.
We tested PPPoE performance under various free UNIX variants including Linux 2.2 and 2.4, NetBSD, OpenBSD and FreeBSD. During the test we focused on CPU usage under heavy PPPoE network load. We also considered under which operating system PPPoE required the least effort to set up.
The main ISP in our area has started to use PPPoE in their high-speed DSL connections, ranging from 8/1Mbit/s to 10/10Mbit/s. Personally I (Juuso Raitala) am changing from a non-PPPoE DSL connection to a PPPoE one, and, being a perfectionist, choosing the most suitable operating system for my router-firewall is a priority - I want my low-end Cyrix 6x86 100MHz box with 32MB of RAM to be enough to handle a high-speed PPPoE connection without glitches.
After I told about my intentions to some friends of mine who were already using PPPoE, we decided to spend a weekend comparing different operating systems in my place with lots of caffeinated cola, mild beer and Finnish sauna.
As our test platform we used a Pentium 90MHz. At first we wanted to put only 32MB of RAM but it was insufficient for Linux 2.4 with kernel-mode PPPoE, so we decided to add another 32MB. A 3Com 900B network adapter was used. We borrowed a few old Seagate 500MB hard disks and installed the operating systems we wanted to compare. We partitioned the hard disks the same way, giving every operating system 80MB of swap space. In all tests, we used the stock kernel which came with the operating system or, if necessary, added PPPoE support into it.
My main workstation, which is an Athlon XP 1600+ with 512MB of DDR RAM, was used as the PPPoE server. A clearly overpowered server was chosen so that it wouldn't affect the test results. The PPPoE server software we used was RP-PPPoE.
The test network was a switched 10Mbit/s half-duplex network with no other traffic.
Under each operating system we measured the PPPoE performance by testing the maximum amount of PPPoE network load the CPU can handle. We also measured ping times, both from client to server and from server to client. Statistics were gathered using iptraf and top. Iptraf was ran on the server side in order not to impact performance. Netcat was used for actual transfers.
These commands were used to transfer data:
On PPPoE server: nc 10.0.0.10 4884 < 200M |
The Linux distribution we used was Debian GNU/Linux, which is a popular and portable server distribution. We chose Debian, because it's familiar to us and easy to set up.
On vanilla Linux 2.2 (we used 2.2.20), only a userland PPPoE solution is available. We used RP-PPPoE. Since no special kernel-side support is required, configuring was straightforward and we didn't face any problems. The PPPoE implementation has no known stability issues and it also supports MSS fixup, which clamps MTU to 1492 so you don't have to reconfigure MTU of your LAN computers.
The performance was poor when using asynchronous transfer mode and decent with synchronous transfer mode (see the table at the bottom for more details). A point worth mentioning is that synchronous transfer mode isn't guaranteed to work.
On Linux 2.4 (we used 2.4.19) the configuration required a bit more work. Debian GNU/Linux, and we believe that every other major distribution, doesn't include the required ppp development version. In-kernel PPPoE support isn't present in the latest stable version (2.4.1) of ppp, but will be included in 2.4.2. Also recompiling the Linux kernel, or at least the PPPoE module, is needed if the vendor supplied stock kernel doesn't include it.
Kernel-side PPPoE refused to run with 32MB of RAM.
The performance was very good (see the table at the bottom). At maximum network load the CPU usage was about 25%. As David F. Skoll, the author of RP-PPPoE, pointed out: "In the kernel, the difference between PPPoE and normal Ethernet encapsulation is pretty much a pointer increment." Besides that, the PPPoE implementation supports MSS fixup.
Kernel-side PPPoE support has been included in NetBSD 1.6. MSS fixup support is available.
[Editor's note: As Martin Husemann pointed out, MSS fixup is supported in NetBSD 1.6. It was clearly stated in http://www.netbsd.org/Documentation/network/pppoe/, but we seemed to stick with staring only the manual pages and didn't notice this.]
Configuring PPPoE under NetBSD was very straightforward and simple - a single glance at the manual page offered all needed information. Configuration options were limited but adequate. The default kernel included PPPoE support, so no recompilation of software was required. However, the error messages weren't so descriptive (eg. "GENERIC ERROR").
NetBSD's performance was exceptional (see the table at the bottom), surprisingly outperforming even Linux 2.4. On NetBSD, system CPU usage was lower, but netcat used more CPU cycles. One important aspect is that NetBSD's PPPoE functioned properly and without mentionable performance loss with only 32MB of RAM, unlike Linux 2.4.
[Editor's note: This chapter is completely rewritten since the version 1.0.x of the article. At first, we weren't able to make PPPoE on OpenBSD function, but after I tried to do everything all over again I succeeded. The results are now available.]
As we said in the previous version of the article, at first we weren't able to make PPPoE on OpenBSD work. As I don't have the same hard disk on which I initially installed OpenBSD for the test, I can't diagnose the situation further. I want to and must state that a configuration error is the most probable explanation why PPPoE on OpenBSD didn't initially work in our test environment. I humbly apologize for any inconvenience and spread of misinformation I have caused.
After facing some trollish criticism I decided to grab another hard disk and gave OpenBSD another try, this time being successful right from the start: few minutes after the installation I was able to form a PPPoE connection.
But the good experience ended right there. The performance was very poor (see the table at the bottom): OpenBSD's userspace PPPoE implementation didn't even closely match that of Linux 2.2 or FreeBSD 4.7. Moreover, the maximum transfer speed wasn't stable at all, ranging from 1.3Mbits/s to 2.5Mbits/s.
FreeBSD includes kernelmode PPPoE support. However, it is not stated in the FreeBSD Handbook in any clear way and the required program, mpd, isn't installed by default. Therefore the configuration required a bit more effort and knowledge that such a solution is available.
The performance of FreeBSD's PPPoE implementation using mpd was good, but it didn't match the in-kernel implementation of NetBSD 1.6 nor Linux 2.4 (see the table at the bottom). Still, the impact difference wasn't significant.
[This chapter is for reference only. The userland solution is well documented in the FreeBSD Handbook unlike the kernelmode solution.]
FreeBSD was a disappointment. Its default kernel didn't include PPPoE support, so compiling a new one was necessary. Even with the self-compiled kernel, PPP encapsulation was not done in kernel. The userland solution showed inadequate performance (see the table at the bottom). Using synchronous or asynchronous transfer mode didn't make any difference. However, MSS fixup is supported.
Configuring FreeBSD's PPPoE was easy with its complete Handbook. Possibilities to debug and log connections were good.
Simply put, the most reasonable choices as PPPoE client operating system are NetBSD 1.6 and Linux 2.4. Linux 2.4 was more configurable while NetBSD was slightly better in performance and required less effort to set up. 32MB of RAM was sufficient for NetBSD, but not enough for Linux 2.4.
It's impossible to say which is better as a router operating system, as it's mostly a matter of taste. However, some facts can be pointed out. NetBSD offers excellent portability and good security record; Linux has superior hardware and, in some cases, software support. Other differences include firewall management: NetBSD uses IPFilter, Linux 2.4 iptables.
FreeBSD with the kernelmode PPPoE implementation (using mpd) is also good enough for high-speed connections, even though it wasn't as efficient as NetBSD 1.6 or Linux 2.4.
Now as the results for OpenBSD are available, it can be said that PPPoE implementation on OpenBSD isn't as refined as it is on other operating systems we tested. In fact, OpenBSD's PPPoE performance was surprisingly poor.
PPPoE performance under Linux 2.2 and OpenBSD is adequate only on slow links or where CPU cycles can be wasted. If you want to use the low-end Pentium found in your closet as your router, these operating systems just don't offer enough for high-speed PPPoE connections.
| Operating System | PPP additional info | Transfer rate | Ping [1] | CPU usage | CPU usage breakdown |
| Linux 2.2 (64MB RAM) | usermode, async | ~4.1Mbits/s | 1.4/1.1 | 100% | pppoe 95%, nc 5% |
| Linux 2.2 (64MB RAM) | usermode, sync | Network max [2] | 1.0/0.8 | 50% | pppoe 40%, nc 10% |
| Linux 2.4 (64MB RAM) | kernelmode | Network max [2] | 0.7/0.5 | 26% | system 24%, nc 11% |
| NetBSD 1.6 (64MB RAM) | kernelmode | Network max [2] | 0.9/0.5 | 23% | system 15%, nc 17% |
| NetBSD 1.6 (32MB RAM) | kernelmode | Network max [2] | 1.1/0.6 | 23% | system 15%, nc 17% |
| FreeBSD 4.7 (64MB RAM) | usermode,sync | ~6.5Mbits/s | 2.0/1.9 | 100% | ppp 80%, nc 15%, interrupt 5% |
| FreeBSD 4.7 (64MB RAM) | kernelmode | ~7.7Mbits/s [5] | 0.7/0.5 | 35% | nc 17%, system 16%, interrupt 18% |
| OpenBSD 3.2 (64MB RAM) [3] | usermode,sync | ~2.0Mbits/s [4] | 2.9/2.3 | 99% | system 87%, pppoe 65%, ppp 23%, nc 7% |
1) Ping (client->server/server->client), milliseconds
2) Network max was about 8.5Mbits/s
3) Although this can hardly affect performance, it must stated that
OpenBSD was installed on a slightly faster hard disk.
4) At first, speed reached it's peak at 3.1Mbits/s, but gradually
slowed down. After the first peak, the speed was very unstable, shifting
from about 1.3Mbits/s to about 2.5Mbits/s.
5) On FreeBSD, the maximum network speed the operating system could
handle, even with plain ethernet, was only about 8Mbits/s.
Disclaimer
Distribution and copying of this article is permitted in any medium provided this notice is preserved. An e-mail notification to any one of us would be preferred if this article is redistributed.
© 2002 Juuso Raitala <jraitala -at- jraitala dot net> (the main author of the article), Olli Vainio <odo -at- lyseo.edu.ouka.fi>, Juha K. Kallio <bunnyh -at- psychedelic.baana.suomi.net>,
Special thanks to Joonas Kortesalmi for offering web space. We would also like to thank Tuomo Mattila, Ilkka Mattila, Kaisa Pohjonen, Andrea Luzzardi and Tero Partanen for previewing the article and Jaakko Rajakallio for providing the hard disks.