The Web Presence of Brady Alleman

When I am occasionally asked about what might be used for networking two random buildings together, the first two thoughts are generally private DSL and wireless.  I’ve not had any personal experience with private DSL, but I know that Patrick had used it in the past, and I think he had used Flowpoint gear.  As I have a possible project in the works, and want to know a bit more about what I’m talking about, I bought a group of  5 FlowPoint 2200s on eBay.

Configuration didn’t go terribly smoothly.  While I was able to setup a typical 9600-8-N-1/no-flowcontrol rollover connection to the boxes, they would seem to not recognize keystrokes after the boxes booted.  If they were configured to do that, I haven’t been able to figure out how.  Fortunately with watching my DHCP server to find their IP addresses (they seem to try to lease them even when statically configured), and using the password override which thankfully works on telnet and not just console, I was able to reconfigure all but one of them.  The last one refused telnet connections (presuming a firewall setting).  If anyone has any idea what might be wrong with the console setup, or other ways of resetting the box, I’d be interested to hear.

As for the point-to-point, or what they seem to call back-to-back portion of the configuration, I found a document describing the setup process.  The routers would link up using this, but would not seem to bridge.  Bridging would be a much easier and more elegant solution for the applications I’m looking at.  I switched the protocol to RFC1483 instead of PPP which seemed to allow the bridging configuration to work.

While approaching ancient, the Flowpoints appear to be the cheapest back-to-back SDSL options out there.  Mine do seem to have a max speed of 1536 kBPS (that’s a T1 less ESF overhead, iirc), but that would probably be fine for my needs.  Anyone using better cheap gear?

I went to a local AT&T store last Tuesday and signed up for service on the G1.  I put the phone on the counter and after the sales guy asked if it were unlocked, and I said that it was, things moved on quickly.  He took my information and began the process.  I walked out with a G1 on AT&T’s network and a receipt for $0 for the SIM.  I checked out data in the car, and it was working despite not having the “wap.cingular” APN settings, which I added later just in case.

I checked my online account access upon getting home.  The sales guy hadn’t setup data access, so I quickly selected the unlimited data and text option.  It also seems that the phone hadn’t been branded a “smartphone” so all of my options were for non-smart phones.  I asked the tech for detailed billing to be enabled (apparently there’s a fee for that in AT&T land) but I couldn’t tell from the website whether it was or not.  Later detailed call logs became available, so I guess it is enabled.  Unlike Verizon, only outgoing numbers are listed, which is a lot less helpful.  UPDATE: Either I missed them before or they didn’t work before, but the detailed call log does seem to have originating numbers now.

I am understandably nervous about data usage charges until I get the first bill.  Recently the bill amount was listed – something like $114 – but I can’t see the bill itself for a few more days.  Some quick figuring suggests this is reasonable for a month’s worth of service and an activation fee, but I’m not sure that’s what it is listing.  Even more recently, itemized data sessions began appearing in the details, all under my unlimited data setup, so I think things are setup correctly at this point.  UPDATE: I have now been able to see the whole bill, and everything seems to be in order.  The changes I made to my account seemed to be retroactively applied on the date I made the change (ie, the date of service start).

The number port from Verizon took about 14 hours.  This was a lot longer than Chris Z. had experienced.

I’ve purchased a few accessories – an adapter to allow both 3.5mm audio and power to be connected simultaneously – for occasional audio use in the car.  Also a mini-USB car charger and additional mini-USB to USB cables for use at home and work. I have not bought a case.  The slide-out keyboard seems to make the possibility of a good skin-type case almost nonexistent.  I also didn’t buy a charging cradle, so even though I have two batteries, I’d have to swap them to charge (like the VX8600 which I also had two batteries and no charger, and unlike the X30 which I had two batteries and a charging cradle).

A few software migrations (updated in previous entry) have been completed, so the X30 will be placed in deprecated status this week.  The 5 items I always check my pockets for before going to work can now be reduced to 4.

One other minor annoyance is that I’ve already found some places that AT&T has lower signal strength than T-mobile had.

I might be turning into an Android fanboy.  The iPhone is still much prettier though.

Part my “phone decision tree” is carrier selection.  The G1 and the iPhone are both GSM phones that can be unlocked from their respective networks.  I’m somewhat familiar with AT&T’s network – I used to have their service and found it good, and Doug currently has their service, and we’ve compared Verizon/AT&T coverage on occasion with comparable results.  To evaluate T-mobile, I purchased a pre-paid SIM for their network.

At home, I get about 2 bars out of 4 on the G1 with T-mobile pre-paid.  This is comparable to my Verizon EV-3-bars and 1x-2-bars, and Doug’s about 2 bars of AT&T.  At my parent’s place I get full signal, same as Verizon.  At work, I expect full strength being a stone’s throw from their tower.

For further comparisons, I did about a 100-mile road test and managed to get Chris B. to go along and observe signal strengths.  Here were the generalized results:

• Home to Roxbury:  T-mobile > Verizon
• Roxbury to Fannettsburg (along turnpike):  T-mobile = Verizon
• Fannettsburg to Ft. Loudon (route 75):  Both poor, T-mobile < Verizon
• Ft. Loudon to Chambersburg (route 30):  Marginal Verizon, but mostly no T-mobile
• Chambersburg to Caledonia (route 30):  Both very good, T-mobile = Verizon
• Caledonia to Shippensburg (route 233, Ship Rd):  Both generally poor, T-mobile = Verizon
• Shippensburg to Home: Both good, T-mobile = Verizon

These observations were generally consistent with the published T-mobile and Verizon coverage maps.

With these observations, T-mobile appears to be a viable carrier for the locations I spend most of my time.  But cell service is nice to have places where you don’t spend a lot of time as well.  Looking at the state-wide zoom level for AT&T, Verizon, and T-mobile, it appears clear the T-mobile is really, really lacking.  AT&T and Verizon appear to be about comparable.

In performing this comparison, I found it hard to believe that there were no good tools to show multiple providers coverage areas simultaneously.  Verizon’s coverage map was good, with a google-maps-like drag-scroll interface.  AT&T and T-mobile were not.  In all cases the zoom levels for the maps poorly corresponded, making comparisons that much more difficult.

What about roaming?  I had read/heard about T-mobile and AT&T having mutual roaming agreements.  Can I purchase T-mobile service and use AT&T’s network?  I tried selecting AT&T as the provider with my T-mobile pre-paid SIM, and it did not work.  That does not necessarily mean that it would not work with a “real” T-mobile plan.  However, I would really expect the provider coverage maps to show areas that they mean to cover with roaming, and T-mobile does not show anywhere near AT&T’s footprint.  Besides, I have read of peering agreements changing over time without notice, so even if it worked now, it might not forever.  I’ve decided it’s not worth getting a real T-mobile account just for testing internetwork roaming.

As a break from the analysis, I did manage to get to try T-mobile’s EDGE data service on the G1.  I’m not sure exactly how it happened.  I upgraded to R33 last night, and while around my parents house, a new “G” icon appeared with in/out data arrows.  An app that tracks data usage confirmed that it was the 2G radio interface, and that it was passing traffic.  I was able to play around with Google maps, web browsing, IM’ing, ping, and ssh using EDGE today, and the performance was fine.  This evening however, it stopped working and the browser shows a page saying that the G1 needs a real data plan.  Still, I was pleased to have the brief opportunity.

What about 3G?  If I use the G1 on AT&T can I use 3G?  The answer appears to be no, due to different frequencies/bands used for T-mobile’s and AT&T’s network and handsets.  But T-mobile does not have 3G anywhere near me anyway.  AT&T does, so if 3G is a requirement, the iPhone has to be the way to go.  Otherwise, using the G1 on AT&T locally should be no different 3G-wise than using it on T-mobile.

Conclusion: T-mobile isn’t a viable carrier for me.  The G1 can work on AT&T’s network as long as I don’t need 3G.  If I need 3G, the iPhone, AT&T (and going to Carlisle or Hagerstown, for now, at least) is the only option.  Do I need 3G?  Probably not.

So, at work today we received two boxes from Vendor C shipped via Vendor U that contained identical parts. It was immediately observed that the one box was far heavier than it should have been. Perhaps they shipped both of the items in the one box, and manuals and such in the other? Stranger things have happened. And, well, had. Both boxes contained what they were meant to. The heavier one had a plastic-wrapped white object in the bottom that looked like shrink-wrapped documentation – at least, until you tried to move it. It had a Vendor U shipping label indicating a weight of 37lbs and was about a foot square and an inch thick. Apparently this piece of solid steel encountered the Vendor C box and penetrated it during shipment (judging from the correctly sized hole in the side of the box, noticed afterwards). Vendor U is sending someone to pick up the stowaway package, and the Vendor C gear looks to be undamaged. Still. Weird.

There are good networking problems. These are the kind that something happens that you don’t understand. You mull over it, eventually diagram out the different layers of the various protocol stacks, and discover that things are working exactly as designed, just not as you expected. Ian and I uncovered an example of this kind of problem a few years back at CTI. There was a periodic surge in broadcast traffic on our LAN destined for a specific box. After ruling out all of the various NAT rules and such going on, we eventually found the simple cause. The box was not sending any packets of its own. So, the switch had no forwarding entry for it. So it did what switches are supposed to do in that case: broadcast the packet. Coming to those kind of conclusions is fun.

There are at least three kinds of bad networking problems. The first is when something is evil by design. An infamous example of this would be the error measurements in the DS1-MIB. Absolutely useless when it comes to making time-series graphs, or setting event thresholds. The design seems to have been one of convenience, since the quantities represented date back to the first of the DS1 channel banks. Anachronistic design sucks.

Another type of evil networking problem is one that is seemingly random. Everyone knows that one of the first steps in troubleshooting is knowing what steps are needed to reproduce the problem. With random events, you’re essentially screwed. Turn up the debugging until it gets painful and wait for the event to recur, and pray you captured something useful when and if it does. DDOS attacks and the like can fall into this category, especially for those of us without the ability to do meaningful flow logging. Tracking down random problems is evil.

The third category of networking problems I feel like discussing this evening is the unreasonable problem. This is the non sequitur, the problem that makes you yell futilely at your terminal or coworker, from the complete and utter ridiculousness of it. If you manage to solve one of these, you might end up with a good networking problem, as described above. Or you might want to take a sledge hammer to a piece of hardware. Let’s explore some examples:

Near the end of my CTI experience, a certain Astrocom CSU/DSU was observed having a most unlikely problem. If I remember correctly, it somehow would drop packets over a certain size. A most improbable feat considering that your average CSU/DSU should not really have any concept of what a packet is, let alone be able to drop one. Patrick offered a bounty on the problem, but as far as I know, it was never solved.

The last example is the reason I am writing tonight. Last summer at Doug’s LAN party, I had difficulties copying large files to my desktop machine. I eventually blamed it on my patch cord, but by that time we were packing things up, so this was never really tested. Even before this, I was having trouble sending large print jobs to my printer. I quickly blamed this on the network card in the printer, or some postscript oddities, but never came to a solution.

Later, when trying to use my desktop as a file server for a CentOS install, I realized that my network issue with my desktop was ongoing. Traffic analysis indicated that during a high speed transfer coming from my desktop to another machine, the connection would stop passing packets. Packets on other TCP connections between the same machines were unaffected, but subsequent retries of packets associated with the dead connection were getting dropped somewhere. Since this seemed to be connection related, firewall settings were verified and found to be fine. I let the problem fester, as it was not causing any day-to-day difficulties, since my desktop isn’t ordinarily sending large amounts of data, just receiving.

So, this evening I was talking with Doug about printer stuff, and he made a connection that I had been missing. Was my printer problem related to my network problem? And what about all of those problems with NFS in the recent past? Yes, they all sound like candidates. With that late realization, I delved into the annoying network problem. I replaced the network card. The problem persisted. The cabling was ruled out. And the problem was narrowed down to a switch. A specific switch port, to be precise. Now, I’m not exactly sure how one of my NetGear GS506 gigabit switches is managing to drop packets belonging to a specific connection when that connection begins spouting lots of traffic in a certain direction, but that’s exactly what it appears to be doing. And it’s reproducible. So yes, a problem as insidious as this problem should be solved with the sledge, but some tape over port 1 of the switch should do. Thanks for the insight, Doug! And if you see a problem like this, check the switch, even though it doesn’t make any sense.

Internet connectivity in the US, particularly rural areas, is awful. And there is really no excuse for it. The Paradox of the Best Network provides some insight into this (thanks to Patrick for the link, via his blog). I also envision general office buildings in the future that provide telecommuters a way to get out of the house and have access to shared resources. And everyone working in the building might be working for a distinct company across the globe. In the mean time, my 1Mbit with extra evil shaping connection to Kuhn and my 11Mbit wireless link to Chris will have to suffice.

No structure here… just some random goings-on:

For the last several kernel updates for FC4, my DVD sharing using GNBD hasn’t worked. I guess those special ioctl()s aren’t getting translated over or something. And NFS or SMB sharing an encrypted DVD just doesn’t do anything good at all (ignoring the fact that NFS seems really sucky with the latest FC4 updates). So, after months of not being able to watch DVDs, I gave up, and bought a USB drive cage to attach a DVD drive to Oak. Works perfectly. Despite performance issues and cabling evilness, I still can’t completely rule out a stack of USB drives RAIDed as a bulk storage solution, especially with all of the device mapper coolness in Linux. Too early to be thinking of that, though, as the computer storage fund hasn’t matured yet, despite the fact that Oak is at 99% capacity.

As Doug has mentioned, Asterisk and VoIP is still pretty neat. I’ve setup a teliax account, since they have pricing like nufone with a whole lot of rate centers worth of DIDs (they’re essentially a Level3 reseller). So, we just need to get some VPN’ing set up. I’m in desperate need of some UT, and I think VPN might be a useful substitute for a LAN party this winter.

My grandfather (on my Dad’s side) has been in the hospital on and off for more than a month now. Currently he has pneumonia, is very weak, and not entirely coherent. Your prayers would be appreciated for what could be a difficult Christmas season for the family.

Things at Ship are going fairly well. I did shoot myself in the foot with the “ip arp inspection” feature of the Sup720 this week though. Does 15 pps of ARP traffic seem like a good default threshold for shutting down trunk ports to you? Me neither. Of course, I asked that question after two ports had been err-disabled. Hopefully Tim and I will get to do a real test of some VMPS soon, too.

I’ve added the first WAN link to the thtech network: 802.11g to Chris’s house. We used garden hose as conduit for about 250′ of CAT5e from my shed to the WAP. The WAP was mounted on a pole inside of a tupperware container. Ethernet handled the distance fine, but a trivial POE implementation arrived at by splicing the power adapter that came with the WAP into the CAT5 didn’t work. There was too much resistance on the line for the 5v power to overcome. So, we tried stepping up to 12v with a DC-DC converter on the other end. This would result in the WAP partially powering up, but continually rebooting. I figured that this was due to the WAP drawing too much power when it tried to power up the radio. I purchased a higher capacity DC-DC converter at WalMart, and all has been well. We’re currently getting 600 KB/s, but earlier tests had us at about 1.1MB/s. We’ll have to do some antenna tweaking to try to remedy that.

UPSes are great things. During the storm today in the Shippensburg area, my house lost power. When I got home from work an hour later, the UPSes were still running fine, but the Internet connection was down. When power returned, all was well, so it would seem that Kuhn has either no or insufficient backup power for its equipment… great.

In other news, Intel seems to be pretty good about warranty returns. Despite the fact that they don’t accept RMA requests online (only by fax or mail), they have acknowledged my RMA request by mail, and receipt of the broken NIC, almost instantly by e-mail. My replacement NIC is now on its way.

I’ve spent maybe 10 hours on this now (not all this week, mind you, but still). Getting WPA-PSK w/ TKIP certainly isn’t as easy as it should be, but that is probably entirely due to driver issues. Seems you can’t buy a great 802.11g card for Linux.

I had tried various versions of the Linux IPW2100 drivers, 1.1.0 most recently, and always ended up getting errors saying that the IPW_IOCTL_WPA_SUPPLICANT ioctl was not available. This is a symptom of a driver that doesn’t have the WPA support, but lsmod clearly showed the TKIP and other encryption-related modules loaded. Google suggested using the load and unload scripts provided with the driver, and to check the initrd for an old driver that might be overriding the freshly-compiled one. modinfo confirmed that the new drivers were getting loaded… still no luck. That’s where I was for a long time, retrying every once in awhile to see if anything was happier.

As it turns out, there is a problem in the way the drivers are compiled as modules which can be fixed with this patch (local cache). Keep in mind that the post I’m referencing here is only two weeks old… so, I’m probably not the only one having this issue. I’m somewhat amazed (and very happy) that Google has indexed it that fast.

Now, technically, that was enough to fix my problems. However, I spent the next 45 minutes or so trying to figure out why my connection would reset several seconds after coming up… which turned out to be another instance of wpa_supplicant in the background screwing things up. Tip: run one wpa_supplicant at a time.