Friday, February 28, 2014

Redneck Broadband - fixed!

the beginning of the story

It was my fault!  Monday's 28Mbps was not an anomaly. At one point in the installation, the WiFi hot spot crashed and I had to do a factory reset.  Little did I know that factory reset disables the 4G radio: our throughput dropped to an abysmal 200Kbps.  Once re-activated, 4G speeds came back and remain consistent.

After crowing success, then lamenting failure, I'm back to shouting "success!"

Thursday, February 27, 2014

Redneck Broadband - Disappointing Postscript


I regret to report that as I leave my friends place in Tennessee, I leave them with very unstable broadband.  In my last posting, I reported that we managed to get 28Mps broadband connection the house.  That lasted only for a day, and suddenly our source Verizon WiFi hotspot failed to deliver that bandwidth.

I had no idea that cell service was so variable over time.  The same equipment that gave us a consistent 4G four bar signal on Monday, now can only get a 3G two bar signal today.  In the last forty-eight hours we've been lucky to get an intermittent 200Kbps signal.

We've canvassed the ridge looking for a new location for the WiFi hotspot and finally found a place that consistently delivers a 1Mbps 3G signal.  Interestingly, this a place that delivered essentially a zero signal on Monday.  The variability of signal strength at a given spot is a serious impediment to success of this project.

Since I leave tomorrow, I'm out of time.  There's not much more than I can do on this visit.  My hosts are still pleased with the results.  The bandwidth is forty-five times faster than the old 28.8Kbps of the old  system. 

We'll see how this works for the long run.  Maybe it will settle into a more consistent pattern.  However, I wonder how that will change as Spring comes and the heavily timbered forest of the area leafs out.

I've briefed my hosts on the necessity to monitor the bandwidth and move the hot spot.  I had hoped that this would be a set it and forget it installation, but fear this will be a system that will need constant baby sitting.

Monday, February 24, 2014

Redneck Broadband - from 28Kbps to 28Mbps in an Afternoon

It surprises me that we in the United States have not treated nationwide deployment of broadband in the twenty-first century in the same manner that phone and electricity service was spread in the twentieth century. Is there a doubt that those initiatives last century were a economic boon and a revolution in the standard of living?  It seems pretty clear to me as Google contemplates gigabit fiber to Nashville just sixty miles away that the free market has not spread broadband effectively to rural areas.  I'm sitting at a friend's house in rural Tennessee that has effectively no access to broadband. 

[author note: satellite was looked into a year ago and no plan was found to be appropriate. However that may have changed and we're currently re-evaluating that option.  One of the factors in the use of satellite is lag due to the distance the signal has to travel - lag is important in evaluating broadband for uses including games or interactive applications.]

AT&T, the local landline provider, from the perspective of this household, is seemingly uninterested in expanding the availability of DSL into these rural areas. The cell companies aren't doing it either. It is rough terrain and cell signals don't propagate well into low valleys and hollows that spread off the plateaus. Cable is miles away and not visibly expanding.

This is a tragedy for the people like my friends here. There is only so far one can go with a 28.8K dialup modem. So much of the Web these days assumes that the users have a fast connection to the Internet.

I've taken it on myself to help my friends get at least some semblance of broadband to their home. I know that the small town about four miles away has both AT&T wireless and Verizon cell towers. If I could get a cell hub and a microwave relay system, I might be able to get broadband to my friends.

Here's the plan: 

From the cell tower in town for miles away, I plan on catching the cell signal at the top of the ridge above the house with a cell data hub (for today's test, I'm just using my Verizon cell phone). That will offer a small circle of WiFi at the top of the ridge. From there I will use a TPLink TR-WR702-N in client mode to repeat the WiFi onto a standard ethernet cable.

The ethernet cable goes from there go to a solar/battery Power-over-ethernet adapter that will forward the network to Ubiquti Nanobridge M5 microwave relay.

All this equipment will be powered with a 120AHr 12V RV battery. Eventually, we'll add solar panels to keep the battery charged so my friends don't have to cart heavy RV batteries up the hill. 

Aiming the microwave dish was easy.  Since it only had a quarter mile to go, I just had to point in the general direction of the house.

Meanwhile, the house will have a paired Nanobridge aimed up at the top of the ridge. That'll extend the network onto the a wired ethernet network in the house. Finally, that network will feed house WiFi network.

All said and done, we've exceeded expectations by more than two and a half times: 26.3Mb, I had hoped for 10Mb. That's quite the revolutionary change from the 28.8K dial up signal that has been the only option prior to this point.

During this week that I am here, we will replace my phone with a Verizon Mobile Hotspot.

Please see the follow-up to this post:  Redneck Broadband - disappointing postscript and then the triumphant follow up to the follow up: Redneck Broadband - fixed!

Monday, February 17, 2014

Single Process Multithread vs Multi Process Single Thread

For the first time in my years at Mozilla I have a wonderful opportunity to seriously work and experiment with Socorro.  Back in the days when there was a great brick wall between Dev and IT, I couldn't touch or even gaze upon a production or staging machine.  If something went wrong with my software, I'd have to make a request to see logs, wait for someone in IT to respond.  Frequently, I'd ask for logs about a specific error, but receive just the log line with the error on it with no context.  I already knew the name of the error.

Times are changing.  We're moving toward a model where developers like me are given sudo everywhere being versed in not destroying things.  This gives me an unprecedented opportunity to tune my software.  I built it to be tunable, but tuning has always been guesswork because tuning is an interactive process – something that is hard to do when you have to relay commands to somebody and you can't see what's on their screen.

My first big tuning project involves the Socorro processors.  They are I/O bound multithreaded Python applications.  We've always just set the number of threads equal to the number of CPUs on the machine.  We've always gotten adequate throughput and have never revisited the settings.

By the way, it is important to know that the Socorro processors invoke a subprocess as part of processing a Firefox crash.  That subprocess, written in C, runs on its own core.  While the GIL keeps the Python code running only on one core at a time, the subprocesses are free to run simultaneously on all the cores.  There is a one to one correspondence between threads and subprocesses.  Each thread will spawn a single subprocess for each crash that it processes. Ten threads should equal ten subprocesses.

My collegue, Chris Lonnen, has really wanted to drop the multithreading part and run the processors as a mob of single threaded processes.  In the construction of Socorro, I made the task management / producer consumer system a pluggable component.  Don't want multithreading?  Drop in the the single threading class instead.  That eliminates all the overhead of the internal queuing that would still be there by just setting the multithread class to use only one thread.

The results of comparing 24 single threaded processes to one 24 thread process startled me.  The mob was between 1.8 and 2.2 times faster than the 24 thread process.  That did not sit well with me as a staunch supporter of multithreading.  I starting playing around with my multithreading and made a discovery.  For the processor on the 24 core machine, throughput did not increase for anything more than twelve threads.   The overhead of the rest of the code was such that it could not keep all 24 cores busy.

Watching the load level while running the 24 process single thread version test, I can easily see that it keeps the server's load steady at 24 while the multithread version rarely rose above 9.   That got me thinking, could I get more throughput with multiple multithreaded processes?  Could that out perform the 24 single thread process?

The answer is yes.  I first tried two 12 thread processes.  That approached the throughput of the 24 single thread processors.  I noticed that the load did not ever rise above 16.  That told me that there still was room for improvement.  I tried twelve 2 thread processes, that matched the single thread throughput, but with running the load level at eighty percent of the single thread mob test.  Various combinations yielded more gold.

The best performer was four 6 thread processes:

# of processes # of threads average # of items processed / 10 mincomment

24 1 2200
1 24 1000 current configuration
1 12 1000
2 12 1800
12 2 2000
4 6 2600

This is just ad hoc testing with no real formal process.  I ought to perform a better controlled study, but I'm not sure I have that luxury right now.

Tuesday, February 11, 2014

Today We Fight Back

Today, my blog is dedicated to the Stop the NSA mass
surveillance campaign.   

I sent the following message to my congressional representatives today and I encourage you to do so, too.  See the giant black box at the bottom of the screen for instructions.
As your constituent, I am deeply concerned about the National Security Agency’s (NSA) mass surveillance programs. We need real reform to stop the NSA's mass collection of our information.

I urge you to support the USA Freedom Act (H.R. 3361/S. 1599), an important first step in stopping mass spying, reforming the FISA court, and increasing transparency. But reform shouldn't stop there: please push for stronger privacy protections that stop dragnet surveillance of innocent users across the globe, and stop the NSA from sabotaging international encryption standards.

I'm also urging you to oppose S. 1631, the so-called FISA Improvements Act. This bill aims to entrench some of the worst forms of NSA surveillance and extend the NSA surveillance programs in unprecedented ways. It would allow the NSA to continue to collect the phone records of hundreds of millions of Americans not suspected of any crime—a program I absolutely oppose—and could expand into collecting records of Internet usage.

The NSA mass surveillance programs chill freedom of speech, undermine confidence in US Internet companies, and run afoul of the Constitution.  We need reform now.
Metadata is Data

Our constitution is supposed to protect us from exactly this
sort of government intrusion into our lives

Yes, we freely give lots of information to corporations that
the government should NOT be allowed to have.
What's the difference?  

Google & Microsoft cannot arrest us. 
Apple & Facebook cannot execute us.

Our government does have BOTH those powers
and MUST honor our constitution 

Read the story of our own government run amok
Brandon Mayfield  
a US Citizen held without charge
within his own country.

Tuesday, February 04, 2014

Squeezing More Onto a Backup Hard Drive

I've got a back up hard drive that is running out of room, but I don't want to replace it.  I have a more interesting plan.

Given a file system tree, I know that there are files that are identical throughout.  Since the tree represents a backup, the files are not going to change.  If I could take groups of identical files and hard link them together I could save a lot of disk space.

Here's my solution:  I've taken the Socorro processor and replaced its iterator, the transform function, the source and the destination:
  1. the iterator is now a file system walker that just yields the full pathnames of each file in the target file system.
  2. the transform function is now a takes a pathname, uses os.stat to fetch the file's inode, then starts a subprocess to run md5sum on the file to produce a hash
  3. the destination is now a table (called "files") in a Postgres database of this form:   
  • pathname text,
  • inode text,
  • checksum text

Running the program, and setting it to 16 threads, it chewed through the four terabyte file system in about seven hours making a md5sum for every file and saving the information about the file in the table in Postgres.  Then I ran this query:

A as (select checksum, inode, count(inode) as n from files 
      group by inode, checksum order by 3 desc),
B as (select checksum, max(n) as m from A group by checksum),
C as (select b.checksum, a.inode from B b join A a 
      on b.checksum = a.checksum and b.m = a.n),
D as (select f.pathname, c.inode from files f join C c 
      on f.checksum = c.checksum and c.inode <> f.inode)
select d.pathname, max(f.pathname) from D d join files f 
      on d.inode = f.inode group by d.pathname;

The output of this query is a series of rows of pairs of file names.


These file names are then used to essentially do this shell command:

    ln file2 file1

This replaces each redundant file with a hard link.

My nearly full 4TB hard drive now has 1.5TB free!

Let's go back and re-examine that query.  It may not be the most efficient or elagant query, but it was fun to write.  I hope it does what I really think it does.  It uses four common table expressions, an alternative to subqueries that generally makes SQL easier to write and understand.

A) this first query goes through the "files" table identifying all the files that are already hard linked together and giving a count of how many files participate in the hard link.

B) this query then identifies the hardlink groups with the most participants associated by the checksum.

C) this query identifies the inode for the each of the hardlink groups for which the number of participants is maximized for a given checksum.

D) this query identifies each of the pathnames for each file that does NOT participate in the hard link group with the maximum number of participants. The inode in the select list is the inode of the hard link group with the maximum number of participants.  In other words, the output of this query is the files that I want to make into hard links to the listed inode.  I don't have a way of making a hard link by specifying an inode, so I must now find a file name associated with the target inode:

E) (unnamed) the final query takes the inode and finds a pathname that uses it.  The final output is the pathnames of the two files that I need to hardlink together.

Are you a DBA?  How could I improve my query?  Does it really do what I think it does?