Former NSA cryptographer speaks out about his eavesdropping program, ThinThread
The Secret Sharer: www.newyorker.com/reporting/2011/05/23/110523fa_fact_mayer
"As Binney imagined it, ThinThread would correlate data from financial transactions, travel records, Web searches, G.P.S. equipment, and any other “attributes” that an analyst might find useful in pinpointing “the bad guys.” By 2000, Binney, using fibre optics, had set up a computer network that could chart relationships among people in real time. It also turned the N.S.A.’s data-collection paradigm upside down. Instead of vacuuming up information around the world and then sending it all back to headquarters for analysis, ThinThread processed information as it was collected—discarding useless information on the spot and avoiding the overload problem that plagued centralized systems. Binney says, “The beauty of it is that it was open-ended, so it could keep expanding.”"
Very interesting. But I would be more surprised if the NSA wasn't doing this.
That's really interesting from a design standpoint. Thinking about that, it's like there's a really good principle for designing complex systems in there. By not exposing the analysts to everything coming in you make the system more effective as they can get the information they need without too much additional data mining or scanning through wads on information.
On top of that, because you don't have to worry about the complexity of the data to the same degree, it provides more room for analyzing other complexity that might with adding new systems. With the simplification, you can channel your focus into more specialized areas by narrowing the scope of whatever problem you may be dealing with (since you don't have as much of your mental effort being focused on other complex tasks that would divert your focus).
It's like choosing a good design for a program to simplify things so you're not neck deep in complexity.