Basically, they took a web crawler like Heritrix (archive.org) or Scrapy (a handy Python implementation good for prototyping) and just started fetching web pages.

Eventually, they have a database of 350,000 websites, along with two million links between these domains. Any set of web pages within a given domain may have hundreds of hyperlinks to a dozen other domains, but a link from any page in one domain to any page in another domain becomes a relationship between two domain nodes in a graph. Presumably they used something like neo4j.org to store these relationships (cf. jokes about relational databases being bad at storing relationship information).

Then the actual hard part comes in. They link to a high level paper on rendering a visualization of that much information, and then used a similar algorithm to determine placement of each node. The size of each node is presumably the number of links in/out and the color coding is geographic (and probably not considered in this algorithm).

So now they have a database describing all these nodes and relationships, and an algorithm to draw a gigantic image of all of them in 2D space. They used GPU-based parallel processing techniques (probably with NVidia's CUDA language) to crunch all the numbers to generate the final image.

Finally, the image ends up being pretty large at a reasonable zoom. A scaled map of the Earth zoomed a bit above street level would still be about 125 miles on a side. So they use Google Maps API to manage small chunks of the image at various zoom levels. (They also end up rerunning that algorithm a few times to generate smaller images for each zoom step, including one at good old 1024x768).

Pretty neat. Would love to see their writeup.