Search by search, click by click, the Internet economy is largely built on a paradoxical but wonderfully simple concept: free searches eventually add up to billions of dollars.
No one knows this better than Google, which has parlayed its search dominance into extraordinary advertising treasure.
But for all its prominence–Merriam-Webster recently crowned “google” a verb–the scramble to link the searching public to countless online sales pitches is far from over. Microsoft, Yahoo and others aren’t giving up.
Earlier this month, Microsoft made the latest move by introducing a new, no-frills search page called Live Search. The idea is to capture and hold Microsoft’s hundreds of millions of existing users so they don’t go to Google.
At stake is the way people handle the most basic of Internet needs: everything from shopping for birthday presents to planning a vacation to tracking down the most arcane scientific research.
Behind this public maneuvering, the Web’s biggest competitors are pursuing ultra-secret strategies to build the world’s biggest supercomputers and deliver the best search results. Extensive interviews on the corporate campuses of Google, Yahoo and Microsoft, as well as a review of public records and academic research, reveal the ferocity of the fight.
Ultimately, winners and losers will be determined by the fickle tastes and habits of consumers like Justin Dean. An electrical engineer attending Northwestern’s Kellogg School of Management, Dean, 27, is loyal to Yahoo, although his friends all use Google.
The reason for Dean’s preference: “I can get where I need to go in a fewer amount of clicks.”
In courting such consumers, all the Internet rivals share a goal: They’re gunning to overtake Google and stake a claim to the Internet riches that lie just beyond the search function on your computer screen.
To understand how Google plans to prevail, and how computer users benefit from this Internet battle, it helps to have an inside look at what makes a Google search different. And how adversaries plan to outsmart the search-engine giant.
When Google search was lost
Last summer, as the worst hurricane in modern American history gathered power south of New Orleans, many computer users around the globe did what they always do when searching for answers. They went to Google.
But when they typed “Katrina,” the initial results turned up nothing about the storm’s path or potential for destruction. Instead, the world’s most powerful search engine returned trivia about the woman’s name and a company in Virginia run by a Web designer named Katrina Blankenship. Her Katrina.com had zero information about hurricanes at the time.
At this critical moment, Google’s search engine was lost–or at least significantly off course.
But those who turned to Google’s biggest rival, Yahoo, found just what they were looking for: news stories and other information about the pending disaster. Editors at Yahoo had begun working on Katrina-related content while it was still just a tropical storm.
After their embarrassing initial response to Katrina, Google executives said they made adjustments. “The main thing that we learned is that we have to be faster,” said Alan Eustace, Google’s senior vice president of engineering and research. “We have to update things faster. We have to get new information in faster. We have to react to things quicker.”
So what happened at the beginning of this year’s hurricane season? When the first tropical storm, Alberto, arrived in June, Google’s top result for an “Alberto” search was Alberto-Culver, a major manufacturer of personal-care products. Next came Atty. Gen. Alberto Gonzales.
Google search experts defended these results, saying a low-level news event should not automatically trump other results for a given word. The majority of people actually might have been looking for information about personal-care products or the attorney general.
At Yahoo, the first entry read: “Downgraded Alberto spawns tornadoes.”
Google’s hurricane stumble underscores that while it remains the behemoth of search, there’s plenty of room for competitors to steal its customers. The Katrina episode also illustrates the fundamentally different approach Google takes to search.
Simply put, its rivals allow for more intervention by the human hand in the form of editorial judgments about what makes for a good search.
Google, by contrast, puts its faith in a massive string of computer code to harness the Web, a sort of Big Brain on autopilot.
So far, it has served the company well. Google’s Web sites generated $3.4 billion in revenue last year, most of it from search. And it has secured 44percent of the U.S. computer search market — more dominant, by comparison, than even McDonald’s in fast food or Coca-Cola in soft drinks.
Microsoft has only a fraction of that traffic, but it wants to catch up by copying Google. Its new Live Search page essentially seeks to mimic Google, offering a bare-bones search interface. The company’s MSN page will remain intact as a separate portal that, like Yahoo, seeks to hang on to users by presenting news, weather, stock quotes and other content on the same page as their search engine.
Google doesn’t do that. It prefers a strikingly barren main page, backed by an intricately fashioned search system.
At the heart of Google’s philosophy is something most people learned, and quickly forgot, in high school: a mathematical algorithm. At Google, the algorithm is paramount. It’s the series of instructions that provides the electronic flowchart needed to tell the synapses in Google’s brain how to work. They proudly call it their secret sauce.
Central to the algorithm is PageRank, a patented formula that helps order results based largely on popularity. If many sites with high PageRank link to a Web page, then it is also deemed important and receives a high PageRank.
For example, in a search for “Chicago baseball,” PageRank likely would give greater weight to the Web site of the Chicago White Sox, with its many inbound and outbound links to other major sites, than it would to a teenager whose obscure blog tracks each White Sox game.
Google co-founder Larry Page puts it this way: “You’re important if other people think you’re important.”
But Google factors in more than just popularity. In total, it examines about 200 signals, ranging from how often keywords appear on a page to their proximity to each other. “John F. Kennedy” appearing next to “biography” has a different value than a document where those two phrases are paragraphs apart.
Before Google or any search engine can rank results, it first must scan the Web for everything it can find. To do that, Google deploys automated programs called Googlebots. Web sites known to be important and frequently modified, such as those containing breaking news, are scanned as often as every few minutes. Those with less traffic and fewer changes may be checked only once every few weeks.
Once the bots collect the information, machines called index servers begin to organize the booty. Like the back of an enormous textbook, the index is the database that allows search engines to essentially alphabetize their information. The more pages and documents in the index–and the fresher they are–the better chance users have of finding just what they’re looking for.
Google claims it has three times as many documents stored as its competitors, although there is no way to independently verify that. With the greater depth, industry observers and even some competitors say Google often is best for finding the most obscure documents.
“If you have a rare query, go to Google,” said Steve Berkowitz, who left a job as CEO of the search engine Ask.com to become Microsoft’s senior vice president of online services. “They can have that 3 percent, and I’ll take the other 97 percent.”
Yahoo also believes its top competitor’s approach has weaknesses. Yahoo employees define a high-quality search as one where users find something of interest. Perceived comprehensiveness, they call it. In Yahoo’s view, depth like Google’s can produce irrelevant or distracting results.
“I can easily double the size of the index and not contribute to the perceived comprehensiveness,” said Jan Pedersen, Yahoo’s chief scientist for search and marketplace.
Think of it in terms of finding books to read. Google is like going to a premier university library where the shelves extend forever. Yahoo is more like a Barnes & Noble, where the offerings are tailored to the latest consumer tastes.
At Yahoo, software engineers regularly test sets of tens of thousands of randomly selected query terms. After those queries are run, editors mark up what results are useful or not. Then Yahoo’s programmers write commands that help its search engine reflect the judgment of the editors.
“We think that it definitely adds to the product, as opposed to the kind of a purist view that you have to have an algorithm that necessarily does these things,” Pedersen said.
Even at Google, the beloved algorithm requires constant tweaking by human beings. And sometimes the smartest fixes arise from unexpected places: One of the company’s biggest breakthroughs came from an intern named Gautham Thambidorai.
Just two months into his internship three years ago, Thambidorai, then 24, couldn’t believe his eyes as he repeatedly ran tests against about 100 lines of new computer code. Hundreds of engineers had shined and polished that programming for years, but he had found a bottleneck and reduced the load on Google’s servers by 15 percent.
If Boeing Co. discovered an equivalent improvement in its production lines, it would shave a day and a half off the 11 days it takes for final assembly of a 737.
“They told me about the magnitude of the money it was going to save, and I got all excited,” said Thambidorai, a native of India.
Three weeks after his discovery was verified, it was incorporated into every Google search routine. And yes, they gave him a job.
The contest among Web giants is not being fought just with such 21st Century technology talent. In one of the ironies of the Internet age, they also are competing in a good old-fashioned bricks-and-mortar building boom.
Arms race along the Columbia
Inside a fenced property known locally only as “Project 02,” eyeball-like security cameras peer down from light poles. A new electrical substation stands ready to pull in megawatts of power. When someone lingers too long, even on property nearby, security guards cruise over to suggest moving along.
“Project 02,” rising in The Dalles, Ore., isn’t some secret government research facility. It is Google’s latest broadside in a high-tech arms race being waged along the banks of the mighty Columbia River, where Google, Yahoo and Microsoft are all building new data centers.
The search for locations to construct supersize facilities has intensified in recent years. With Internet users demanding high-capacity services such as three-dimensional images, video, online applications and storage, the need for computing power also is skyrocketing.
This helps explain the construction contest in Oregon and Washington, where the fast-flowing Columbia produces cheap hydroelectric power.
Search companies need to fill the hunger for expanded computer capacity the same way farmers supply food, said Debra Chrapaty, the vice president of MSN operations.
“They are the farms of the future,” she said. “People are hungry, and we need to find new ways to farm.”
Microsoft, in the midst of a major capital-spending spree to counter Google, early next year will open the first phase — 450,000 square feet–of the largest data center it has ever built. That project, on 74 acres in central Washington state, is about 175 miles to the northeast of where Google is building. Just up the road from Microsoft, Yahoo is working on a new 45,000-square-foot center in Wenatchee, Wash.
Besides powering all the advertising, e-mail and applications that search companies offer, data centers are essentially the assembly line of the Internet. But instead of holding parts to cars or refrigerators, the Web manufacturing plant stores and organizes all human knowledge, a goal as old as the 3rd Century B.C. Royal Library of Alexandria.
Outsiders say Google still maintains bragging rights for size and efficiency when it comes to computing infrastructure. It builds most of its own computers, strapping PCs together with Velcro so they can easily be swapped like burned-out light bulbs. Since monitors are not needed for its servers, it builds the machines without video jacks, saving a dollar or two per server.
The two new warehouses in The Dalles–each a football field long, with four-story cooling towers to chill the tens of thousands of computers inside–show just how serious Google is in its belief that virtually every scrap of information has value. It believes everything from obscure blog postings to academic papers to chicken soup recipes to bizarre homemade videos can be turned into advertising revenue.
Like other search engines, Google is able to offer all of this through something called distributed computing.
While some supercomputers are built with computers linked in series–like sled dogs pulling in one direction–Google’s version is more like an owner holding each animal on a separate leash.
This parallel architecture, built on low-cost computers cobbled together, allows the load of millions of queries a day to be distributed across thousands of different machines.
As recently as 2003, Google had only about 15,000 data center PCs. But competitors suspect it might now have nearly 500,000 harnessed worldwide. Google won’t confirm the figure.
With more than $1 billion allocated this year for data centers and real estate, Google is spending heavily on infrastructure as it tries to outmuscle competitors and block the entry of new players. And Google’s traffic is growing so fast, it needs to add capacity simply to meet demand.
“We’re not banking random data centers for a rainy day,” Google CEO Eric Schmidt said in an interview. “Everything is running. And every time we add another one, it’s running too.”
In the competition to be the biggest player on the Internet, few of Google’s competitors can match such resources.
It has the biggest search engine, a remarkable computer backbone and top-notch human talent. The challenge for Google as it contemplates its future: Make certain that’s enough. After all, to the average computer user, Google’s obsessive secrecy can’t hide the fact that its rivals have narrowed the gap for most searches.
“In this game now, there is no `secret sauce,'” said Ashok Chandra, a longtime industry guru who recently became Microsoft’s general manager for Search Labs. “There is just sauce.”
———-
mccormickj@tribune.com
dgreising@tribune.com
– – –
ON THE INTERNET
Find Part 1 of Gunning for Google and video, audio and photos from the Googleplex at chicagotribune.com/google
– – –
How engine works
When searchers use Google, they are harnessing one of the world’s most powerful supercomputers. A search typically takes less than half a second, but it is a complex journey from keyboard to results, one that typically makes at least two stops often thousands of miles apart.
1 Google creates its own version of the Internet for people to search, using automated programs known as Googlebots. They constantly crawl the Web’s billions of pages in search of new information. Web sites known to be important and frequently modified, such as those containing breaking news, are scanned every few minutes. Sites less frequently updated may be scanned every few weeks.
2 The Googlebots feed data to Google’s central network (most likely in Silicon Valley) with the key information from a Web page: the URL (www.abcd.com), the full text of the page, references to images and other embedded files from the page, and specific information the site owner creates about the page called metadata.
3 At the central network, the information is organized into an index, much like the one in the back of a textbook. Every word that could be used in a search query is listed in Google’s central computers along with information referencing Web sites where the word can be found.
4 The giant index is broken into “shards” and sent to data centers worldwide. (Data centers are increasingly larger facilities made up of thousands of servers wired together.) Because data centers may have slightly different versions of the index, based on when they received the last update, users on two opposite ends of the country may get slightly different results for the same search.
Searching and ranking
When people search Google, they are essentially asking the company to find every instance of the term in its massive index and rank the corresponding documents by their relevance.
1 The user types a search query.
The typical query is only two or three words, which can make finding the most relevant results challenging. Roughly 1 in 10 queries is misspelled.
2 Before Google can provide any information, it identifies the searcher’s approximate location through his or her Internet Protocol (IP) address. The IP helps speed up the search by sending the request to the nearest data center. It also allows Google to identify geographically appropriate ads.
3 The query is sent to Google’s central network, then redirected to the nearest data center.
4 At the data center, the search term is run through Google’s index. Matching terms are sent back to Google’s central network and then back to the user with a summary of the Web page, called a snippet.
THE “SECRET” SAUCE
Google determines which Web sites are most relevant to a search term by using its “secret sauce.” The formula weighs more than 200 measurements such as the number of times the search term appears on a Web page, the number of visitors to the page and the PageRank–the number of sites linking to the page (and the popularity of those sites).
Sources: “The Search,” Tribune reporting
Chicago Tribune / Max Rust, Steve Layton and John McCormick
– See microfilm for complete graphic.
