10 big changes with search engines over my 20 years of covering them

Today is my 20th anniversary of covering search and search engines. To mark the occasion, I wanted to reflect on some of the big changes that I’ve seen over the past two decades of covering the space.

1. The search revolution

Chances are, the first resource you turn to if you have a question about something is a search engine, whether it be Google, Siri, Bing, Yelp, others or a combination of services.

This simple act, that you likely don’t think twice about, was a highly revolutionary change to how people sought information. Before popular consumer-focused search engines emerged just over 20 years ago, people got answers the same way they had for hundreds and thousands of years: largely by asking other people.

If you needed an answer, you turned to people like a teacher, a professional, a best friend or a librarian. Sure, there were also tools to use: libraries, library catalogs, yellow pages and professional databases like LexisNexis. But for most people, getting an answer to many questions meant asking others.

Enter search engines, our new best friends that seemed to have an answer for anything we needed. They revolutionized how we gather information in the way the smartphone changed how we communicate — yet unlike the smartphone, search has been a quiet revolution. Perhaps it seemed so natural that we didn’t think to be in awe of the profound change it delivered.

If you have eight minutes and want to understand more about the search revolution, here’s a TEDx talk I did on the subject a few years ago:

2. The Google revolution

Apple didn’t invent the smartphone. It just perfected the concept, creating a version that everyone wanted to mimic. Similarly, Google didn’t invent the search engine. Before it existed, we had several of them — popular, important ones like Open Text, Infoseek, WebCrawler, Lycos, Yahoo, AltaVista and Excite.

Google perfected the search engine, or at least greatly improved what it could be. It proved that by analyzing how people linked to web pages, you could get both highly relevant and comprehensive search results. You could search for both popular and obscure topics and receive great answers in response.

Google also thrived by being a “second-mover” player. The early search engines were largely prevented from making money directly off search because of outcry over paid results. These are commonplace and accepted today, but after an initial experiment by Open Text in 1996 sparked upset, other players shied away.

If Google had emerged earlier, it might have gotten distracted trying to become a “portal” like its competitors rather than focusing on great search. Instead, it emerged at just the right time. GoTo (later Overture) largely revived the idea of paid results. That new acceptance, coupled with Google’s better search technology, made it a powerhouse.

Over the years, Google eclipsed its competitors. Today, Google is top of mind when you think of search. In most countries, other than a few like China and Russia, it’s the most-used search service. It has earned that ultimate compliment for a business, having its name turned into a verb. To google is to search.

3. Google as a destination

When Google was young, its goal was to point people to other sites as quickly as possible. Cofounder Larry Page spoke about this in an interview with Playboy around the time the company went public in 2004:

PAGE: We want you to come to Google and quickly find what you want. Then we’re happy to send you to the other sites. In fact, that’s the point. The portal strategy tries to own all of the information.

PLAYBOY: Portals attempt to create what they call sticky content to keep a user as long as possible.

PAGE: That’s the problem. Most portals show their own content above content elsewhere on the web. We feel that’s a conflict of interest, analogous to taking money for search results. Their search engine doesn’t necessarily provide the best results; it provides the portal’s results. Google conscientiously tries to stay away from that. We want to get you out of Google and to the right place as fast as possible. It’s a very different model.

PLAYBOY: Until you launched news, Gmail, Froogle and similar services.

PAGE: These are just other technologies to help you use the web. They’re an alternative, hopefully a good one. But we continue to point users to the best websites and try to do whatever is in their best interest. With news, we’re not buying information and then pointing users to information we own. We collect many news sources, list them and point the user to other websites. Gmail is just a good mail program with lots of storage.

I’ve bolded the key points. Page, and Google generally, saw that having their own content was a conflict of interest with the search engine’s core job of pointing people to the best information.

At that time, Google already had content conflicts, such as non-search services like Gmail or Blogger. Over time, the conflicts have grown. For example, it’s clearly in Google’s interest if people are viewing videos on Google-owned YouTube. Consider these results:

That’s a search for “rogue one trailer” on Google. Google’s own YouTube service has the top listing, with a giant image that dominates the page. Apple, which also has the trailer, comes way at the bottom of the page. Advantage: Google.

Why’s Google favoring itself this way? It could be that the YouTube experience is better than the Apple one — and it is, in my opinion. But it could be that Google knows there’s an advantage to showing YouTube content more prominently. It could also be any number of other things, as well. But by having its own video service, whatever the reasons, the conflict issue remains.

Perhaps it was inevitable that some conflicts would happen. Local search is so tied to local reviews that it’s difficult for Google to have a good local listing service without hosting its own reviews — though Yelp, for one, would suggest that Google might avoid conflicts by licensing reviews from Yelp and others.

For the most part, I think Google has balanced conflicts between search and its own products well and in the interest of users (Though there have definitely been exceptions). Still, as Google has grown — as its ambitions have given birth to an entire new company called Alphabet to house them — the conflicts have multiplied and will likely continue to haunt the service.

For more on this topic, see my past article: How Google went from search engine to content destination.

4. The “searchphone” revolution & “always-on” searching

That’s not a smartphone in your hand. It’s a searchphone.

Sure, you use it to post to Facebook, send messages, do your email or Slack, watch videos and play games. But you almost certainly use that phone to get lot of answers. Wherever, whenever, you’re asking it questions.

If search engines were a revolution in how we sought information, smartphones were a second revolution that unleashed “always-on” searching. No longer was there a need to wait until you were back in the office or with a laptop to seek an answer or conduct research. Now, you could search just as soon as the need emerged. And we do, with most of our searches now happening on mobile.

This is greatly changing things such as shopping, where searches leading to purchases now happen at any time of day or night. It even ties into things like television shows, where what people are watching can spike searches for related content.

Our smartphones haven’t shrunk to the size of Star Trek-like communicators, but in practice, they might as well be the same thing, in terms of that ever-present connection to a “library computer” with answers to everything.

I mean, imagine Captain Kirk being on some alien planet and asking “how long is this river” when standing in front of one. Yeah, that’s real now, here on Earth:

5. The death of “10 blue links”

If you’ve heard the phrase “10 blue links,” that’s largely due to Ask Jeeves and Jim Lanzone — its former vice president of product, later CEO and now head of CBS Interactive.

If they didn’t coin the term, they certainly were among the first and definitely popularized it as a way to reposition Ask Jeeves as something new and fresh. Rather than returning the usual “10 blue links” of search listings, Ask Jeeves promised to deliver more useful information in a new way.

Ten blue links have never really died. Major search engines still provide the traditional web links. But they also go well beyond them, providing direct answers and new ways to interact with search results, such as carousel displays.

Forgive this long screenshot (which you can click on to enlarge), but it illustrates the change:

On the left, a search for “cars” on Google in 2002 is shown. It’s all links, both for the ads and the free listings. The only image is Google’s logo.

On the right, the same search today on Google, as it appears on mobile. There are still plenty of links, but they’re more graphical in display. Apps links are shown in a colorful format. News stories have images and a swipeable carousel. Local listings are mapped and include buttons to initiate phone calls. This is far from the 10 blue links of old.

Expect search results to become even more graphic and interactive as the years progress. Related: Expect search to continue to go well beyond the keyboard, where we are increasingly speaking what we want and into a variety of devices, such as the Amazon Echo.

6. Agents 2.0: Predictive search & bots

In the early years of search engines, there was a belief in some quarters that “agents” would eventually automatically find what we need, making the idea of actively searching unnecessary.

That failed to happen. It’s also likely never to fully happen for years, if not decades. It’s hard for an agent to search for something you don’t know that you need until something immediate occurs, such as a broken pipe necessitating a plumber.

Still, some amazing things have been happening in the area of predictive search over the past few years. Google Now still surprises me on a regular basis when it suggests information I need before I actually do a search. Microsoft’s Cortana is growing as a challenger; Apple’s Siri remains farther behind but one to keep watching. A few of my past articles about them:

• The Amazing “Google Now” — When Google Searches Before You Think To
• Life With Cortana, Microsoft’s Predictive Search Challenger To Google Now & Siri
• With iOS 9, Apple’s Siri & Spotlight Search Get Smarter
• How Google Now, Siri & Cortana Predict What You Want

Also exciting is the cousin of predictive search, bots.

Predictive search is passive. The predictive search agent learns what you might need over time and suggests it, just as a good assistant might book your needed appointments before you think to ask.

Bots are active, similar to regular search. But rather than searching to find a page where in turn you may conduct a transaction, bots seek to be the all-in-one. They’re like telling a real-world assistant that you want to send flowers, and they make it happen.

Bots have especially gained attention over the past month as Microsoft and Facebook have introduced tools to help developers and publishers produce bots that can automatically handle tasks such as booking concert tickets and ordering flowers.

In the US, bots are still in the early stages. They also very task specific. If you want to send flowers, a specific florist’s bot will help you. But if you want tacos, then you need a tacobot instead. Eventually, robust meta-bots, bots that can accomplish many tasks, may emerge.

To learn more, here are some recent articles from me and others from our staff on bots:

• Conversation as the new UI: Microsoft makes its chatbot pitch at Build 2016
• Facebook’s Messenger Platform launched, allows businesses to build bots
• The week when digital marketing entered the bots era
• Inside the making of Taco Bell’s artificially intelligent, drunk-tolerant TacoBot
• How I sent Mark Zuckerberg flowers using a Facebook Messenger bot

Watch this space. Predictive search and bots are both exciting areas of search, and potentially the most disruptive areas.

7. The dawn of machine learning

Another emerging area of search is machine learning. Current search engines lean heavily on algorithms, complicated recipes written by human beings that involve analyzing hundreds of signals to decide what content should rank tops in response to a search.

What if the machines could just teach themselves? Last year, Google made the stunning revelation that machine learning was involved in “a very large fraction” of the searches that it processes. The machine learning process even has a name: RankBrain.

We’re still a long way from RankBrain independently knowing what it should show, however. You know those infinite moves in a game of Go? A game so complicated that Google recently stunned the world when its machine learning system AlphaGo bested the world grand champion?

There are infinite “moves” when it comes to determining search results. Search ranking isn’t a game with clear winning and losing moves. It’s a game of complete subjectivity. Two people in the same neighborhood searching for “abortion” may have entirely different views of what’s most relevant. A person in the US and one in the UK have different views of what they want to appear for a search on “football.” Add to that the difficulty of innately knowing what’s really the best content on any particular subject, if you’re not a subject expert on everything.

Search is hard. As best I can tell, the machine learning that Google is doing so far is primarily around matching complicated searches to more common ones, so that it can show similar results. That’s helpful because with more common searches, Google has better confidence in its results, especially because it can tell if there are problems if the expected click-through rates on searches begin to drop.

Still, the machines will get smarter. More and more, the machines may create their own rules to analyze various signals. Most fascinating, they may discover signals and relationships to improve search that might never occur to humans.

For more, I recommend reading the articles below:

• FAQ: All About The New Google RankBrain Algorithm
• How Machine Learning Works, As Explained By Google

If you have time and want to better understand the complexities of Google’s ranking system, I recommend this recent talk by Google’s Paul Haahr, below. The main talk is first, Q&A next and his slides at the end:

https://www.slideshare.net/SearchMarketingExpo/how-google-works-a-ranking-engineers-perspective-by-paul-haahr

8. How search has changed society

Who would have imagined when search engines emerged in the mid-1990s that they’d lead directly to a “Right To Be Forgotten” declaration by the European Court Of Justice 20 years later?

It’s just one of the many ways that search engines have shaped society since they emerged. The tools that make it so easy for us to find what we want also make it easy for some people to find what other people don’t want to be found.

Just last week, the University of California, Davis, made headlines when news emerged of its attempt to change Google’s search results to be more favorable to it in the wake of its infamous pepper spraying of protestors. UC Davis is hardly the first institution or company to try this. Indeed, there’s an industry around it. For search marketers, the most noteworthy thing about the attempt is probably how poorly it was done.

Just as people want to disappear content, others want to ensure it does appear. Search has enabled many businesses to emerge and grow; it sends them huge amounts of free traffic in a world where before search, businesses largely depended on advertising. It’s become so important that losing search traffic can impact the bottom line. Just ask eBay, Overstock, Demand Media and Verisign, all of which referred to Google algorithm changes in the past to explain revenue drops.

The legal moves involving search are fascinating. Search engines have been found, within the United States, to have First Amendment free speech protection. In Germany, an “ancillary copyright” law was passed to effectively tax search engines from linking to news providers. Spain passed a similar law. The battles over these continue.

The search engines themselves grapple with issues of what to list and how to do it. Do they leave suicide notes in their listings? Do they remove graphic war footage? Do they censor autosuggestions that include stereotypes? These are all real issues among others that Google has grappled with, discussed more in the video below:

The impact on society goes on…

• Students use search engines to prepare term papers; teachers use them to determine if students are plagiarizing in their papers.
• Criminals have used search engines to commit murder; search engines have been used to discover if dead people are still living.
• Politicians have been “googlebombed” to make unflattering things appear for their names, such as former US President George W. Bush and former US Senator Rick Santorum (not technically a googlebomb, but in the same spirit).
• People have turned to search to self-diagnose, making doctors worried. There’s even an unofficial disease, cyberchondria, for those who make themselves sick searching to find out if they’re sick.
• Potential employees use search engines to find jobs; businesses use search engines to check on potential employees.
• People “google” their dates. Don’t Be Shy, Ladies-Google Him! Check Out His Search Engine First is a classic of early advice on this, from the New York Observer in 2001.

And we google ourselves. Katy Perry does. Marge Simpson does. Most of us do.

9. The search marketing revolution

Until search, the easiest way to reach a mass audience was expensive and horribly targeted. You bought a TV ad and hoped it aired during a time when those interested in your ad might be viewing. Most wouldn’t be interested. Many likely fast-forwarded through your commercial.

Search changed all this. Suddenly, you could reach a mass audience for literally pennies. Moreover, your content was showing to an incredibly targeted audience, those who had expressly said they were interested in your product or service by performing a search for it.

Consider if you sold cars. TV charged you a high price to tell a mass audience that they should buy your car, when only a tiny number actually wanted to. Search charged you a tiny price to reach a mass audience that was explicitly searching for “cars” and thus likely ready to buy one. Plus, you only paid if someone actually clicked on your ad. You also paid nothing if they clicked on your free listing.

It is any wonder that search grew to be the powerhouse of online advertising, making up half of all online ad spend in the US?

In addition, search was the basis for the measurable or performance marketing that we consider commonplace on the internet today. Those who came up through search fought the battles for budget, aided by the tools and reporting to prove that returns were well worth the spend. Those learnings, those battles, that evolution set the stage for the other forms of digital marketing that have grown, from video to social to display.

I know those involved in search, both the on the SEM and SEO sides, can often feel like they’re still fighting battles and feeling unappreciated. Take a bow. You deserve it. At the very least, when someone expresses awe over Google’s auto-driving cars, you can say that it’s your content and ads that are earning Google enough money to build them.

10. The rocket science of search as normal

It’s sad how underappreciated the technology of search is. SpaceX lands its Falcon 9 rocket back on a floating ship, and the video goes viral. Apple releases a new phone, and the lines always form. Google’s own auto-driving cars draw gasps about how amazing the tech is.

Search? No rockets launch. There’s no glitzy new model that people line up for each year. It’s just search, science fiction turned into a reality so commonplace that it gets dismissed as normal or even boring.

Search is really is one of the most important “gadgets” that’s emerged in recent years. It’s right up there with the internet itself, which we also take for granted. It’s a miracle of science and technology.

The next time you perform a search, take a moment to consider the billions of pages that were gathered, stored and ranked to serve you up the best ones in less than second. Take that moment to appreciate the technology and people behind the process. They deserve it. I feel fortunate to have had a front row seat in watching it all unfold.