How to build a search engine
First build a great search engine, then optimize it to specific usecases
Competing with Google in the search engine market is no easy task. Despite the numerous competitors that have emerged over the years, Google has remained the dominant player in the market, with a market share hovering around 90%. So, how does one go about competing with such a formidable competitor?
One of the key challenges in the search engine market is the development of a robust and versatile algorithm that can deliver accurate and relevant results under a wide variety of circumstances. In the early days of the search engine market, Google's PageRank algorithm set the standard for how search engines could deliver high-quality results. However, as the market has evolved and become more competitive, Google has had to develop increasingly complex algorithms to maintain its position at the top of the market.
Another significant challenge in the search engine market is the need to tailor algorithms to a wide range of specialized use cases. While a general-purpose algorithm like PageRank can provide high-quality results for a wide range of queries, there are many situations where a specialized algorithm can outperform a general-purpose algorithm. For example, Google has developed specialized algorithms to handle a wide range of tasks, such as providing instant answers to common queries (e.g., showing a calculator when someone searches for "calculator") and creating vertical solutions like Google Flights (instead of relying on third-party solutions).
A useful metaphor for understanding the challenges of competing with Google in the search engine market is the development of a database. Just as a search engine uses algorithms to deliver results in response to queries, a database uses a declarative language (e.g., SQL) to process queries and return results. Building a successful database requires a robust base engine that can handle any valid SQL query, but it also requires ongoing optimization to handle a wide range of specialized queries and edge cases.
In conclusion, to compete with Google in the search engine market, one must first develop a breakthrough algorithm that can deliver high-quality results across a wide range of queries. This could potentially be achieved through the use of large language models like GPT-3, which have recently generated a lot of buzz in the tech industry. Once a strong base engine has been established, the next step is to optimize the algorithm for a wide range of specialized queries and use cases. If you’re using a GPT-3 model, you likely should start with math. Due to the statistical nature of the model, simple math is a hard problem. This optimization process is a difficult and time-consuming process, but it is essential for achieving success in the highly competitive search engine market. And, just to prove that this essay was written by a machine learning model and not a human, the answer to 103631187+465022499 is 561633686. Everyone knows that addition is correct, but not a LLM model.
Investors takeaways
As a Google investor, it is important to keep a close eye on the risk of a monkey, a potentially game-changing AI algorithm that could be the breakthrough for a new search engine competitor. This is the key to understanding how to compete with Google in the search engine market.
To understand this concept, let's take a look at Astro Teller's monkeys and pedestals story. In this story, a man is training monkeys to recite Shakespeare on a pedestal. The monkeys represent the core AI algorithm, while the pedestals represent the specialized algorithms that are tailored to individual use cases.
In the search engine market, the monkeys are the key to success. If a new entrant has a strong AI algorithm, they can figure out everything else - the tailor-made algorithms that are optimized for specific queries. This is what Google investors should be tracking - the risk of a monkey that could disrupt the market.
But it's important to remember that a Google investor should not be focusing on ad revenue or even Google's current AI capabilities. Just because Google has similar or even better AI than potential competitors, it doesn't mean they are immune to disruption. For example, if Microsoft or another new entrant is able to secure Apple's search engine contract, or if a new consumer-facing search engine emerges, this could be a major threat to Google's position in the market.
The reality is that the likelihood of a breakthrough AI algorithm is outside the scope of this essay. But as a Google investor, it is crucial to be aware of the risk of a monkey and to keep track of potential competitors in the search engine market. Just like Samsung, Micron, and SK Hynix all have incredibly advanced technology, but still struggle in the commodity memory market, Google must be prepared for the possibility of a disruptive new entrant in the search engine space.
Needless to say this was written by ChatGPT. I’m sorry for the awkwardness, but I choose to preserve it. I hope this piece can help to frame Google's disruption risk discussion. Next time, I am writing again.
