There are a lot of specific ways you can prospect: on LinkedIn, using a sales intelligence database, or even asking around for recommendations. But when you boil it all down, the concept of prospecting is pretty simple: it’s all about searching. We asked Sigalit Sadeh, Lusha’s Director of Data Products, to take us on a […]
There are a lot of specific ways you can prospect: on LinkedIn, using a sales intelligence database, or even asking around for recommendations. But when you boil it all down, the concept of prospecting is pretty simple: it’s all about searching.
We asked Sigalit Sadeh, Lusha’s Director of Data Products, to take us on a journey through the evolution of search functions: how they work, how results are organized, and how trends for search engines like Google, Bing, and enterprise search solutions affect the world of prospecting software. Sigalit has worked at companies like Dropbox and Hewlett Packard on projects that included Machine Learning, search, and data content organization. So if there’s anyone to geek out with over all things related to data, search functionality, and AI, it’s her.
How Search Engines Work
Search engines not only retrieve information but also rank it for relevance to the user input. Broadly speaking, their process includes 3 main components:
- Data Collection: Search engines use various methods to gather content, including crawlers, APIs, large databases or repositories. This creates a comprehensive, interconnected repository of information.
- Indexing: Once information is collected, it needs to be organized and stored. Parsers extract relevant data from the collected links and send it for indexing into the search engine’s index. This index functions like a massive digital library with cataloged information, making it easy to retrieve.
- Retrieval and Ranking: When a user searches for information, the search engine matches the query to data in its index and returns pages that are a match. Since not all pages are equally relevant, the search engine ranks them on the search engine results pages (SERPs) based on their potential utility to the user. Intricate algorithms evaluate content quickly to ensure users find the most relevant results to their queries within milliseconds.
Types of search
Before we dive into the deep end of search, let’s set the groundwork with some search basics. Generally speaking, there are a few different types of search: textual, voice-based, visual, and conversational. This is painting with broad strokes, of course. We’ll get into specific types of text-based search in a bit, but first we’ll briefly define the basic search types:
- Textual search: Originally based on keyword matching, textual search forms the foundation of search technology. It has evolved with generative AI, transforming from simple keywords to conversational interactions and semantic understanding. This evolution parallels other search types: voice search converts speech to text using NLP, and visual search uses metadata extraction to turn images into text-based queries. These advancements bridge the gap between computers and human minds, promising more intelligent and intuitive search experiences.
- Voice-based search: With the rise of smart speakers and voice assistants, voice search has gained popularity. Nowadays, over 1 billion voice searches take place every month. Sigalit explains how it works: “It requires advanced Natural Language Processing (NLP) to interpret spoken queries, which are often more conversational and context-dependent than text-based searches.”
- Visual search: allows users to search using image instead of text. For example, you can take a picture and use Google Lens, Pinterest, or another search tool to find similar things (or even exactly what you’re looking for). “This type of search is particularly useful in e-commerce and design industries, where finding similar items or visual inspiration can be really effective.”
- Conversational search: AI-generative search involves conversational and question-answering capabilities, such as those provided by models like ChatGPT or Gemini (previously Bard). It leverages advanced AI to understand and generate human-like responses, allowing for more interactive and dynamic search experiences. As we dive into 2024, AI and machine learning are reshaping search engines in exciting ways. These cutting-edge technologies are making search engines smarter, helping them grasp the true meaning behind what we’re looking for.
From keywords to conversations: The evolution of textual search
Textual search is a broad category that includes different levels of understanding search intent. Over the years, the standard text-based search has evolved from simple keywords all the way to sophisticated, AI-driven conversational interfaces. Let’s dive into the evolution of textual search.
Keyword search: to Boolean and beyond
As the initial phase of textual search, “keyword search is the simplest instance of text-based search,” according to Sigalit. “Users input specific words, and search engines retrieved documents that contained those exact terms. It’s a foundational method, but it was limited by the search engine’s lack of contextual understanding and the user’s ability to guess the exact right keywords.
“The evolution of that was the Boolean search. To enhance keyword search, Boolean operators (AND, OR, NOT) were introduced. This allowed users to combine keywords into more complex queries that produced more refined, accurate results. It added a layer of precision and control, but still relied on users’ ability to formulate the right queries to be effective.”
Another evolution within the realm of keyword search was fuzzy search. Here’s how Sigalit describes it:
“Fuzzy search finds results that are close to what you searched for. It accounts for typographical errors, misspellings, and approximate words. This technique increased the search engine’s tolerance for inexact matches so that users could find relevant information even with imperfect inputs. It enhanced the robustness of search engines while also making them more user-friendly, allowing for a more forgiving and accessible search experience.”
There’s a downside to fuzzy search, says Sigalit:
“Sometimes fuzzy search can bring back inaccurate results.” But it’s still worth it: “it’s a good methodology to prevent users from seeing zero search results, which is often a disappointing and inaccurate experience. And there are features like ‘did you mean…’(when you type one thing and the search engine suggests a different query) which help improve the accuracy of the fuzzy search results.”
The move to Natural Language Processing
The introduction to Natural Language Processing (NLP) “marked a significant shift,” says Sigalit. It’s one of the major evolutions of text-based search.
“NLP enabled search engines to understand and process natural language queries. They went beyond exact keyword matches to grasp the meaning and context of words. This allowed users to search with more conversational language, which made it easier to find relevant information without having to craft precise, keyword-based queries.”
Getting into semantics
Semantic search is an important advancement in text-based search that builds on NLP.
“Semantic search revolutionized how search engines operate by aiming to understand what you’re looking for based on your intent,” says Sigalit. “The engine knows the meaning of your search beyond the words you type.”
How does it work? “Semantic search harnesses the power of NLP, machine learning (ML), and knowledge graphs to comprehend the relationships between words and concepts and provide more contextually relevant results. This shift from syntactic to semantic understanding has bridged the gap between human cognition and machine understanding so that search engines could better interpret user intent and deliver more accurate answers.”
Search on a personal level
As text-based search technology continued to advance, another key feature emerged: personalization. “Search engines started using user data – things like search history, location, and preferences – to start tailoring the results to the user,” says Sigalit. “This search made each search much more relevant to individual users and enhanced their overall search experiences. Personalized search delivers results that match the user’s unique context and needs, which makes it more useful on an individual level.”
The rise of AI in search
The most recent – and likely most significant– evolution in textual search is driven by generative AI.
Here’s how Sigalit explains it:
“Models like ChatGPT allow for a more conversational search experience.Users can engage in natural, human-like dialogues with search engines. These AI systems understand context, generate detailed responses, and can even create new content based on user queries. This leap transforms textual search from static keyword matching to dynamic, interactive conversations, making search more intuitive and responsive to user needs.”
The power of semantic search
Semantic search can be very powerful when it comes to delivering users with the answers they’re really looking for. How does it all work? Sigalit explains:
“Semantic search within textual search has the ability to understand intent and context using various signals.
“Google, for example, is renowned for leveraging extensive data about your search behavior, including your location and popular searches in your area. At a basic level, Google uses your location to provide contextual relevance. It also analyzes your interaction data and search history, including your activity – the links you click, the time you spent on pages, an the content you frequently search for – to build a detailed profile of your interests and habits. Using this profile, Google can better predict and understand your search intent to deliver more accurate and personalized search results.”
“Unlike search within a closed database, Google searches the open web and uses a more comprehensive set of signals. But there will still be similarities between Google and how other softwares may begin to implement semantic search.” While they don’t all operate the same, here are some of the ways that semantic search has transformed aspects of search functionality.
- Entity-based semantic search: “This approach identifies and understands entities like people, places, and organizations within search queries and content,” says Sigalit. “By evaluating the relationships and attributes associated with these entities, search engines can provide more precise and relevant results.”
- Contextual semantic search: “Taking the context of a search query into account with factors like the user’s location, time, and preferences, this type of search delivers personalized results tailored to the user’s unique situation.”
- Intent-based semantic search: “Going beyond the literal interpretation of keywords, this form of search aims to understand and fulfill the user’s underlying intent. For example, when a user searches for ‘best restaurants in New York,’ the search engine interprets this as a request for restaurant recommendations in New York City and responds accordingly.”
- Relationship-based semantic search: “This approach provides comprehensive, interconnected search results by drawing connections between different pieces of information. It allows users to discover more relevant and related information based on these relationships.”
- Semantic question answering: “This is an advanced form of search that involves interpreting questions (worded in natural language) to provide direct, concise answers. Instead of just listing relevant web pages, it also leverages NLP and knowledge graphs to extract accurate answers from trusted sources.”
Why am I getting these search results?
So now we know how search works, but what goes into displaying interesting and effective search results?
Sigalit, again, uses Google as an example: “When you search on Google, you’ll notice two or three lines of text below each result that explain its relevance. This snippet provides a clear correlation between your search query and the result, so you don’t have to guess why it appears in a particular position. Additionally, Google enhances the experience with a highlight feature that emphasizes the exact parts of the content matching your query. This combination of informative snippets and highlighted content ensures that users receive clear, contextually relevant, and easily understandable search results.”
How does this relate to other search functions, like those within a software platform?
“Those results will look a little different, but you still need the results page to highlight why these specific results are relevant to the search query you just entered.
“Any platform with a database will do many different things to optimize the order in which your search results appear. It’s a well-known industry practice called ‘ranking.’ There are various considerations for ranking and the precise way any given ranking machine works is a type of secret sauce for the company. Depending on where you search, where the user is in the search flow, and the context, the ranking should be flexible and accurate enough to seem like magic, ensuring that what you need appears at the top of the search results.”
What does this ranking process look like for Sales Intelligence platforms like Lusha, which have to display the most relevant possible prospecting info to users?
“It’s an ongoing innovation and optimization play at Lusha, too. It all connects back to our Demand-Based Data machine. We rely on behavioral and engagement signals to determine which data to enhance in our database. Similarly, we use these insights to develop and refine our search logic.”
The future of prospecting search in the Sales Intelligence industry
We’ve talked a lot about search in general, and a bit about how that applies to searches within software platforms and commercial databases. What’s the direction of search specifically within Sales Intelligence platforms that help salespeople (and others) prospect?
Sigalit has a few ideas: “The next steps of that semantic search are personalized semantic capabilities, NLP, and AI in search. The industry is headed toward prioritizing intent over keywords and really understanding what the user is looking for beyond what they actually typed. Another development could be personalized ranking that organizes data to meet user needs, prioritizing showing what’s most relevant to each user first. These search results could also provide insights, snippets, and highlights that help users understand why they got the specific results. Essentially, having really, really relevant results for every user is the next frontier within the industry.”
Good search results come from good data
No matter what format a search engine uses, there is one universal truth – it can only be as useful as the information it gives you.
A lot of it comes down to the quality of the data in the database. Google couldn’t give you relevant results if there was nothing related to your search on the internet. It’s the same for any search function. No matter how the search is built, it needs a solid database to provide you with solid results.
Good data is what Sigalit and her team are all about:
“My team at Lusha builds out Lusha’s database and formulates and implements our data methodology. We also support the search and ranking functions within the Lusha Prospecting team. It’s all related.”
So what makes a “good database” that supports effective search?
“Everything begins with your infrastructure,” says Sigalit. “Depending on your infrastructure, you can build a really good index to host your data and allow specific complex searches. If your infrastructure allows it, it will understand semantic layers and give you the ability to control your ranking machines, ultimately optimizing the search experience.”
Our partner for this journey is Elastic. “The core of Elasticsearch’s indexing and search technology is based on the Lucene library. It organizes the data in a way that allows for highly specific queries, enabling fast responses and personalized results to complex queries. Elasticsearch is definitely an industry-leading infrastructure in the search space.”
Building a sophisticated data infrastructure to support user-friendly search is something Lusha constantly works on.
“We’re trying to provide the best prospecting search experience to users and help them get what they need really fast, prioritizing the top results. The ultimate goal is to create personalized data experience that also has a high level of explainability.
We are building towards that. We started with revising the foundation of our database and building out its ability to understand the relationships between different pieces of data. The way data connects is very important to our ability to give the most meaningful results to our users.
The intelligent (AI-powered) database is like a brain, and it can help us understand the really complex structures of companies and contacts to make search results more meaningful. Once you understand the scope and context of not just the user but also the data, you can give the most accurate results.”
See for yourself
Now that we’ve talked all about prospecting search in Sales Intelligence, you can try it out with a whole new perspective. Want to see what Sigalit and her team have been working on? Sign up for Lusha for free to get relevant and accurate prospecting information.
We'll help you understand if Lusha can solve your business needs.
