The five steps I have identified for selecting a search engine will help you avoid undertaking a technology project that can’t be successful. Some of the steps can be strenuous for both you and your search vendors. Depending on obstacles in your organization, you might not manage to complete all the steps. But your selection process will be greatly enhanced by considering all—even if only completing some—of the steps.
I’m assuming you have laid the groundwork for technology selection: You have a clear vision and the support of key stakeholders. (If you don’t, check out my advice at www.psgroup.com/detail.aspx?ID=837.) Your vision defines search success metrics to be achieved, and the metrics establish the value of achieving the vision. The five steps in successful search engine selection are as follows:
- Inspect scenarios for technical requirements
- Assess content
- Estimate limits
- Plan for special cases
- Perform comprehensive technical evaluation and planning
Step 1. Inspect Scenarios for Technical Requirements
The right way to identify requirements is to inspect your audiences and their key scenarios, using real-life situations, questions, and answers. A customer looking for guidance on using a product might be best answered by a video another user posted to YouTube. The answer another customer needs might be explained in a 30-second clip buried deep in a 2-hour webinar. Your distributors need to embed your search results into their applications. Does your requirements list support those scenarios?
In determining the technical requirements for your search engine, it’s critical that you consider a broad set of audiences and use cases, both current and future. Think big during requirements, so you’ll select a product that will fit your needs for some years. (Think small during implementation, so your projects can attain quick success.)
Consider all the audiences your information retrieval service will serve and list their key scenarios or use cases in business terms. We’ll use this list again in Step 5. Make sure you include not only seekers of all types, but also your search managers, people who manage or contribute content, business managers, and IT architects and developers. All of these people have specific needs. People responsible for managing the search experience need reports, tools, and technology to support their efforts.
(For a detailed list of stakeholders, scenarios, and requirements, check out www.psgroup.com/detail.aspx?id=862.)
Step 2. Assess Content
I believe the most common cause of poor search experiences,and therefore the most common reason for replacing search technology, is theorganization’s failure to consider how content impacts findability. You mustassess content to determine whether it covers all topics adequately and if itis properly tagged for findability.
There are a couple of quick tests you can perform: Downloada free trial search engine or use your current search engine to test some ofthe scenarios you identified in Step 1. To what degree are poor results due topoor content? To state the obvious, if the content doesn’t exist, the searchengine can’t retrieve it. To state the not-soobvious, if the content has poor(or no) titles and headings (or is badly written), a search engine may find itbut rank it very low in the results … and the user will never find it. If itdoesn’t include a date, the search engine may rank 6-year-old content ahead ofcurrent information.
If your content is in poor shape, you must acquire toolsthat will automatically analyze and tag it for findability, and report oncoverage of what searchers are asking for. Quite a few search offerings includetools that perform these tasks, and there are also stand-alone tools andservices. Whatever sourcing you chose, plan to assign resource to manage theinitial and ongoing efforts of ensuring your search engine can present theright content. If you can’t fix poor content, there is little point isproceeding with better search technology.
Step 3. Plan for Special Cases
Special cases in search are the information, audiences, or scenarios that aren’t readily handled by your technology or processes.The most common examples today are complex requests from researchers and scientists, experience and expertise search, and audio search. Failing to plan for special cases is, sadly, the de facto standard in dealing with these thorny problems. But not handling them creates pockets of justifiable resentment in the organization, and erodes the benefit you should be receiving from your search technology investment.
Devise a strategy for dealing with each of them, including a high-level resource plan and timeline for implementation. If the strategy is not acceptable (too costly or too lengthy), then you are not taking an approach to search that is going to meet your organization’s requirements. Don’t just assume the situation is not practical. Rethink your approach and look for a solution that leaves the door open for everyone.
Step 4. Estimate Limits
Overestimating the organization’s patience, budget, and commitment is a sure path to trouble. As you begin the selection process, make your best estimate of the following:
- Maximum time to pay back investment
- Maximum time to achieve noticeable results
- Amount of resources each business unit will commit during the implementation project and going forward
- Degree of success that must be achieved (e.g., Search SuccessPercent or Conversion Percent)
- Effort required to get content in adequate shape
- Effort required to maintain content in adequate shape
This information will help you evaluate the solutions and prioritize your requirements. And if the effort required clearly transcends the investments you think are committed, then it’s time for a heart-to-heart talk with your sponsor. There may be some pressure to back off on your estimates of initial and ongoing effort. Before you bow to that pressure, remember that failure to invest properly in content and special cases is probably the root cause of your users’ dissatisfaction with the current search technology.
Step 5. Technical Evaluation and Planning
A stringent proof of concept (POC) test is the final hurdle to successful selection. It is critical that the POC test be comprehensive and difficult. Devise tests using searches that don’t work today, that touch on your special cases, and that address all the key scenarios you identified in Step 1. Use test automation tools to test dozens or even hundreds of queries. Have knowledgeable people identify the two best answers for each query. The single most important criteria is this: How many of the right answers appear in the top five results?
If your requirements include content improvement, such as metadata extraction and classification, you should test these capabilities in your POC as well.
Part of your evaluation should also be the implementation plan. It will no doubt start with a pilot, typically delivering big results in a short time, perhaps 30 days. Too many implementations never get farther than the pilot. You need commitment and plans that carry through your entire set of requirements (including those special cases).
Following these five steps, or even just using some of them, will greatly increase the success of your search selection, and of the implementation project that follows. One final word of advice: Keep your vision for your search project at the front of your thoughts, and make sure that every decision will move you closer to, or at least no farther from, the ultimate goals of the investment.
About the Author
SUSAN E. ALDRICH is a senior vice president and senior consultant at the Patricia Seybold Group. Aldrich manages the Search, Navigation, and Discovery Research Practice, with a personal research focus on customer self-service, information management, and technologies and practices for monitoring, measuring, and managing the Quality of Customer Experience(QCE).