Preview
The market for enterprise search technology has changed in key ways in recent years, particularly as more companies have entered the sector. These changes provide potential customers with new ways to implement technology that searches their networks for critical data. The changes also create a more complex array of choices, so understanding the state of the market and its future direction is critical for making the right choices about enterprise search technology. Click here for your free PDF of this report from ITI's Faulkner Information Services division.
Report Contents
- Executive Summary
- Market Dynamics
- Market Leaders
- Market Trends
- Strategic Planning Implications
- Web Links
Executive Summary
Enterprise search technology enables users to employ a single interface to perform natural language searches across multiple storage sources containing a variety of file types. The market is crowded and quickly changing. Whereas until recently it was dominated by smaller specialty players, large companies have used a mixture of acquisitions, partnerships, and internal development to gain strong positions in the sector.
The volume of data in most corporate networks, combined with the variety of ways the data is stored, means that no search tool will be able to effectively find all relevant data in a typical environment. Although in some respects this limitation makes it possible for specialized vendors to establish niches for themselves in the face of competitive pressures from much larger competitors, it also makes it difficult for the specialists to maintain meaningful technological advantages over new entrants. With these opposing difficulties creating pressure from two sides, many smaller vendors are creating revenue by forming partnerships through which large software providers (e.g., enterprise resource planning, enterprise content management vendors) integrate the smaller company's search technology into their own products.
Organizations that are evaluating enterprise search technology should bear in mind that no single product will likely be able to search all data effectively and provide results in exactly the manner desired. Therefore, it is important to prioritize which types of data are most important and then seek a product that handles that type well. The use of multiple search tools should also be considered.
Market Dynamics
Important corporate data was once largely confined to databases, which are structured and therefore facilitate search. Today, however, a great amount of corporate data is contained in word processing files, spreadsheets, e-mails, and other sources that don't lend themselves to searching. Some organizations attempt to make files easier to locate by tagging them with metadata. Metadata is information about a file that is not in the body of the file. For example, it could designate the subject into which the file should be categorized. Metadata is either produced automatically (for instance, the author's name would be automatically stored) or it is manually entered by a person. In either case, however, metadata cannot account for all the queries that a user could potentially make. Natural language queries are too diverse and unpredictable, and the labor involved with metadata tagging is too intensive to be performed for all corporate data.
Enterprise search technology aims to address this problem by enabling users to search across a network or a collection of networks to find files regardless of their type or their storage medium. Most products aim to enable users to search using natural language queries rather than Boolean searches, the syntax of which confuses some end users.
The goals of reducing the time employees spend looking for information and making better use of existing data for decision support functions have historically been key drivers of search technology. In addition to these broadly-held goals, the specific needs of two sectors have been particularly influential in spurring the growth of enterprise search in recent years:
US Government--The US Electronic Government Act of 2002 compels the public sector to increase the use of the Internet as a means of making information and services available to citizens. The need to electronically organize such data and services and make it easy for citizens to find has made the US government a prime user of search technologies.
E-Commerce--E-commerce companies that aim to let their customers easily search through large product catalogs are another major customer for enterprise search technology.
Market Leaders
The enterprise search market is densely populated, with entrants ranging from multi-billion dollar giants to small, specialty providers. The market is also in flux. Among the four vendors listed below as being the most dominant and influential, two--IBM and Google--have only recently become major players in the sector. IBM gained its spot largely through acquisitions, in particular its acquisition of Iphrase. Google attained its position by internal development, expanding its Web search technology into enterprise search products and services.
Autonomy -- Already one of the main players in the enterprise search technology market, Autonomy strengthened its market position in 2005 by acquiring Verity, itself a major vendor in the sector that expanded its portfolio in recent years by purchasing Inktomi. Autonomy reports that it has integrated its own enterprise platform, IDOL, with Verity's comparable K2 Enterprise. The integrated product is called IDOL K2. IDOL K2 categorizes data, provides collaboration functionality, offers data retrieval, and enables users to personalize search results. The company's IDOL Federator integrates with K2 to facilitate searching multiple sources, including searches using third-party technology.
FAST-- In addition to its FAST Enterprise Search Platform (ESP)--its flagship, general-purpose enterprise search application designed to search large networks for a variety of file types--the company offers two lines of specialty search solutions with more narrowly-defined purposes.
Data Search. The Data Search line of products, which are based on FAST (ESP), can be purchased and used separately or in conjunction with each other.
- Fast Data Search for Site Search--A search tool that customers can add to their Web sites.
- Fast Data for Search Compliance--Searches for customer-defined flags that indicate that a document might affect a company's compliance with prevailing regulations. It can be set to search documents in real-time as they are created or distributed.
- Fast Data Search for eCommerce--A search tool that customers can add to their e-commerce sites. It can adjust search results to place showcased products more prominently.
- Fast Impulse--Designed to let e-commerce companies better control how their products are presented to users who are browsing the site. It also helps control marketing techniques such as cross-selling and up-selling.
- Fast Data Search for Intranet--A tool for searching enterprise networks and external sources.
- Fast Data Search 360--A tool for enterprise searching, designed to be rapidly integrated into a corporate network.
Search Derivative Applications (SDA). FAST search derivative applications are custom-developed from FAST ESP technology. Others may be developed in the future.
- FAST Marketrac--A tool for finding information relevant to market intelligence, inside and outside an enterprise.
- FAST AdVisor--An application for electronic directories (e.g., online yellow pages, business-to-business verticals) that provides tools for users to find content and for advertisers to reach customers.
- FAST mSearch--Lets manufacturers of mobile devices give their customers a search tool for finding Web sites online via their mobile devices.
- FAST mSearch for the Enterprise--Enables mobile devices to search enterprise data sources as well as the mobile Web.
In addition, FAST also offers three publishing-oriented SDAs that have search capabilities.
Google -- Google has used its dominant position in the Web search market to quickly establish a strong foothold in the enterprise search sector. It offers dedicated search hardware as well as software that can be installed on desktop systems or servers. The company's enterprise search technology is closely based on its Web search technology, offering ranked results, summaries, and cached pages, among other features familiar to users of its online service. Although its products initially focused on HTTP documents, the company's products are now designed to search a broader range of document types.
Search Appliances. The company offers two product lines of rack-mounted search devices: the Google Search Appliance and the Google Mini. The basic model in the Search Appliance line is the GB-1001, which can search up to 1.5 million documents and handle 300 queries a minute. Multiple units can be clustered to create the GB-5005, which can handle up to 5 million documents a minute, or the GB-8008, which can handle up to 15 million. The Google Mini is a lower-cost rack-mounted device; the base model enables searching of 100,000 documents and can be upgraded to search up to 300,000.
The Mini starts at $2,995, and the GB-1001 starts at $100,000.
Search Software. Google's search software can be downloaded for free. The Google Desktop for Enterprise is designed to enable users to search a corporate network for files, e-mails, cached Web pages, and other types of files. (The company offers a support package for $20,000.) The Google Toolbar for Enterprise, which is currently in beta testing, adds extra buttons and features to Google's browser-based toolbar.
IBM -- IBM is investing heavily in enterprise search, both in terms of developing the technology and in building a product portfolio. The bulk of the development work is being done as part of an initiative called the Unstructured Information Management Architecture (UIMA).
The UIMA framework is the foundation for IBM's Websphere Information Integrator OmniFind Edition, which provides the ability to search intranets, the Web, databases, file systems, and other data repositories. It uses IUMA technology to determine the ranking and relevancy of search results. In late 2005, IBM acquired the business of iPhrase, whose products enable e-commerce companies to let Web users search electronic catalogs and other online information sources. The products gained in this acquisition are now part of IBM's WebSphere suite. For organizations that wish to develop software that has features not offered by OmniFind, IBM offers the UIMA Software Development Kit.
IBM views enterprise search as part of a larger information management program, which also includes technologies such as content and records management. In addition to its acquisition of iPhrase, IBM has acquired several other companies in recent years to advance its information management efforts. These include Ascential, DWL, and SRD in 2005 as well as Venetica and Alphablox in 2004.
Other Noteworthy Vendors
Despite recent merger and acquisition activity, the enterprise search market remains crowded. Small and mid-sized vendors such as Convera, Endeca, Entopia, and InQuira continue to collectively play an important role in shaping the market. Some of these smaller companies may become the acquisition targets of larger organizations that are looking to quickly enter the market or that wish to expand their current search application portfolio. (For perspective on the difference in size between certain vendors in the market, Convera had $25.7 million in revenues in 2005, and IBM had $96.2 billion, although only a fraction of IBM's revenues were from search technology.)
Large vendors whose specialty is not search are also having an impact by adding search technology to their existing products. For instance, Microsoft now has two search offerings: SharePoint Search Services, which is part of its SharePoint Services, and Windows Desktop Search, to which the company has recently added intranet searching, integration with SharePoint, and other features that make it more suitable for enterprise deployment. Hummingbird, Oracle, and SAP also now have search products. Although applications with search technology added as an enhancement may be, in most cases, much less powerful and feature-rich than the leading products on the market, many companies, particularly small to mid-sized ones, will find such offerings suitable, especially when the products are freely offered with other software.
Market Trends
A few years ago, the sector was dominated by small- to mid-sized companies that were primarily focused on enterprise search. In recent years, however, large companies have used a mixture of acquisitions and internal development to quickly and aggressively gain a strong footing in the market. Boutique vendors do not appear to have technological advantages that are strong enough to compensate for their lack of name recognition and marketing power, if they have any technological advantages at all. Consequently, newer entrants into the market such as IBM, Google, and Microsoft are poised to increase their positions within the sector, even if their offerings lack some of the functionality of their competitors.
Although boutique vendors may loss ground, the market will likely still have room for them, at least for the next several years. Enterprise searching still works with only moderate effectiveness, and the continued growth of stored electronic information--particularly audio, video, and image files--makes the task even more difficult. Organizations have varying needs in terms of the types of information for which they must search, the total amount of data to be searched, and where the data is stored. Consequently, there is ample room in the market for niche players. No search product will meet the needs of all customers, or even a large number of customers. Individual products will find success in certain vertical markets, or they will establish themselves as the best tool for certain types of searching.
In addition to several acquisitions, there has also been a significant amount of technology consolidation through partnerships. Many developers of content management, customer relationship management, and other types of enterprise software are integrating third-party search technology into their own products. For instance, portlets from Autonomy integrate with IBM's WebSphere Portal Software. Likewise, EMC's Documentum, one of the leading content management products, can be integrated with search technology from companies such as Convera, Fast Search & Transfer, and Google. Furthermore, Fast and Siebel formed an agreement in 2005 for Fast InStream to be integrated into Siebel's core business applications. Search capabilities can therefore be installed by purchasing a dedicated search product or by implementing another enterprise application that includes searching functions. These types of partnerships offer specialty vendors another opportunity to solidify their place in the market and find additional revenue streams.
It is uncertain whether search products will have more success in the coming years as a standalone technology or as technology that is integrated into other applications. The trend of large software companies adding search capabilities to their products suggests that the integrated approach will become more popular, although in many cases the search technology that is integrated into other applications may be developed by smaller, third-party companies, not developed in-house by the large vendors.
Currently, one of the key trends in Web search technology is verticalization, which refers to search services that focus on a certain topic (e.g., financial services, travel). The push toward verticalization is driven by the need to improve results: searches across the entire Web produce results that are not focused enough for many purposes. Although there is still considerable emphasis on comprehensive, multi-purpose enterprise search applications, there are some vertical offerings, such as FAST's Data Search products. Given the continued difficulty enterprise search technology has in capturing all relevant data and returning results in an effective way, there is a good chance that more verticallyfocused enterprise search applications will be developed.
Strategic Planning Implications
Organizations that are evaluating enterprise search offerings have many choices to consider. Enterprise search is available as software, hardware, or a service. Also, it can be standalone or part of a broader offering. Further complicating the evaluation process, enterprise search can be highly specialized, so a particular search solution may not be capable of sorting through certain types of files or it may not be able to index and present results in the way a certain customer requires.
With these difficulties in mind, prospective customers would be prudent to begin the selection process by identifying what information sources they wish users to be able to search. Although any organization would want all information to be searchable, this is an unrealistic expectation in most cases. The currently available search technology is limited in terms of its ability to effectively cover every source, and the cost of search technology is high. Therefore, the list should be prioritized to target sources that must be searched and leave as optional sources that might be less critical.
Although many search products are intended to be comprehensive, potential customers should remain open to the possibility of using more than one solution to meet their search needs. Although a multivendor approach works against the goal of having a single search interface, and although it adds to expenses, it may be the best option in some cases. The technologies might overlap in some areas, but they may also compensate for each other's deficiencies.
In addition, some search needs are specialized and are best met with specialized tools. For instance, some organizations may be primarily interested in searching through large text documents, such as manuals; for these organizations, a product such as TeraText Solution's TeraText Database System, which is designed specifically for text searching, should be strongly considered. The overall effectiveness of a particular product matters less than the product's ability to meet the specific needs of an individual organization.
A search application's utility is determined not only by the quality of its algorithms but also by its user interface. Therefore, an organization should use a trial implementation to test a product's ability to search the targeted storage sources and return useful results. End users will be the best judges of whether search results are useful, so a handful should be included in the trial implementation.
Enterprise searching is difficult in large part because corporate data is stored in so many sources, in such an unstructured way. Rather than relying on enterprise search technology to alone address this problem, a complimentary approach is to change the way that data is created and stored. Enterprise content management applications aim to manage data from its inception. Although likely there will still be some unstructured data despite the use of content management technology, content management can minimize such data and improve an organization's ability to effectively find key information. There have been some signs that enterprise content management and enterprise search are merging to an extent. For instance, EMC, a content management leader, now includes search capabilities as part of its EMC Documentum content integration offering. The search capabilities include EMC's own technology as well as technology from Google. Also, FAST offers its search technology to providers of content management application and other independent software vendors through a platform called InStream. Partnerships in which enterprise search vendors provide their technology to large enterprise software vendors will become more common in the coming years, providing potential customers with another option for implementing enterprise search.
About the Author
Geoff Keston is a project manager for a leading technology consulting and services company. In this role, he has been responsible for the successful completion of enterprise software implementations, network upgrades, and telephony implementations for major retailers, financial firms, and public institutions. Geoff also writes extensively on issues relating to software, data networking, and ecommerce, as well as on the cultural, economic, and political issues raised by technology. He is a Microsoft Certified Systems Engineer and a Certified Novell Administrator.
Web Links
Click here for your free PDF.