Botify partnership with Microsoft brings real-time content indexing to Bing

Best suited for very large sites with lots of content updates. Adrien Menard co-founder & CEO Botify said by phone that there’s no money changing hands between Bing and Botify in this arrangement. Bing is testing the API with a few large partners, although Menard says this relationship is unique to his knowledge.

Menard also said that the approach is best suited for websites larger than 10,000 pages and those that publish new content on a regular basis, such as large retail and e-commerce sites, classifieds sites and marketplaces, among other large content publishers. Botify’s customers include Macy’s, Expedia, Condé Nast, eBay, Monster and others.

Immediate indexing. The benefit for site owners is immediate indexing, more control over content and potentially better coverage, while Bing benefits from lower costs (vs. crawling) and more recent and, presumably, higher quality content. Menard observed that it can take a couple of days for new content to be indexed by search engines. He also said they often fail to index “more than half of a typical enterprise website’s pages.”

“Search engines don’t have resources to crawl every page of the internet, and websites are changing all the time,” said Menard. With the API, “You can select the pages to be indexed.” Botfiy also analyzes its customers’ sites and content and makes recommendations about which pages should and should not be submitted.

Why we care. Google also offers an indexing API for a more limited number of content types: “websites with many short-lived pages,” such as video and job postings. The Bing API will accept core site content by comparison.

Depending on how successful and well received Bing’s indexing API pilot turns out to be Google may decide to follow suit and open a wider range of content to its API. It would be an overreach to call this the future of search indexing but we may see APIs much more widely used in the future.