Build triggers and Site Search are now two different entities
We've just rolled major changes to the Site Search feature: settings for how to index websites now have a dedicated section in the Project settings, while build triggers have become leaner.
What's new
Until today, Site Search was buried deep within build triggers and tightly integrated with the build system: you could crawl a website only if you had a build trigger.
That wasn't always what you needed. Maybe on a non-static website, there's no build phase at all, but you rightly want to leverage the Site Search functions.
So we improved it to bring about a few major changes:
Site Search is now independent of Build Triggers being used. to benefit from Site Search, you simply have to create a Search Index, and you're good to go. No more artificial coupling.
Site Search logs split apart from Build Triggers logs. Build triggers and indexing logs are now two different sections in your Project settings.
Independent control over indexing. You can control when and how your sites are indexed for search purposes independently of their build process (even if you don't have one).
ℹ️ If you had Site Search enabled, your setup has been migrated to the new configuration: for each build trigger with indexing enabled, we created a search index and linked it to the build trigger.
How does it work?
Navigate to the Search Indexes section in Project settings to create or manage your search indexes:
Why the change?
Decoupling search indexes from build triggers gives customers explicit control over when indexing happens and provides independent failure reporting and logging for indexing.
Plus, it opens up new possibilities: you can now add a custom suffix to the user-agent we use when crawling websites. With that, you can work on your robots.txt and sitemaps to get different search indexes for various sections of your website (for example, one for the blog, another for the documentation, and another for the FAQs).