This has been a frustration of mine for a while now with a few of my blogs. Several years ago, there was no issue at all – my sites always showed up in SERPs on all major search engines – Google, Bing, DuckDuckGo, Yahoo, Yandex, etc.
And this is still partially the case, but the Bing and therefore DuckDuckGo/Brave search engines (who effectively pull their results from Bing) have stopped showing some of my sites up in their SERPs.
Content that used to rank #1 is not even de-ranked, but just not showing at all in their search results, site-wide. It’s like they’ve de-indexed my entire site, having indexed it fine previously.
The content quality is not an issue, so I searched for other answers, including submitting sitemaps to Bing, which sometimes works but isn’t always the answer.
Let’s look at the steps you can take if you’re finding your site not being crawled/indexed/ranked at all by Bing and connected search engines.
Step #1 – Submit Sitemap To Bing Webmaster Tools
This is the first and most obvious step that it’s also advised to do for Google as well. Submitting a sitemap (basically a page mapping all the other pages on your website) to search engines has always been a useful way to “tell” search engines that a) your site exists; and b) here’s all the content on it, please index and display it in SERPs (search results).
In Google, you’re meant to do this via their Google Search Console tool, and the equivalent for Bing is Bing Webmaster Tools.
Step #1 – Create a sitemap if you don’t already have one. Many good themes will do this for you, or else use an SEO plugin like Yoast SEO or All In One SEO that will create an XML Sitemap for you. Create one and grab the link to it.
Step #2 – Open Up Bing Webmaster Tools, and sign in (you can do so using your existing Gmail account that’s attached to your Google Search Console for your site if you have one).
Step #3 – Follow the steps to submit a sitemap to their webmaster tools. You may need to verify ownership of your site, and they usually give you a couple of ways to do this.
Once you’ve done this, you can be confident that you’ve told Bing everything you need to from your side about your website, and all the pages that exist on it. From this point on, it’s up to them to crawl, index and rank the content if they choose to.
But from personal experience, sometimes even this doesn’t work, and your site still doesn’t show up in Bing Search (and also by extension DuckDuckGo and Brave Search). A lot of people are having this problem not just with brand new sites, but with established sites that used to be indexed and ranked fine by Bing, but no longer are.
It’s not clear why this is, but there’s another new tool that might be able to help.
Step #2 – Enable Cloudflare Crawler Hints Feature
If you run your site through Cloudflare DNS servers, there’s another new tool that can help get your site properly indexed, ranked and showing up in Bing and connected search engines.
It’s called the Crawler Hints feature from Cloudflare, and is still in beta mode, indicating it’s quite new. It’s designed to help tell search engines automatically whenever something changes on your website, so they can crawl and index it only when they need to, and not just for the sake of it.
Here’s how you enable this:
Step #1 – Log into the Cloudflare account associated with your website(s). If you don’t run your sites through Cloudflare DNS servers, it’s free to set up a basic account and change over if you want to (see here for a guide, but consult your host/ad network provider first to make sure you can do this without breaking anything).
Step #2 – Once inside your account, your connected email should appear in the top left corner right at the top of the left hand sidebar menu. Click on it (or on Overview) and your sites run on Cloudflare should display. Click on one particular domain to move into that site’s own settings (you know you’ve done this when the domain name appears at the top of the left hand sidebar menu).
Step #3 – Once inside a domain’s settings, scroll down to Caching on the left hand sidebar, then click Configuration:
Step #4 – In the Configuration page, scroll down to the Crawler Hints section, and make sure it’s toggled on:
Step #5 – Go back to your main Dashboard and repeat the process for any other sites you have on the account (you need to enable Crawler Hints one by one for each domain you have on Cloudflare).
This will basically send a signal to Bing, plus other search engines, whenever something is changed/updated on your website, prompting it to crawl and index the site again. It’s meant to be an environmentally friendly tool that allows search engines to more efficiently crawl sites only when something has actually changed, rather than repeatedly scanning websites that are identical to the last time they were scanned, and wasting resources doing so.
Video demo of enabling Cloudflare “Crawler Hints” (6:35 timestamp)
This is a nice new feature that’s really easy to turn on (less than 1 minute) and forget about it as long as you have sites that run their DNS through Cloudflare.
Whether this also fixes the problem of well established sites that used to rank fine on Bing/DuckDuckGo, suddenly not being ranked anymore, remains to be seen. I’ll be keeping an eye on this, having had this problem myself with several blogs. But it’s a quick, easy thing to turn on, so is worth doing for sure.
