Stack Overflow
Stack Overflow

Questions tagged [bingbot]

Bingbot is a web-crawling robot, deployed by Microsoft to supply Bing. It collects documents from the web to build a searchable index.

23 questions
16
votes
3 answers

How do I prevent Bing from swamping my site with traffic irregularly?

Bingbot will hit my site pretty hard for a couple of hours each day, and will be extremely light for the rest of the time. I'd either like to smooth out its crawls, reduce its rate limit, or block it altogether. It doesn't really send through any…
Tim Haines
  • 1,496
  • 3
  • 14
  • 16
4
votes
1 answer

Do Yahoo and Bing crawlers interpret JavaScript the way Google does?

I have an Angular single page app. Google started rendering JavaScript when crawling pages in mid-2014, but do Yahoo and Bing also interpret JavaScript when they crawl pages? Do I need to include them in my Prerender.io configuration?
Seamus James
  • 981
  • 3
  • 12
  • 25
3
votes
1 answer

Bingbot attempting to access non existent pages

I recently made some major changes to an ecommerce website that include url structure. The url to view a product is modified by .htaccess and contains a short product description that if changed will not affect the results on the page. example:…
weewa
  • 125
  • 3
  • 4
  • 14
3
votes
0 answers

Bingbot causing 404 errors

We have recently been seeing a large number of 404 errors that are being created from the Bing web crawler. I have verified that the IP is in fact a Bing machine but just don't know why they are attempting the URL's they are trying. I don't want to…
Jason
  • 17,276
  • 23
  • 73
  • 114
2
votes
1 answer

TYPO3: Bingbot creates an ext_form error which get cached

We have a problem whit one of our TYPO3 installations. The Bingbot which visits the site calls a controller of the old ext_form extension without parameters and creates an error. 207.46.13.XXX - - [16/Oct/2018:00:18:48 +0200] "GET…
Markus Gehrig
  • 45
  • 3
2
votes
1 answer

ActionController::InvalidCrossOriginRequest exception due to bingbots

I have rails applications where I am loading comments using Ajax after page load. class CommentsController < ApplicationController respond_to :js def index @comments = Comments.all respond_with @comments end end It is working as…
Rahul Chaudhari
  • 2,230
  • 21
  • 29
2
votes
2 answers

How do I redirect nonexistent pages to the 404 error page with .htaccess?

Apparently Bingbot is getting caught in an infinite loop on my site. It downloads pages like…
zylstra
  • 740
  • 1
  • 8
  • 22
1
vote
0 answers

BIngbot on my network IP

I was looking my apache logs and I find out this. Log Line: 192.168.1.2 - - [30/Nov/2016:15:46:52 +0100] "GET http://www.Mywebsite.... HTTP/1.1" 200 5539 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" This happen a few…
PenDragon
  • 13
  • 4
1
vote
1 answer

I want to only shown my web site to googlebot, yandex or bing. How can i set?

If I set to only Google Bot; I can this setting with this code: if(!strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "googlebot")) { if($_SERVER["HTTP_CF_IPCOUNTRY"] =! 'USA') { echo "empty"; die(); } } But I want to add…
Sinan Aydın
  • 69
  • 8
0
votes
0 answers

Why is Bingbot unable to crawl my wordpress website even after implementing extensive troubleshooting measures?

I am facing an issue with Bingbot not crawling my website effectively. Despite taking several measures, including disabling the iThemes and Wordfence plugins, turning off Cloudflare, and resubmitting the sitemap to Bing, I am still encountering…
Behrooz shy
  • 76
  • 5
0
votes
1 answer

is it possible to remove a bingbot from accessing my website?

I appear to have a bingbot accessing a couple of my pages on my website that send out emails to employees. It will access the page at random times and it triggers the email being sent out over and over again. Most of the pages on the website check…
bmacx7
  • 1
  • 3
0
votes
0 answers

Google Cloud Function injecting 'Cache-Control: Private' header in response to Bingbot User Agent independent of function code

We have a Google Cloud Function in live, which is essentially returning the correct redirects for us from a now-defunct site using a very simple Python script, backed by a CDN which is caching the responses to avoid triggering the function more than…
TorTurner
  • 11
  • 4
0
votes
1 answer

Block search engines from indexing local search results but not search page

Bingbot keeps indexing search engine result pages so I want to: Allow search engines to access everything in general. Allow search engines to index the search/ url. Disallow only search queries (search/?q=example) without blocking the search/ URL…
John
  • 1
  • 13
  • 98
  • 177
0
votes
0 answers

How include robots.txt in IIS 6 to exclude Bing scan my page?

I have a ASP.NET aplication, using Framework 2.0. It's already publish on Internet. It's deploy in IIS 6 on Windows 2003 Server. How I can exclude the Bing browser scan my page? I use Sensitive data from users, and I can't expose to internet. How I…
Hernaldo Gonzalez
  • 1,977
  • 1
  • 21
  • 32
0
votes
1 answer

Strange nginx behavior specific for bingbot requests

we have found something strange in nginx logs. Some requests only from bingbot goes directly to backend, but if we fire this requests from curl, they processed as usually. Please take a look of configuration. I've got two servers, one (server1) if…
Daniel
  • 534
  • 4
  • 16
1
2