1

Let's say I have list of user agents in my robots.txt like

User-agent: Applebot
Disallow: something

User-agent: baiduspider
Disallow: something

User-agent: Bingbot
Disallow: something

User-agent: *
Disallow: something

Would bots reference User Agents in OR manner or AND manner? If Bingbot crawls this site, do Bingbot rules and * rules apply together?

Matt Raines
  • 4,149
  • 8
  • 31
  • 34
Jason
  • 1,298
  • 1
  • 16
  • 27

1 Answers1

2

According to Google, each User-agent line starts a new group of records and only one group of group-member records is valid for a particular crawler. The correct group is the one with the most specific user-agent.

So, Bingbot should only read the rules after User-agent: Bingbot, and not the rules after User-agent: *.

Matt Raines
  • 4,149
  • 8
  • 31
  • 34