I want to hide a folder named ( beta ) in the public_html from the search engines also all it's subfolders and files, do i have to put the file in the root folder ( / ) and do the content of the robots.txt like the following
User-agent: *
Disallow: /public_html/beta
OR
put robots.txt in the public_html and do the content as the following
User-agent: *
Disallow: /beta
The robots.txt must be in the root.
www.yoursite.com/robots.txt
This is what you need:
User-agent: *
Disallow: /folder/something
With something being the name of the folder you want to disallow
Also, note that you only want to block the subfolder named something. If you also have a file named something that you don't want to disallow, than do this:
User-agent: *
Disallow: /folder/something/
What matters is the public URL, not the actual physical folder of your server.
So when you have a site at example.com/cooking/index.html, your robots.txt needs to be at example.com/robots.txt. For sports.example.com/football/blog it would be sports.example.com/robots.txt.
The best solution is to hide your all sub-folders and files from beta. You can define robots.txt as follows:
User-agent: *
Disallow: /beta*/
Now, all search engines will not crawl all sub-directories and files followed by beta.
