First of all, itâ€™s not a necessity for every website to have a robots.txt file. This file exists to tell the search engines you want them to limit their access to specific pages or directories of a website, so without one, the entire site would conceivably be indexed. It can address either all or just specific bots.
Depending upon the site, of course, allowing the search engine to index any and all pages could present a problem. For instance, if an ecommerce site with on-site search function doesnâ€™t limit access at all, then every single search would result in a new URL being created â€“ which could appear in Googleâ€™s search results (as well as any other search engineâ€™s SERPs). This can easily result in a site with only 400-ish pages having tens of thousands of pages