In order to maximize your website exposure and to gain ranking from search engines your webpages must first recognized and indexed by search bots. All your webpages will get indexed in Google based on your robots.txt configuration, but some will be pushed to Google’s omitted results. Most of the site will have some bad URL’s