A search engine like Google is like a huge library with a very fast staff. Before it can hand someone a page that answers their question, it always does three jobs, in the same order.
1. Crawl
Think of a program called a crawler (also called a "bot") as a library assistant walking every aisle. It follows links from page to page, and it also reads a list of pages you want it to visit, called a sitemap.xml file.
A file called robots.txt works like a small sign on a door: it tells the crawler which rooms it may enter. But putting up that sign does not, by itself, hide a page from search results — it only stops the assistant from walking in.
2. Index
After the crawler visits a page, the search engine reads it and writes it down on an enormous list called the index — like a library's master catalog. Only pages on this list can ever be handed to someone searching.
A page can also ask to be left off this list with a rule called noindex. That is different from robots.txt: noindex removes a page from the catalog, while robots.txt only stops the assistant from visiting the room in the first place.
3. Rank
When someone asks a question at the front desk (types a search), the search engine looks through every page on its list and decides which ones answer that question best, and in what order to hand them over. That order is the "rank." Many things affect rank, and the exact rules are not public.
Why this matters
These three steps happen in a strict order, like dominoes: a page that skips the first step cannot reach the later ones.