Have you at any point thought how Google really Works? When you type few words into Google search and 
the outcomes are there before you within a twinkle of an eye. That is astounding! Well maybe not.
Google has complex calculations which help bring this information within seconds. These calculations called Google algorithms aka “Google spiders” which drag codes net through your webpages and record your site content. This process goes on out of sight under the pages.
Google utilizes programming known as “Web-Crawlers” to find pages that are freely accessible. In most cases, web-crawlers or spiders are called Googlebots. This whole process involves what is called “crawling” and “indexing”. These two terms are sometimes misused interchangeably.
Crawling
In simple terms, Google crawlers visiting your webpages and following links on those pages; something similar to if you were to browse content over internet, but Google gets deep to the roots of each web page. Google crawler follows your pages from link to link and hence crawl your whole website.
As a website owner, you can somehow manipulate crawling process with a list of web addresses you want Google to index or to avoid, using robots.txt and sitemaps which you provide to Google. Therefore robots.txt and sitemaps are very important and play key role in SEO functionality. Using robots.txt you can include or exclude webpages, you want Google to crawl.
Crawler pays detail attention to websites, pages, modified pages and dead links. Hence make sure that your website does not contain broken links which can affect your website ranking.
Google Indexing
Google indexing means Google crawling your website and storing the relevant information in their servers or adding your webpages into Google search.
Google gathers information from pages in the crawl process and creates an index on their servers so they can look things up later. This is very much similar to an index on the back of a book.
Google will not crawl your page if the meta tag is set to No-index. Hence make sure you set the meta tag equal to index to include your page in search results.
Google indexing considers many aspects of the pages like:
- How many times a keyword is repeated in the page
- When the page was published
- Whether pages contain pictures and videos
Hence it is very important that a proper keyword research is done before any content is published, for better crawling and indexing of your webpages. Google indexes millions of web pages daily. There are many factors which may affect indexing and crawling of your site.
So there are proprietary some tools out there that one can use to crawl webpages to fully visualize who Google does voluntarily. E.g:
In this video, we have OnPage director explains further.
