Hello guys, I’m Akash. And welcome back to CybercityHelp once again. So in this article, we are going to talk about our very first module of SEO, that is known as how search engine works. So let’s start with our first SEO module that is, how search engine works.
In our first module, we are going to discuss about what is search engine, how search engine works, what is crawling, how crawlers work, it’s some policies, what is indexing, what is crawl budget and many more concepts. So let’s get started.
What is Search Engine?
Alright, so a search engine is a software program that helps people find the information they are looking for online using keyword or the phrases. For example, search engines are just like your google, bing, yahoo etc.
You can use these search engines to figure out any particular information you want to get. Also, these search engines can store information about billions of the web pages in an organized format, basically that is what we call as search engine.
How Search Engine Works?
Now we have to know how the search engine works, so let’s say when you search any particular keyword or query on google, so lot of activities start happening in the backend side.
For example, there are some components of search engine, first one is your query engine which proceeds your query to your crawler, and then crawler proceed your query to your search engine database just to figure out that information, and then the crawler which can read every single information from your website process the information to google search engine database for indexation, means your information will be stored in google’s database plus it will be available in Google forever unless you request for removal.
What is Crawling?
Okay now you might be thinking, why I am using crawling word a lot, so let me first discuss what is this crawling here. So crawling basically means a virtual robot which can access and read every content, every information of your website. And what it does? It stores your all content on the web pages and process the information in the search engine’s database.
How Crawling Works?
Basically, what happens here, let’s suppose this is your website okay, and on your website you can add multiple information, your information is in the form of text, in the form of images, in the form of video, so whatever is the information you added in your website, here we have one web crawler or you can also say web spider, so your web crawler or web spider visit on your website or any particular post URL and then the crawler start crawling every single information and then with the help of internet, it extracts that information and save this information in the search engine’s database okay. So this is what we call as crawling. But there is a twist, these crawlers follow some policies to store the information in the search engine.
Which Policies does Crawlers follow?
So let’s figure it out and understand this. So it has many policies by which it determines which website to choose or which to not. For example, we have selection and revisit policy, parallelization policy, then we have politeness policy, then web scrolling and then indexing. So let understand each policies one by one below:
1. Selection & Revisit Policy
So first of all, we have one selection policy and according to the selection policy, crawler decide which page it should download and which it should not, and next we have revisit policy, revisit policy means that, crawler schedule the time like when it should reopen the workface and added the changes in its database okay.
let’s suppose if this is your website and web crawler visit to your website and read every single information and after some days, if you do any change in your web page in that case, here google crawler again visit to your website and again crawl all the recommended average changes which you’ve done in your website and store that changes in their database.
2. Parallelization Policy
Next we have parallelization policy, so in the case of parallelization policy, crawlers use multiple processes at once to explore the links known as distributed chronicle.
So what happens here in this case, here let’s suppose, this is your website and your website’s link is mentioned in a different website and again your website link is mentioned in a different website, so what happens here, when google crawler read your website they can treat every single content available in your website. Here again and again, when web crawler read information of the website, then they can also find your website link and then they again visit to your website with the help of this particular link and again read all the information available in this particular page.
Alright so by this way, they can read your website content with the help of multiple sources here so this is known as multiple process and distributed trolley.
3. Politeness Policy
Now next we have politeness policy, so in the case of politeness policy, what happens here, let’s say, we have a term known as crawl delay. So what is this crawl delay means? So when google crawlers extract any information from your website, then they just take a pause for some millisecond and that millisecond pause is known as crawl delay. So when crawlers can access your website without any errors and you follow google’s language policy where your language does not support any hate speech or any illegal work then the crawlers can extract the language used from your website. So this is known as politeness policy.
4. Web Scrolling
Now next we have, how the web scrolling, so let’s see how scrolling works. So let’s say, we have a search engine, here let us suppose google is your search engine. So when someone put any query in the search engine, so what happens, here your web spider or your web crawler visit to your website, read every single web page, crawl your website, copy the information in your website and store that information in the database. And when the user put any information there, then that information is figure out from this database and display the result from this database to the user, this is what we call as web scrolling.
5. Indexing
So now we have indexing. So as google crawler or your search engine crawler crawl every information from your website then after crawling your information, so now the next step is to do index that information. So what indexing means? So indexing is just like you can do indexing of your information.
okay let’s suppose if you want to figure out any particular topic from any book, so how you will find? you will start searching from reading every topic or you will simply look at the index page and then find out that particular topic. Just like that, same thing happens with the google search engine.
So in google search engine also, we have lot of information, so just to fetch the user’s query from that particular database, it is really difficult to find, that’s why here search engine do indexing on the basis of keywords. So when someone search for any particular query then corresponding to their keyword, we can check their relevant keywords and then that particular query or information will reflect to your user.
How Indexing happens on Google?
Let’s say, here we have a user, and the user do any search through keywords so the search query is passed with the help of your query engine and then your indexer will get keyword from your webpages, after that, it will start fetching your keyword, they store these keywords in the form of indexing to their index file or repository.
Let’s understand because it’s important, what basically is happening here, let’s say, if your website is working on same keyword, another website that’s also working on the same keyword, in that case, the indexer provide them ranking like, this particular keyword will rank first, this particular keyword second on the basis of that, they add these keyboard in their repository and when user put any query then your query engine look at in the index in the indexed file and get list of the match pages here and show result page to your users like whatever is the relevant result page, relevant information, your user want to get that information will be visible to your users.
What are the types of Indexing?
Alright, now let’s also discuss about its types. So we have have two type of indexing. So one is your forward Indexing and next one is your backward Indexing. So what is this exactly? Let’s understand this as well below.
1. Forward Indexing
So let’s say, in case of forward Indexing, here these are the keywords let’s just suppose, we have document 1 and in our document 1, we have these keywords like, cow, seeds etc. So basically, it maps the keywords in different positions and key ids for the indexing purpose, so that google can find them easily when users request them while searching. So this is called as forward Indexing.
2. Reverse Indexing
Now we have reverse indexing too, so in this case, the forward indexers are stored and converted to reverse index in which each document containing a specific keyboard and is put together with other document containing that keyword. Let us understand this in simple word, so what happens here, in case of forward Indexing, we have primary segment, and we can check our particular keyword in document using maps or orders.
But in case of reverse indexing, we have keyword here and we can figure it out the document corresponding to that keyword, here let’s suppose, we have a keyword, corresponding to any keyword, then we have this document here, so this is known as reverse indexing.
What is Crawl Budget?
Now next is crawl budget, so what is the crawl budget here? So the number of time a search engine spider crawl your website in a given allotted time is what we call your crawl budget.
okay let’s suppose this is your website, and then google crawler visit your website first time and crawl every single information, when you are doing some changes in your website again, web crawler visit to your website and crawl every single information. In that case, here you can see that your crawl budget is 2, because two times your web spider will visit on your website, so this is what we call as Crawl Budget.
How to optimize Crawl Budget?
Now we have to optimize our crawl budget so that our website will rank on our search engine first page, for this we have to opt some strategies here, so what are these strategies? In this you have to avoid using your rich media files, and fix your internal and the external links. Now I know, you might not be aware about these terms. Since this is first module, I will explain everything for better understanding.
1. Internal Linking
So what is internal linking here, so let’s suppose this is your home page then we have your category page, inside your home page and all these pages are interconnected to each other, so these inter connected pages links are known as internal linking.
So if your google crawler reads information from this particular page, then they can also figure out the link of these pages and they can again visit these pages and read every single information available on these pages, so this is how your internal linking help your search engine spider to read all the information from your website, so internal name inform google to other relevant pages on your website and even the keywords for which you would like them to rank the internet linking helps point both to key pages much faster, that’s why we have to do internal linking to optimize our crawl budget.
2. External Linking
Now next we have external linking, in case of external linking, these links help search engine to understand the context of the pages as well as providing a good user experience.
So in this case, what happens here, let’s suppose this is your website and in your website, we have a link which redirect your website to an external link, in this case, you can tell your google spider about this particular website they can again visit that website and read all the single information available on that website. This is what we call as external linking.
Also, we have to make use of our social channels that will also help you to rank your website faster on your search engine result page, so these are the ways, you can optimize your crawl budget.
Alright, so our first module is completed now. We have tried our best to teach you each and everything in the best possible easiest language. We kept the concepts light and explained everything using examples. We hope you must have understood the concepts.
However, there is lot to learn about SEO, so we will meet again in our next SEO article. If you have any questions or doubts regarding any concepts, feel free to comment below. So that’s all for today’s article. Thank you for reading!
“So keep Learning, keep Growing!”
