How to add a site to search engines
Organic search traffic is the basis for the development of any site. To get it, you first have to get into the search index. It is a global database from which Google “pulls” results when a user enters a query. If you are not in the database, the chances of getting search traffic is zero. We offer a small guide to help you add the site to search engines.
Why do I need to add a site to search engines?
Most Internet users look for information through search engines. Google is especially popular. Search engine crawlers crawl websites, collecting URLs into a huge database, an index. Without this indexing, your site will not appear in search results.
A Crawler, or search robot, is a program that clicks on links on a website. In this way, the search engine records all web pages and expands its index. Robots periodically visit already known sites to discover new pages and index them.
Search engine robots are highly likely to find your site themselves and find that you have added new content. But it may take time – it’s hard to say exactly how long. You can help the crawlers get into the site and scan it as quickly as possible.
How to add a site to search engines
There are several ways to add pages to the search index. Sometimes you can use manual addition, but it does not support all search engines, and with a large number of pages, the process will take a very long time.
Adding manually
The easiest way to add a page to Google is to request indexing manually. You can do this in Google Search Console, using the URL validation tool. If the web page is not already in the index, click “Request Indexing” and the search engine will add it if there are no errors. This method is good if you only need to add a few pages. But if you want to add a whole site, including hundreds or thousands of pages, there is a more efficient way.
Site map sitemap.xml
If you need to scan a large number of pages, use a Sitemap file. A Sitemap helps search engines find pages on your site. If you’re just starting a website, it’s worth submitting your Sitemap file as soon as possible. This file contains data about pages with products, text, videos, images and news.
Find the sitemap file at one of the addresses:
- your-domain.com/sitemap.xml
- your-domain.com/sitemap-index.xml
If you don’t have a sitemap yet, create one. In various CMS this can be done with the help of plugins. Also suitable online generators or special programs with an automated process.
How do I add a site to Google?
On the left side of the Google Search Console you will find the Sitemaps item. On the tab that opens, you will see a field where you need to copy the URL of the site map and send it. If you have multiple sitemap files, submit each of them.
When you add new content to the site, we recommend asking Google to index it manually through the URL checker tool. So you speed up the indexing of new content.
How do I add a site to other search engines?
With Bing Webmaster tools, you can improve the performance of your website in this search engine. In the menu on the left there is an item Sitemaps (Sitemaps). Select it, click “Submit” and specify the URL of the card.
The search engine Yahoo also works with the Bing index, so you don’t have to take extra steps to get into this search engine.
Another search engine – DuckDuckGo – looks for new content by itself, it has no tool to add new pages or site map. The system uses its own crawler, and also works with the Bing index, uses crowdsourcing resources (e.g. Wikipedia), Sportradar and many others.
How do I check that the site is indexed?
The most reliable way to check if your site is indexed is to use the “Coverage” report in Search Console. Here you can see pages with different categories:
- Valid – pages have been indexed and can be shown in search results, no need to do anything.
- Valid with warnings – web pages can be displayed in Google search results, but there are problems.
- Error – These pages cannot be indexed and will not appear in search results because of errors.
- Excluded – pages have not been indexed and will not appear in search results because Google believes the content is not worth indexing.
Select the desired state (error, warning or exception) and look at the details on the page. You will find a list of errors and the number of pages affected.
To check if individual pages are in the index, you can use the URL checker.
Many guides recommend that you check your indexing status by typing “site:your-domain.com” into Google. It can also be used, but it is not the most reliable source of information.
In the search engine Bing to check the indexing of the site can be with the tool Site Explorer. Here you can filter the indexed URLs and check if all the necessary pages are included in the index.
How to fix indexing problems
An indexing error is a problem that affects the ability of search engines to track and include one or more pages of a website in the index. Problems with indexing can occur for a variety of reasons. The sooner you eliminate them, the sooner the search robots will re-screen the site.
- Server error (5xx) occurs when the site cannot be accessed due to server malfunction. Check which status code 5xx is returned. Some 5xx errors are temporary and require no action. Make sure that the server is not overloaded and is configured correctly. As a rule, such problems do not arise if you use a reliable hosting provider. If the 5xxx error still occurs, contact technical support or the administrator of the dedicated server.
- Indexing ban in robots.txt file. Google always tries to follow the directives in the robots.txt file. Blocking can apply to the entire site or individual pages. To remove it, remove the text from the robots.txt file:
User-agent: Googlebot
Disallow: /
User-agent: *
Disallow: /
- No indexing is prohibited by the noindex tag. This tag is located in the head section of the web page and prohibits indexing. If the page needs to be indexed, remove the tag.
- Duplicate page without canonical tag. This page does not point to the canonical page, so Google itself searches for it and adds the canonical version to the index. If you want to index this page, you will have to specify the canonical attribute for it.
- Redirect page. Google does not index pages with permanent redirects, so the redirect must be removed if the web page is to be indexed.
- The page returns a 404 error status code when Google makes a request. Googlebot did not find the page through the sitemap, but there is probably a link on another website. It is also possible that this URL existed in the past and was deleted. If the 404 response is intentional, you can leave it. It won’t affect SEO. However, if the page has moved, set up a 301 redirect.
Sometimes pages can be excluded from the index for legal reasons, but this is quite rare.
Why isn’t it enough just to add your site to the index?
Most of the search traffic gets sites that are on the first page of results. Users usually find the answer to their question in the top 3 results, so it is extremely important to promote the site for targeted keywords. SEO optimization, which includes a whole list of actions, including:
- internal optimization of the site;
- External optimization – building quality link mass;
- Creating interesting content with the addition of targeted keywords.
The ranking algorithms take into account a variety of indicators, including:
- page load speed;
- behavioral factors – time of visit, clicks on links;
- The information corresponds to the key queries;
- the number and quality of backlinks.
To help Google and other search engines rank your site higher, you need a range of measures aimed at improving all indicators.
Conclusions
Google and other search engines know how to search web pages on their own. Crawlers are constantly improving to find interesting and useful content. By creating a sitemap or manually requesting indexing, you will help search engines discover updates to your website. We recommend manually adding new pages to Search Console to speed up indexing. And if you only create a website, pay attention to the designer from TutHost.
