The SEO Spider structure section
Structure is crucial from the SEO point of view. The easier a site is to navigate, with simple paths for the crawler and the user, the more it will be rewarded by search engines.
From this point of view, the Taxonomies provide us with a graphic representation of the distribution of the pages – and the connections between them – within the domain.
Https is a security protocol whose absence, for some time, is considered penalizing by search engines. To implement it on your site, follow this guide.
In this section, you can filter the pages according to two criteria – with and without https –and to export the list.
The canonical attribute is very useful if our site has multiple pages with the same content. When we use this attribute on a page, we tell the search engine that this is the page that must be considered as the one with the original contents. If we do not use it, the search engine crawler considers all pages with the same contents as duplicates, penalizing the entire site.
We can filter the list by choosing to display only the pages with a canonical attribute, or those without.
We can export the list by clicking on the orange “Export” button:
The status is what the client (i.e., the user) receives from the server, to which it has requested to view a certain web page via the HTTP communication protocol. These statuses can have different codes depending on the type of response. You will surely know the code “404”, which corresponds to “resource not found” and the code “301” which signals a redirect to another page. We can use the filter in the upper left to view which pages each of these three codes return.
In this case, too, we can also export one or more lists (for example codes 200 and 404) with the Export button.
Under “Language”, thanks to the “filter” function, we can view which pages do not have the hreflang or x-default attributes and export them to a file. When we have a multilingual site, on each page, it is good practice to insert the link to its equivalents in each language using the <link hreflang> attribute. The x-default attribute instead, indicates which “version” should be considered by default when the visitor of the site comes from a region whose language is not present on the site.
The robots.txt file on the site contains the “instructions” to be given to the crawler about the pages it has to explore, how to do it, and which to ignore.
With the “filter” function you will be able to view which pages in the robots.txt file have the noindex attribute (an attribute which blocks the indexing of that page in search engines), noodp and noydir – which prevent search engine robots to use the site information in the Yahoo and Open Directory Project directories respectively. In any case, both of these directories no longer exist, so they are obsolete attributes.
In this case, too, you can export the pages according to one or more of these filters: