How to audit technical SEO and find out what has happened to your website
In the first article, we looked at how to organise <title>, H1–H3 headings and internal links. But even when everything looks clear and logical on paper, technical SEO configuration can turn a website into CHAOS.
A search engine might miss some pages, misunderstand their content or discover several almost identical versions. Meanwhile, the website owner looks at the attractive design and wonders: “Why aren't we getting customers from search?”
That is why I do not begin a technical SEO audit with magical tools and hundred-page reports. First, I need to understand what is actually on the website, which pages crawlers can see and what their main SEO elements say.
Here is my approach. I begin with a map of the entire website and record the following for each page:
- its URL;
<title>;- meta description;
- H1;
- the main H2 and H3 headings;
- the purpose of the page.
This immediately reveals pages that repeat each other, headings that do not match the content and titles full of nonsense that prevents Yandex and Google from understanding what a page is about.
The result is a kind of website skeleton. You can use it to check the structure, find duplicates and decide which pages to fix, combine, block from crawling or remove entirely.
How search engine crawlers discover and index pages
After checking the structure, you need to understand how crawlers discover your pages. First, a crawler must learn that a page exists: by following an internal or external link or finding its URL in a sitemap. It then downloads the page, examines the text, headings, links and technical settings, and sends the information to the search engine's database. Successful crawling does not necessarily mean Yandex or Google will include the page in search results.
You can open every door to crawlers, lay the table and wait for guests, but that still does not guarantee every page will appear in search. If Yandex or Google finds several identical or nearly identical pages, it may choose one and treat the others as duplicates, leaving them out of the results.
How robots.txt guides search engine crawlers
A website also has a robots.txt file, a kind of guard at the entrance. It tells crawlers which sections they may visit and which they should avoid. A single incorrect line can accidentally block an important page, a whole catalogue or the entire website from Yandex and Google. Everything may still appear to work for visitors, while crawlers cannot access the content properly.
How to avoid duplicate pages
To avoid cleaning up another round of CHAOS later, plan the website's development in advance. List your existing service pages, the services still to add, the cities that actually need separate pages and how those regional versions will differ. If the Kazan page repeats the Cheboksary page with only the city name changed, a search engine may consider it a duplicate. Each new page therefore needs its own content, offer and value for a particular audience.
Why Astro works well for SEO and AI search
I chose Astro because it turns pages into ready-to-read HTML when the website is built. A crawler opens a page and immediately sees <title>, headings, text, links and other important elements. It does not have to wait for a large amount of JavaScript to load before the website finally displays its main content.
Which SEO settings can you control in Astro?
With Astro, I can control almost the entire technical skeleton of a page: <title>, meta description, H1–H6 headings, canonical URLs, page addresses, internal links and the sitemap. This lets me plan the website's structure and ensure each page has its own purpose, a clear address and unique content.
Astro itself will not put a website at the top of search results. It will not write useful copy, create demand or make a company authoritative. It does help remove technical CHAOS and show search engines what is actually on the page.
How an Astro website's speed affects SEO
By default, Astro sends minimal JavaScript to the browser and adds it only where interactivity is needed. Pages load faster, the main content appears sooner, and visitors do not have to wait for the website to come to life. Speed and usability matter for SEO, but they are not a magic ticket to the top. A fast website with useless content is still of little value to a search engine.
Why ordinary SEO matters for AI search
AI search does not exist in isolation from ordinary search. A search engine first needs to discover a page, access its content, index it and understand its subject. The page's information can then be used in AI-generated answers.
That makes Astro's ready-to-read HTML useful for AI search too: headings, text and links are immediately available to the search engine within a clear structure.
How duplicate pages get in the way of SEO
A common story: a website owner decides to create service pages for several cities at once. The underlying content stays almost identical: the URL, the city name in <title>, H1–H3 headings and a few phrases change. It seems like an excellent approach: configure it once and immediately get dozens of pages for regional search queries.
How service pages for different cities are usually created
Separate versions of each service page are generated automatically for Cheboksary, Kazan, Moscow and other cities. For example: “Turnkey landing page in Cheboksary”, “Turnkey landing page in Kazan” and “Turnkey landing page in Moscow”.
What regional page URLs look like
Each city gets its own address:
/cheboksary/services/landing/kazan/services/landing/moscow/services/landing
Technically, everything looks tidy: separate URLs, different headings and the appropriate city on each page. But the main content barely changes, so search engines still see very similar pages.
How SEO pages get multiplied for cities abroad
The same approach is often carried over to an English version: since the process is automatic, pages can be created for Almaty, Dubai, Abu Dhabi and other cities around the world. The city name is inserted into <title>, headings and URLs, while the underlying content again remains almost identical.
The URL structure is the same: /en/almaty/services/landing, /en/dubai/services/landing, /en/abu-dhabi/services/landing. Russian and English versions are generated automatically using one approach.
How a sitemap exposes 400 pages to search engines at once
After the regional pages are generated, they have to be shown to Yandex and Google. A sitemap is used for this: a file listing URLs that crawlers are invited to visit.
With Astro, you can generate a sitemap automatically whenever the website is built. Add new routes, rebuild the project, and the new pages appear in the sitemap.
It seems ideal: the crawler does not have to discover each page by chance through internal links. It gets a ready-made map and the message:
“Here are all our pages. Come in, explore them and add them to search!”
Why a sitemap does not guarantee inclusion in search
There is a catch. A sitemap helps search engines discover URLs, but it does not force them to include every page in the results.
In this situation, the sitemap faithfully exposes all the CHAOS that has been created: hundreds of service pages, versions for different cities, and Russian and English URLs.
The pages technically exist. They open, have their own URLs, <title> and H1–H3 headings, and appear in the sitemap. But a search engine evaluates more than the presence of separate URLs. It compares the content and sees that many pages differ mainly in their city names.
It looks something like this:
- Cheboksary: one version of the text;
- Kazan: almost the same text;
- Moscow: practically the same text again;
- Dubai: an English version of the same approach;
- Abu Dhabi: another similar page.
A few words change, but hundreds of truly distinct, useful pages never appear.
Why regional pages disappear from the index
At first, things look promising. Some pages appear in the search index, and it feels like a win:
“Not bad. Around 400 pages almost all at once!”
The celebration rarely lasts. A few days later, many such pages begin dropping out of the index. In Yandex this tends to happen particularly quickly.
Search engines find the pages, crawl and compare them, then decide there is little reason to show every version.
Changing the city in the URL, <title>, H1–H3 headings and a few sentences does not make each page unique and useful for that region.
Automatic city substitution is not necessarily the only reason pages disappear. Many factors affect indexing:
- demand for a particular page;
- the amount of similar content;
- internal linking;
- canonical URLs;
- the quality and length of the text;
- website authority;
- the page's technical condition;
- real value for the user.
But mass-produced similar regional versions quickly turn a website structure into the very CHAOS that then takes a long time to untangle.
How to create useful pages for different cities
Regional pages are a reasonable idea. If a company actually works in several cities, separate pages can show visitors relevant terms, prices, timelines and offers.
The problem begins when you create the same page for every city and merely replace the place name in a few locations.
A regional page needs a reason to exist.
For example, it might have different:
- available services;
- prices and working arrangements;
- project timelines;
- examples of work in that region;
- reviews from local customers;
- delivery or on-site visit arrangements;
- contact details for a regional representative;
- information about the local market and customers' needs.
If the Kazan page differs from the Cheboksary page only in the word “Kazan”, why should a search engine store and show both?
It is better to expand into regions gradually. Start with one general page and one complete city-specific version. Check whether it generates impressions, visits and enquiries. Only then move on to the next city.
Otherwise, you can quickly create 400 URLs and acquire 400 new problems even faster.
How to fix technical SEO after mass-produced pages
Once such a setup exists, you have to decide what to do with it.
The usual picture: some pages never appear in search, others are indexed and then disappear. Cities, language versions and services become tangled together.
Fixing everything at once is rarely possible, so it is better to change the strategy gradually.
Which website versions are worth keeping
Instead of dozens of nearly identical regional sections, it makes more sense to keep:
- the general Russian-language website;
- a general English-language website, if you need one;
- separate sections only for cities where the company actually works and has something specific to say.
A city gets its own section not because search engines particularly favour it, but because there is real work, customers and local conditions there. A home page and several main service pages are enough for such a city. Other materials stay in the general version without being automatically duplicated for every city.
Unnecessary regional pages are first closed from indexing, then deleted or served with a 410 status code.
Why deleted pages do not disappear from search immediately
Deleting a URL is not the end of the story.
Search engines already know about these pages, have stored information about them and have added some URLs to their databases. A deleted page therefore does not disappear from Yandex or Google's memory instantly.
A crawler needs to revisit the website, check the old URL, see the changes and update the index.
For a while, webmaster tools may still show:
- old addresses;
- notices about excluded pages;
- duplicates;
- crawl errors;
- URLs that no longer exist on the website.
This can noticeably delay reindexing the corrected website. You can tidy the structure in a day, but search engines need time to crawl the pages again, see the changes and update their indexes.
How to separate general and regional pages
The general website and the regional sections need different catalogue blocks and separate copy.
A regional page then differs in more than the city name in its heading. It has its own purpose, content and clear value for the local audience.
The structure becomes much simpler:
- general services appear in the main website;
- regional pages support the section for a specific city;
- the English version is an independent section;
- extra cities are no longer generated automatically without a reason.
It becomes easier for search engines to understand what each page is responsible for. And easier for the website owner, too.
Do you need special settings for AI search?
You do not need to rebuild the website separately for AI search. A search engine still has to discover a page, access its content, index it and understand what it is about.
The foundation remains the same: accessible HTML, a clear structure, meaningful headings, internal linking and unique copy. Special AI markup will not rescue a page that merely copies another article or contains more nonsense.
Astro helps with the technical side: search engines receive ready-to-read HTML containing text, links and headings. The framework itself does not determine how useful or unique the page is.
The main lesson about technical SEO and mass-produced pages
Generating hundreds of pages automatically is easy. Explaining to a search engine why each one is needed is harder.
Separate URLs, <title>, H1–H3 headings, fast HTML and an automatically generated sitemap do not guarantee indexing. If the pages underneath are nearly identical, with little more than the city name changed, some will never enter the index and others will appear and then disappear.
So pages should not be generated in advance. First build one complete version with its own copy, offer and value for a specific audience. Create the next page only when it actually differs from an existing one.
Technical SEO helps search engines find a website, crawl it correctly and understand its structure. Even a perfectly configured website does not automatically become authoritative.
One question remains: why should Yandex, Google or AI search trust our website and show it above competitors?
We will explore that in the next article, Website link authority: building authority for SEO and AI search.