How to audit technical SEO and find out what has happened to your website

In the first article, we looked at how to organise <title>, H1–H3 headings and internal links. But even when everything looks clear and logical on paper, technical SEO configuration can turn a website into CHAOS.

A search engine might miss some pages, misunderstand their content or discover several almost identical versions. Meanwhile, the website owner looks at the attractive design and wonders: “Why aren't we getting customers from search?”

That is why I do not begin a technical SEO audit with magical tools and hundred-page reports. First, I need to understand what is actually on the website, which pages crawlers can see and what their main SEO elements say.

Here is my approach. I begin with a map of the entire website and record the following for each page:

• its URL;
• <title>;
• meta description;
• H1;
• the main H2 and H3 headings;
• the purpose of the page.

This immediately reveals pages that repeat each other, headings that do not match the content and titles full of nonsense that prevents Yandex and Google from understanding what a page is about.

The result is a kind of website skeleton. You can use it to check the structure, find duplicates and decide which pages to fix, combine, block from crawling or remove entirely.

How search engine crawlers discover and index pages

After checking the structure, you need to understand how crawlers discover your pages. First, a crawler must learn that a page exists: by following an internal or external link or finding its URL in a sitemap. It then downloads the page, examines the text, headings, links and technical settings, and sends the information to the search engine's database. Successful crawling does not necessarily mean Yandex or Google will include the page in search results.

You can open every door to crawlers, lay the table and wait for guests, but that still does not guarantee every page will appear in search. If Yandex or Google finds several identical or nearly identical pages, it may choose one and treat the others as duplicates, leaving them out of the results.

How robots.txt guides search engine crawlers

A website also has a robots.txt file, a kind of guard at the entrance. It tells crawlers which sections they may visit and which they should avoid. A single incorrect line can accidentally block an important page, a whole catalogue or the entire website from Yandex and Google. Everything may still appear to work for visitors, while crawlers cannot access the content properly.

How to avoid duplicate pages

To avoid cleaning up another round of CHAOS later, plan the website's development in advance. List your existing service pages, the services still to add, the cities that actually need separate pages and how those regional versions will differ. If the Kazan page repeats the Cheboksary page with only the city name changed, a search engine may consider it a duplicate. Each new page therefore needs its own content, offer and value for a particular audience.

Why Astro works well for SEO and AI search

I chose Astro because it turns pages into ready-to-read HTML when the website is built. A crawler opens a page and immediately sees <title>, headings, text, links and other important elements. It does not have to wait for a large amount of JavaScript to load before the website finally displays its main content.

Which SEO settings can you control in Astro?

With Astro, I can control almost the entire technical skeleton of a page: <title>, meta description, H1–H6 headings, canonical URLs, page addresses, internal links and the sitemap. This lets me plan the website's structure and ensure each page has its own purpose, a clear address and unique content.

Astro itself will not put a website at the top of search results. It will not write useful copy, create demand or make a company authoritative. It does help remove technical CHAOS and show search engines what is actually on the page.

How an Astro website's speed affects SEO

By default, Astro sends minimal JavaScript to the browser and adds it only where interactivity is needed. Pages load faster, the main content appears sooner, and visitors do not have to wait for the website to come to life. Speed and usability matter for SEO, but they are not a magic ticket to the top. A fast website with useless content is still of little value to a search engine.

Why ordinary SEO matters for AI search

AI search does not exist in isolation from ordinary search. A search engine first needs to discover a page, access its content, index it and understand its subject. The page's information can then be used in AI-generated answers.

That makes Astro's ready-to-read HTML useful for AI search too: headings, text and links are immediately available to the search engine within a clear structure.

How duplicate pages get in the way of SEO

On my own website, I decided to create service pages for several cities at once. The underlying content stayed almost identical: the URL, the city name in <title>, H1–H3 headings and a few phrases changed. At the time, it seemed like an excellent approach: configure it once and immediately get dozens of pages for regional search queries.

How I created service pages for different cities

Separate versions of each service page were generated automatically for Cheboksary, Kazan, Moscow and other cities. For example: “Landing pages for SEO and AI search in Cheboksary”, “Landing pages for SEO and AI search in Kazan” and “Landing pages for SEO and AI search in Moscow”.

What the regional page URLs looked like

Each city had its own address:

• /cheboksary/catalog/ai-landing
• /kazan/catalog/ai-landing
• /moscow/catalog/ai-landing

Technically, everything looked tidy: separate URLs, different headings and the appropriate city on each page. But the main content barely changed, so search engines still saw very similar pages.

How I expanded the SEO pages to cities abroad

I did not stop there. Since the process was automatic, I decided to create English versions for cities abroad in the same way. Separate pages appeared for Almaty, Dubai, Abu Dhabi and other cities around the world. The appropriate city name was inserted into <title>, headings and URLs, while the underlying content again remained almost identical.

The pages for cities abroad used the same URL structure: /en/almaty/catalog/ai-landing, /en/dubai/catalog/ai-landing, /en/abu-dhabi/catalog/ai-landing. Russian and English versions were generated automatically using the same approach. You can see the website structure and page-generation logic in the Git repository: https://github.com/Daos0/RM_Syst_Astro.

How the sitemap exposed 400 pages to search engines at once

After generating the regional pages, I needed to show them to Yandex and Google. The website used a sitemap for this: a file listing URLs that crawlers are invited to visit.

With Astro, you can generate a sitemap automatically whenever the website is built. Add new routes, rebuild the project, and the new pages appear in the sitemap.

It seemed ideal: the crawler did not have to discover each page by chance through internal links. We handed it a ready-made map and said:

“Here are all our pages. Come in, explore them and add them to search!”

Why a sitemap does not guarantee inclusion in search

There is a catch. A sitemap helps search engines discover URLs, but it does not force them to include every page in the results.

In my case, the sitemap faithfully exposed almost all the CHAOS I had created: hundreds of service pages, versions for different cities, and Russian and English URLs.

The pages technically existed. They opened, had their own URLs and H1–H3 headings, and appeared in the sitemap. But a search engine evaluates more than the presence of separate URLs. It compared the content and saw that many pages differed mainly in their city names.

It looked something like this:

• Cheboksary: one version of the text;
• Kazan: almost the same text;
• Moscow: practically the same text again;
• Dubai: an English version of the same approach;
• Abu Dhabi: another similar page.

I had changed a few words without creating hundreds of truly distinct, useful pages for search engines.

Why regional pages started disappearing from the index

At first, things looked promising. Some pages appeared in the search index, and I was already celebrating:

“Not bad. Around 400 pages almost all at once!”

The celebration did not last. A few days later, many pages began dropping out of the index. This happened particularly quickly in Yandex.

Search engines found the pages, crawled and compared them, then apparently decided there was little reason to show every version.

Changing the city in the URL, H1–H3 headings and a few sentences had not made each page unique and useful for that region.

I cannot claim that automatically substituting city names was the only reason pages disappeared. Many factors affect indexing:

• demand for a particular page;
• the amount of similar content;
• internal linking;
• canonical URLs;
• the quality and length of the text;
• website authority;
• the page's technical condition;
• real value for the user.

But after generating so many similar regional versions, the website structure became the very CHAOS I then spent a long time untangling.

How to create useful pages for different cities

Regional pages are a reasonable idea. If a company actually works in several cities, separate pages can show visitors relevant terms, prices, timelines and offers.

The problem begins when you create the same page for every city and merely replace the place name in a few locations.

A regional page needs a reason to exist.

For example, it might have different:

• available services;
• prices and working arrangements;
• project timelines;
• examples of work in that region;
• reviews from local customers;
• delivery or on-site visit arrangements;
• contact details for a regional representative;
• information about the local market and customers' needs.

If the Kazan page differs from the Cheboksary page only in the word “Kazan”, why should a search engine store and show both?

It is better to expand into regions gradually. Start with one general page and one complete city-specific version. Check whether it generates impressions, visits and enquiries. Only then move on to the next city.

Otherwise, you can quickly create 400 URLs and acquire 400 new problems even faster.

How I fixed technical SEO on my own website

After the experiment, I had to decide what to do with the whole setup.

Some pages never appeared in search. Others were indexed and then disappeared. Cities, language versions and services had become tangled together. Even I was afraid to open the English section.

I could not fix everything immediately. The website stayed online, I had little time to work on it, and the CHAOS quietly took root.

Gradually, I changed my strategy.

Which website versions I decided to keep

Instead of dozens of nearly identical regional sections, I kept:

• the general Russian-language website;
• the general English-language website;
• a separate Cheboksary section.

I chose Cheboksary because I live here, rather than because search engines particularly favour the city. For Cheboksary, I kept the home page and several main service pages in the catalogue. Other materials remained in the general version without being automatically duplicated for every city.

I first blocked the unnecessary regional pages, then deleted them.

Why deleted pages do not disappear from search immediately

Deleting a URL is not the end of the story.

Search engines already know about these pages, have stored information about them and have added some URLs to their databases. A deleted page therefore does not disappear from Yandex or Google's memory instantly.

A crawler needs to revisit the website, check the old URL, see the changes and update the index.

For a while, webmaster tools may still show:

• old addresses;
• notices about excluded pages;
• duplicates;
• crawl errors;
• URLs that no longer exist on the website.

This can noticeably delay reindexing the corrected website. You can tidy the structure in a day, but search engines need time to crawl the pages again, see the changes and update their indexes.

How I separated general and regional pages

I created different catalogue blocks and separate copy for the general website and the Cheboksary section.

A regional page now needed to differ in more than the city name in its heading. It needed its own purpose, content and clear value for the local audience.

The structure became much simpler:

• general services appear in the main website;
• separate Cheboksary pages support the regional section;
• the English version is an independent section;
• extra cities are no longer generated automatically without a reason.

It became easier for search engines to understand what each page was responsible for. And easier for me, too.

Do you need special settings for AI search?

You do not need to rebuild the website separately for AI search. A search engine still has to discover a page, access its content, index it and understand what it is about.

The foundation remains the same: accessible HTML, a clear structure, meaningful headings, internal linking and unique copy. Special AI markup will not rescue a page that merely copies another article or contains more nonsense.

Astro helps with the technical side: search engines receive ready-to-read HTML containing text, links and headings. The framework itself does not determine how useful or unique the page is.

What I learned from my technical SEO experiment

Generating hundreds of pages automatically was easy. Explaining to a search engine why each one was needed was harder.

I had separate URLs, H1–H3 headings, fast HTML and an automatically generated sitemap. Everything looked tidy technically. Underneath, though, were nearly identical pages with little more than the city name changed.

Some pages never entered the index, others appeared and then disappeared, and I had to untangle all the CHAOS I had created.

Now I try to avoid generating pages in advance. I first build one complete version with its own copy, offer and value for a specific audience. I create the next page only when it actually differs from an existing one.

Technical SEO helps search engines find a website, crawl it correctly and understand its structure. Even a perfectly configured website does not automatically become authoritative.

One question remains: why should Yandex, Google or AI search trust our website and show it above competitors?

We will explore that in the next article, Website link authority: building authority for SEO and AI search.