Guides

Why your website does not appear on Google: crawling, indexing and ranking

Being online does not mean being in the index. Before changing copy or adding plugins, identify the exact point where the page stopped progressing.

The short answer

A page may be absent because Google does not know it, cannot crawl it, decides not to index it, or indexes it without considering it relevant enough for a particular search. These are four different problems: diagnose them through URL Inspection, the HTTP response, published HTML, directives, canonicals, links, the sitemap and content quality. No manual request can guarantee indexing or ranking.

Gian Luca Partengo
Web developer since 1995 · bespoke websites · Updated: 1 September 2026

A page must pass four stages before it can appear on Google

Search does not automatically receive every published page. Google must first find the URL, reach it, understand it and finally select it for a query.

01Google discovers a reachable URL
02Googlebot downloads a valid response
03Google processes the page and may add it to the index
04The system may show it for relevant searches

Discovery, crawling, indexing and serving in search results are connected but not equivalent stages.

“It does not appear” can describe four completely different situations

Assign the URL to the right state before making changes. Otherwise you may work on content while the server is failing, or on the sitemap when the page is already indexed.

State	What it means	How to verify it	First sensible action
Unknown URL	Google shows no evidence of having discovered that specific URL.	URL Inspection, sitemap and actual internal links.	Make the page reachable and declare it in the canonical sitemap.
Discovered, not crawled	The URL is known, but Google has not downloaded it yet or has postponed crawling.	Indexing status, server logs and last crawl date.	Check architecture, server stability, duplicates and the URL’s value.
Crawled, not indexed	Google read the page but is not currently retaining it as an indexable result.	HTML seen by Google, selected canonical, content and quality signals.	Fix the actual cause rather than merely requesting indexing again.
Indexed, not visible for the query	The page is in the index but is not shown, or appears far down the results.	Performance report: queries, impressions, position and page.	Work on intent, usefulness, authority, links and competition.

The site: search operator can provide a clue, but it is not a complete inventory and does not replace URL Inspection and reports for the verified property.

How does Google know a page exists?

The web has no central register of pages. URLs are discovered by following links, reading sitemaps and returning to paths that are already known.

BEFORE CRAWLING

An isolated URL can exist for a browser and remain almost invisible to a crawler

A published page that is absent from navigation, index pages and the sitemap depends on accidental signals. Typing the address manually only proves that the server exposes it.

Reliable discovery begins with real HTML links inside an understandable architecture, supported by a consistent sitemap and any relevant external links.

01

Internal links

Hub pages, navigation and related content make the URL reachable and explain how it relates to the rest of the website.

02

XML sitemap

It communicates canonical URLs the website wants to make available, but remains a hint rather than a guarantee.

03

Coherent redirects

When a resource changes address, a direct path to the new URL avoids chains and contradictory signals.

04

External mentions and links

They help discovery and context but cannot repair a page that is blocked, duplicated or lacking useful content.

Discovered does not mean crawlable

Once Google knows the URL, Googlebot must obtain a response, follow any redirects and read the required resources without encountering blocks or instability.

Access

robots.txt blocks the request

A Disallow rule can prevent Googlebot from downloading the page. It is not, however, a reliable way to remove an already known URL from the index.

Infrastructure

The server is slow or returns errors

Timeouts, 5xx errors and overload reduce crawling capacity and can cause further requests to be postponed.

Perimeter

The page requires login or a challenge

Authentication, geoblocking, CAPTCHAs and firewall controls can make a nominally public page unreachable to the crawler.

Path

Long, cyclic or broken redirects

Chains, loops and empty destinations consume requests and prevent the crawler from reaching a stable final response.

Rendering

Essential resources are blocked

If content and navigation depend on inaccessible JavaScript or APIs, rendering may produce an incomplete page.

Efficiency

Thousands of useless URLs dilute crawling

Filters, parameters, infinite calendars and duplicates can multiply URLs without adding pages that deserve to be indexed.

The right check

Inspect the HTTP response, redirects, robots.txt, initial HTML, resources and logs together. Looking only at what the user’s browser displays may hide the problem encountered by the crawler.

Crawled does not automatically mean indexed

Google processes text, images, metadata and technical signals, compares similar versions and decides which URL to treat as canonical.

INDEX DECISION

A 200 response is necessary for many pages, but it does not make them search results by itself

During processing, Google may recognise a duplicate, select another canonical, interpret the page as a soft error or fail to find enough main content.

The goal is not to put 100% of URLs into the index: the canonical versions of important pages should be indexed. Duplicates, filters, redirects and 404s may be correctly excluded.

01

Different canonical

The page declares, or causes Google to select, another URL as its representative version.

02

Duplicate content

Parameters, print versions, filters or almost identical copy cause several URLs to converge on a single page.

03

Soft 404

The server returns 200, but the page looks empty, missing or unable to provide the promised resource.

04

Insufficient value

The page is accessible but adds no recognisable information beyond content already present on the website or the web.

Being indexed does not mean being ranked

The index makes a page eligible for evaluation. Whether it appears for a search then depends on relevance, quality, context, competition and ranking systems.

Level	What it proves	What it depends on	What it does not guarantee
Indexing	Google retains the page as a possible result.	Access, processing, canonicals and indexable content.	Impressions, clicks or a first-page position.
Ranking	The page competes for a specific query and context.	Intent, relevance, usefulness, authority and competition.	A stable position that is identical for everyone.
Search appearance	Title, snippet and available features may present the page.	Visible content, metadata, structured data and the query.	That Google will always use the website’s supplied wording.
AI answers	A page may be retrieved as a source in generative experiences.	Index, access, relevance, clarity and available evidence.	Guaranteed citations or a replacement for SEO.

How to read the main Search Console states without being misled

The Page indexing report groups known URLs and reasons. Not every exclusion is an error, and not every URL should enter the index.

Absent

URL is unknown to Google

Check that the address is correct, reachable, linked and present in the sitemap when it is meant to be public.

Waiting

Discovered – currently not indexed

Google knows the page but has not crawled it yet. Architecture, URL volume, server conditions and perceived value can all matter.

Processed

Crawled – currently not indexed

Repeating the request does not replace analysis of content, canonicals, soft 404s, duplication and rendering.

Consolidated

Alternate or duplicate page

This may be correct if Google indexed the desired canonical version. Check the declared and selected URL pair.

Excluded

Blocked by noindex or robots.txt

If the exclusion is intentional, there is no problem. If the page should appear, remove the appropriate directive or block.

Available

Indexed URL

This confirms presence in the index, not ranking. Visibility requires impressions and queries in the Performance report.

Reports and inspection have different roles

The report shows groups and trends; URL Inspection checks one page, distinguishes the live version from index data and displays Google’s selected canonical.

robots.txt, noindex, canonicals and sitemaps are not interchangeable

Four tools often treated as generic SEO commands actually answer different questions.

Tool	What it does	What it does not do	Common mistake
robots.txt	Controls crawler access to paths.	Does not guarantee removal of an already known URL.	Blocking the page so Google cannot read its noindex directive.
noindex	Requests that the page not be included in results.	Does not prevent crawling when the URL is accessible.	Leaving it on pages published from staging or templates.
rel=canonical	Indicates the preferred version among duplicate or very similar pages.	Is not a redirect and remains a signal, not an absolute command.	Pointing every page to the homepage or another language.
XML sitemap	Lists canonical URLs the website wants search engines to discover.	Does not guarantee crawling, indexing or ranking.	Including redirects, 404s, noindex pages and non-canonical URLs.

Signals should describe the same reality: internal links, canonicals, sitemaps, redirects and content should not point to different versions of one page.

HTTP status codes tell the crawler what actually happened

A visually correct page can send the wrong code. Google uses the server response to decide whether to process, transfer, remove or retry a URL.

Available

200 OK

The resource was served. This does not prove that its content is valid, unique or worthy of indexing.

Moved

301 permanent

Communicates a stable move and helps consolidate the destination when the path is direct and coherent.

Temporary

302 temporary

Indicates that the move may not be final. Using it for a permanent migration makes the signal less clear.

Missing

404 not found

Correct for a missing resource with no replacement. The error page can be useful, but the response must remain 404.

Removed

410 gone

Explicitly states that the resource was removed. It should not be used when an equivalent replacement exists.

Server error

5xx and timeouts

Signal a temporary infrastructure problem. If they persist, Google reduces requests and pages may leave the index.

When content only exists in JavaScript, Google has to complete an extra stage

Google can execute JavaScript, but crawling, rendering and indexing remain separate phases. Other crawlers do not necessarily have the same capabilities.

HTML FIRST, INTERACTION AFTERWARDS

Main content should not depend on a fragile chain of scripts and APIs

With client-side rendering, the initial response may contain only an empty shell. Google must queue rendering, download resources and execute code before it can see text and links.

Server-side rendering, static HTML or progressive enhancement make the document foundation immediately available and leave JavaScript to improve interaction.

01

Content in the initial HTML

The title, main copy, links and essential signals are readable without waiting for application execution.

02

Verified rendering

URL Inspection can compare the live page with what Google is actually able to process.

03

Links with a real href

Elements that imitate links through events do not offer the same discovery and accessibility as an ordinary anchor.

04

Stable canonicals and directives

Changing canonicals or robots directives with JavaScript relative to the initial HTML can create late or contradictory signals.

Sometimes there is no technical error: the page does not add enough value

A perfectly reachable page can remain outside the index or without visibility when it is duplicated, generic, mismatched to intent or produced merely to multiply URLs.

Content is too thin

A few interchangeable sentences do not explain the service, answer the question or distinguish the page.

Almost identical pages

Changing a city or keyword while leaving everything else unchanged creates alternatives that compete without a genuine editorial purpose.

Wrong intent

A generic commercial page is unlikely to replace a guide when the search requires instructions, comparison or diagnosis.

Ambiguous subject

Author, business, service, geographical area and responsibility should emerge consistently from content and structure.

Scaled production

Generating many pages without review, experience or usefulness can expand the site’s URL count without increasing authority.

Unmaintained information

Decorative dates do not replace prices, laws, procedures and sources being genuinely reviewed as the topic changes.

Do not optimise a page that should not exist

When several URLs answer the same question, consolidating content and signals into one stronger page is often better than creating more variants.

Sitemaps and internal links should describe the same architecture

The sitemap helps discovery, but links explain hierarchy, relationships and importance. A page present only in the XML file remains weak in the actual experience.

01
One canonical URL
Navigation, sitemap, canonicals and redirects converge on the address that should represent the content.
02
Usable navigation
Important sections are reachable through real links and do not depend on internal search or opaque JavaScript.
03
Hub pages
Categories and archives connect related content and help people and crawlers understand the scope.
04
Sitemap generated from real content
It includes only absolute, public, indexable and canonical URLs rather than becoming a repository for every technical path.
05
Meaningful lastmod
The date changes after a substantial content update, not every time the site runs or its footer changes.
06
No important orphan pages
If a page deserves traffic, it should have a readable role in the website and contextual links from relevant content.

Diagnose indexing in eight checks, in the right order

The method avoids changing URLs, copy and directives at the same time. Each check produces evidence and determines the next step.

01
Define the URL that should appear
Check protocol, host, path, language and trailing slash. Diagnosing a variant rather than the canonical leads to the wrong conclusions.
02
Read status and redirects
Check the final response, chains, intermittent errors and any difference between a browser and an automated request.
03
Check access and directives
Inspect robots.txt, the robots meta tag and X-Robots-Tag in the response that is actually published.
04
Compare every canonical signal
HTML, sitemap, redirects, hreflang and internal links should not indicate incompatible versions.
05
Compare source and rendering
Verify that title, content, links and structured data exist for the crawler rather than only after local interactions.
06
Reconstruct discovery
Find which pages link to the URL, whether the sitemap contains it, and whether duplicate or orphan paths exist.
07
Use URL Inspection and reports
Separate index data from live testing and check the selected canonical, last crawl and reason for exclusion.
08
Fix, request and measure over time
After a verifiable change, you can request a new crawl. Indexing and ranking still require processing and have no guaranteed timing.

The technical standard should make indexing controllable, not mysterious

On the websites I build, technical SEO is not delegated to a plugin. URLs, HTML, status codes, metadata, sitemaps and structured data derive from the same architecture and are tested on the published output.

ONE COHERENT SYSTEM

The website should explain every important URL from the server through to the index

Hand-written code does not guarantee ranking and cannot force Google to index a page. It does allow direct control over what the server returns, reducing layers, dependencies and contradictory configurations.

The same technical foundation is used for my own projects and client work: content and scope change, but the principle of making every signal inspectable and maintainable does not.

01

Semantic HTML is available

Main content, links and hierarchy exist in the initial response; JavaScript adds behaviour without hiding the document.

02

Explicit URLs and routing

Every public page has a stable address, coherent status, canonical and language correspondence when required.

03

Metadata from one source

Titles, descriptions, Open Graph, breadcrumbs and structured data stay aligned with visible content.

04

Generated and verified sitemap

The list derives from real public pages and includes canonical versions, meaningful dates and correct language alternates.

05

Real errors and redirects

Missing pages return 404, permanent moves use 301 and chains are not created out of habit.

06

Testing on the published output

Source, HTTP response, canonicals, hreflang, sitemaps, structured data and search-engine tools are checked as one chain.

For related situations, also explore how to redesign without losing SEO, robots.txt and AI crawlers and how to make a website citable by AI.

Primary sources for understanding crawling and indexing

The guide’s definitions and procedures come from official Google Search and Search Console documentation, not formulas or tools that promise guaranteed indexing.

Google Search Central — How Google Search works
The official stages of crawling, indexing and serving results, including limits and the absence of guarantees.

Open source
Search Console — Page indexing report
States, exclusion reasons and the distinction between URLs that should and should not be indexed.

Open source
Search Console — URL Inspection tool
Index data, live testing, selected canonical, availability and requesting a new crawl.

Open source
Google Search Central — Build and submit a sitemap
Canonical URLs, meaningful lastmod values, limits and the clarification that a sitemap remains a hint.

Open source
Google Search Central — Consolidate duplicate URLs
Redirects, rel=canonical, sitemaps and signal consistency for indicating the preferred version.

Open source
Google Search Central — Robots meta and X-Robots-Tag
Indexing and preview directives that can be read only when the crawler can access the page.

Open source
Google Crawling Infrastructure — HTTP status codes
Effects of 2xx, 3xx, 4xx, 5xx responses and network errors on crawler behaviour.

Open source
Google Search Central — JavaScript SEO basics
Crawling, the rendering queue, initial HTML, links, canonicals and directives in JavaScript applications.

Open source
Google Search Central — Technical requirements
Minimum requirements for indexing eligibility and the clarification that eligibility does not guarantee presence in results.

Open source

Sources reviewed and verified on 1 September 2026. Interfaces, report names and search systems may change: always check the current official documentation.

Frequently asked questions about Google and indexing

Direct answers to situations that are often confused as one single SEO problem.

How long does Google take to index a new website?

There is no guaranteed timeframe. Discovery and crawling can take a few days or much longer depending on the site’s age, links, stability, URL volume and the priority assigned by Google’s systems. A sitemap and indexing request help discovery but do not set a deadline.

Does submitting a sitemap index every page?

No. A sitemap reports preferred URLs and supports discovery, but Google decides which URLs to crawl and index. It should contain only public, canonical pages that genuinely belong in results.

Does requesting indexing repeatedly make the process faster?

It does not fix the cause of an exclusion. After a substantial change, requesting a new check makes sense; repeating the request without changing content, status, canonicals or accessibility does not create a stronger signal.

If a page is indexed, why can I not find it in a search?

Because indexing and ranking are separate. The page may be in the index without being relevant or competitive enough for that query, location, language or context. Search Console shows actual impressions and searches more reliably than one isolated manual test.

Does site:example.com show every indexed page?

No. It is a diagnostic operator rather than an exhaustive inventory. Use URL Inspection for a specific URL and the Page indexing report for the verified property as a whole.

Do robots.txt and noindex do the same thing?

No. robots.txt controls crawling; noindex requests exclusion from results. If crawling is blocked, Google may be unable to read the noindex directive on the page.

Is a page with a 200 status always indexed?

No. A 200 response means the resource is available, but Google may select another canonical, identify a duplicate or soft 404, or choose not to retain it in the index.

Can Google index content loaded with JavaScript?

Google can render JavaScript, but this adds another stage and blocked resources or errors can produce incomplete content. Making main copy, links and signals available in the initial HTML reduces dependencies and helps other crawlers too.

Does a canonical force Google to choose that URL?

No. rel=canonical is a strong signal, but Google also compares redirects, sitemaps, links, content and other elements. Consistent signals increase the likelihood of the intended version being selected.

Can a page appear in AI answers if it is not indexed?

It depends on the system because crawlers and indexes differ. For AI features within Google Search, eligibility depends on Google’s index. In general, an unreachable, unclear or non-indexable page has fewer opportunities to be retrieved and cited.