Motta Industries Blog

Filters are one of the most useful features a growing online store can offer. A shopper looking for a washable linen cushion cover in a particular size should not have to scroll through every textile product in the catalog. Faceted navigation lets them narrow the list by color, size, material, price, and other attributes in a few clicks.

The same feature can quietly create a technical problem. Every combination of filters can produce a different URL, and a catalog with a modest number of attributes can generate far more addresses than it has products. Search engine crawlers may spend their time requesting those variations instead of the pages that matter. Good faceted navigation SEO is mostly about deciding, deliberately, which of those addresses deserve attention.

This guide explains where the problem comes from, which choices are available, and how a growing store can keep filters helpful for shoppers without creating crawl chaos.

Why a few filters can create a very large number of URLs

Consider a fictional home textiles store with one category of cushion covers. Shoppers can choose one of eight colors, one of six sizes, and one of five materials, or leave any of those filters unselected. That is nine color states, seven size states, and six material states: 378 distinct filtered views for a single category, before adding sorting, price ranges, or pagination.

Add three sort orders and the number triples. Allow shoppers to select several colors at once and it grows much faster. Many of those views will show the same handful of products, or nothing at all. They are useful to the person clicking, but most are not useful as separate pages in search results.

Google described this clearly in its December 2024 summary of faceted navigation guidance. It explained that faceted navigation can generate a near-infinite number of URLs, leading to overcrawling and slower discovery of important new content. It also said faceted navigation is by far the most common source of overcrawl issues site owners report, and that most cases could have been avoided by following best practices.

For a small store, the practical consequence is simple. When a new product or seasonal collection launches, crawlers may take longer to reach it if they are busy with thousands of filter variations.

Decide which filtered views deserve to be pages

Not every filter combination should be treated the same way. Some filtered views match how people actually search and shop. “Linen cushion covers” or “outdoor cushion covers” may deserve a stable, well-described landing page. A view such as large, green, cotton, sorted by newest, with two items, almost certainly does not.

Start with evidence. Look at the terms customers use in site search, the questions staff receive, and the categories that already perform well. Choose a small set of attribute pages that represent genuine demand and that the store can maintain with useful descriptions and a reliable product selection.

Give those chosen pages ordinary, crawlable links from category pages and navigation. They should not depend on a shopper clicking through a particular sequence of filters to exist. Treat the rest of the combinations as tools for shoppers rather than pages to promote.

Motta Industries’ e-commerce strategy work often starts with this decision, because it connects catalog structure to what customers actually look for rather than to every attribute the product database happens to store.

Two broad approaches to the remaining filter URLs

Google’s December 2024 summary outlined two broad options. If filtered URLs do not need to be indexed, prevent crawling, either by disallowing them in robots.txt or by using URL fragments for filters. If they do need to be crawled, make the URLs as well-behaved as possible.

The choice depends on the store. A small catalog where only a few attribute pages matter may block most filter parameters and rely on carefully chosen landing pages. A large marketplace with strong demand for specific combinations may decide some deeper filtered views should be crawlable, accepting the extra server work involved.

Either way, make the decision explicitly and document it. Faceted navigation problems often appear because filters were added one at a time over several years, each reasonable on its own, without anyone reviewing the combined effect.

Understand the difference between query strings and fragments

The two blocking approaches rely on how URLs are structured. RFC 3986, the January 2005 standard for URI syntax, distinguishes the query component, which begins with a question mark, from the fragment, which begins with a hash. It explains that the fragment identifier is separated from the rest of the URI prior to a dereference, so the identifying information in the fragment is handled solely by the user agent.

In practical terms, filters written into a query string, such as ?color=green&size=large, create addresses that a crawler can request from the server as separate resources. Filters written after a hash, such as #color=green, are handled in the browser. Google’s summary noted that search engines generally ignore fragments.

Fragment-based filtering has trade-offs. It typically relies on JavaScript to update the product list, so it needs careful implementation to keep the browser back button, shareable links, and accessibility working properly. It is not automatically the right choice, but it can be a clean option when filtered views do not need to be separate pages.

Use robots.txt carefully and test the rules

For query-based filters that should not be crawled, robots.txt is a common tool. Google’s March 2025 robots.txt refresher described it as a flexible way to tell robots which URL patterns they may or may not crawl, noted that it is broadly supported by crawler operators, and pointed out that many content management systems provide built-in ways to edit the file. It also recommended testing rules to confirm they work as intended.

Pattern rules can be powerful and unforgiving. A rule meant to block a sort parameter can accidentally block the important attribute pages the store chose to keep. Write rules against the specific parameters to exclude, check them against real example URLs, and keep a record of why each rule exists.

Remember what robots.txt does and does not do. It manages crawling. It is not a privacy or security control, and it is not a substitute for deciding which pages the store wants to present. Pair crawl rules with clear internal linking to the pages that should be found.

If filtered pages must be crawlable, make them predictable

When a store decides certain filtered views should remain crawlable, Google’s December 2024 summary listed several practices. Use the standard ampersand separator for parameters. Keep filters in a consistent order, so green-then-large and large-then-green do not produce two different addresses for the same view. Return a 404 status for combinations with no results, and avoid redirecting empty results to a generic “not found” page unless there is no other option.

The summary also mentioned rel=”canonical” as a way to consolidate signals toward a main page, noting that it takes time to be picked up. It described rel=”nofollow” on filter links as something that can discourage crawling but must be applied consistently to every link pointing to those pages, internal and external.

Those details matter for the platform. Some e-commerce systems let developers control parameter order and empty-result behavior; others require custom work. Motta Industries’ e-commerce development team can review how filters are generated and which behaviors can be changed without breaking the shopper’s experience.

Treat sorting and other parameters as their own question

Filters are not the only source of extra URLs. Sort orders, items-per-page settings, view toggles, and campaign tracking parameters can all create new addresses that show the same products in a different arrangement. They rarely deserve separate attention from search engines.

List every parameter the store generates and what it does. Then decide, parameter by parameter, whether it changes what the page is about or only how it is displayed. A material filter may define a meaningful selection; a sort order usually does not. That inventory makes robots.txt rules, canonical choices, and internal linking decisions far easier to reason about, and it gives future developers a reference before they add another option.

Consider internal search results pages as well. A site search box can generate unlimited addresses from whatever shoppers type. Those pages help visitors right away, but they are rarely good destinations from search engines and deserve the same deliberate decision.

Keep the shopper experience at the center

Technical control should not make the filters worse for customers. Shoppers still need fast, accurate results, visible applied filters, an easy way to remove them, and sensible messages when a combination has no products.

Check filters on phones as well as desktops. Long filter panels can push products far down the screen. Product counts beside each option help shoppers avoid dead ends. Clear labels for sizes and materials reduce mistaken selections and, eventually, returns.

Product data quality underpins everything. If one product is tagged “navy” and another “dark blue,” the color filter will mislead customers no matter how carefully the URLs are managed. Consistent attribute values make both shopping and crawl planning easier.

Watch the effects after changes

Faceted navigation changes can have broad effects, so introduce them carefully. Test in a staging environment where possible, keep a dated change log, and confirm that the chosen landing pages, product pages, and categories remain reachable through ordinary links.

Afterward, review server logs or crawl reports for signs that crawlers are spending less time on filter variations and reaching new products. Watch whether important pages continue to appear in search, and whether site search and conversion behavior change. Avoid attributing every later movement to the filter change; seasonality, new products, and other updates also affect results.

If shoppers report that filters stopped working, treat that as urgent. A crawl improvement that damages the buying experience is not a success.

A structure that can grow with the catalog

Faceted navigation is worth getting right because it serves shoppers well. The aim is not to remove filters, but to give the store a clear model: a small number of intentional landing pages, predictable rules for everything else, and a review whenever new attributes are added.

Assign ownership for that review. When merchandisers add a new attribute, such as pattern or care method, someone should ask whether it will appear as a filter, whether it creates new URL parameters, and whether any of its values deserve landing pages. A short checklist in the product team’s routine prevents the slow drift that created the problem.

If your catalog is growing and you are unsure how filters, URLs, and crawling interact on your platform, Motta Industries’ e-commerce solutions can help review the current setup, identify which filtered views deserve to be pages, and plan changes that protect both discovery and the customer’s ability to find the right product quickly.