HTML5 game publishing

Filters and Parameters on Game Lists: Avoiding Crawl Traps

Let visitors filter by device, orientation and sort without creating thousands of near-identical URLs. Clean paths for real pages, canonical, noindex and robots in the right roles.

PlayKovo Team4 min read
Games currently available on PlayKovo. Open a title to see its current listing.

A games list with filters is useful: show only games that work on phones, only portrait, only two-player, sorted by newest. Each choice usually adds a parameter to the URL, and the combinations multiply. Ten genres times three devices times two orientations times four sorts is already 240 URLs for what is one list. A search engine crawler that follows all of them wastes its time on near-identical pages, and may take your useful pages less seriously. This post is about letting visitors filter freely while keeping crawlers on the pages that matter.

How a crawl trap forms

Every filter link is a new URL, such as /games?device=mobile&orientation=portrait&sort=new. A crawler follows links. If every filter option is a link on every filtered page, the number of reachable URLs explodes. Sort orders and page numbers multiply it further. The crawler spends its attention on thousands of pages with the same games in a different order, and may crawl your real new pages less often.

Bing's and Google's guidance both come down to the same point: the pages you want indexed should be easy to reach, and the pages you do not should not waste crawl effort.

Decide which combinations deserve an index entry

Most filter combinations have no search demand. Almost nobody searches for "portrait games sorted by newest". A few combinations do, and those deserve real pages:

  • Genre pages such as "puzzle games" or "racing games". Always indexable, with clean URLs like /games/puzzle.
  • Genre plus a popular attribute, such as "two player racing games" or "puzzle games for phones", if you have the games to fill them and something to say. Make them proper pages with their own text.
  • Everything else (sorts, page numbers, three or more filters) is a convenience for visitors, not a landing page.

Write that decision down as a rule. It is the whole strategy.

Keep the useful pages on clean URLs

Use a path for pages you want indexed and a query string only for transient filters:

/games/puzzle                    indexable, in the sitemap
/games/puzzle/two-player         indexable if it has its own content
/games/puzzle?sort=new           utility, not indexed
/games/puzzle?device=mobile&page=3   utility, not indexed

Only the first two belong in your sitemap. This also means the pages you care about are stable URLs that other sites can link to.

Tell crawlers what to do with utility URLs

Several tools exist, and they do different things:

  1. Canonical tag. On a filtered URL, point the canonical at the clean page (/games/puzzle). That says "this is a variant of that". It is a hint, not a command, but it works well for sorts.
  2. noindex meta tag. On utility pages, <meta name="robots" content="noindex, follow"> keeps them out of the index while letting crawlers follow links to games. Note it must be crawlable for the tag to be seen.
  3. robots.txt Disallow. Stops crawling, and saves crawl effort, but a blocked URL can still appear in results without a snippet if it is linked elsewhere, and a blocked page cannot show a noindex. Use it for obviously useless patterns, such as session ids or sort parameters, not as your only tool.
  4. Link attributes. Mark internal filter links with rel="nofollow" where it makes sense, so you are not inviting crawlers into the maze. It is also a hint.

Do not combine Disallow and noindex on the same URLs and expect both to work. A crawler that cannot fetch the page never sees the tag.

The cleanest approach is to make filtering happen in the browser without changing the crawlable link structure. A client-side filter shows and hides cards using JavaScript, and updates the URL with history.replaceState, so the visitor can share the filtered view, while the HTML the crawler fetches is the plain list. You get a good experience without a thousand crawlable URLs.

function applyFilters(state) {
  cards.forEach(c => c.hidden = !matches(c.dataset, state));
  const q = new URLSearchParams(state).toString();
  history.replaceState(null, '', q ? '?' + q : location.pathname);
}

The feed already contains the fields you need: deviceSupport, screenOrientation, categories and tags. Put them in data- attributes on each card. Filtering the feed by device and orientation shows the values. For the data side of building listing pages, see building category and tag pages from the feed.

Pagination

Long lists need pages. Give each page a clean, crawlable URL (/games/puzzle/page/2), link between them with plain anchor links and make sure page 1 is the canonical entry point (/games/puzzle, not /games/puzzle/page/1). Do not canonicalize every page to page 1, because the games on page 3 would then be hard to discover. Make sure each game is reachable within a few clicks from the home page.

Check what is being crawled

  1. In Google Search Console and Bing Webmaster Tools, look at crawl statistics and indexed pages. A large count of "discovered but not indexed" or "alternate" URLs with parameters is the sign.
  2. Search site:yourdomain.com inurl:sort= to see whether utility URLs are indexed.
  3. Check your server logs for crawler hits on parameter URLs, if you have access.
  4. Review the sitemap: only pages you want indexed should be listed.

The aim is simple: visitors can filter anything, and crawlers see a small, tidy set of strong pages. Our post on hub pages versus category pages describes how to choose which pages that set contains.

We use optional cookies for site analytics and site advertising. You can change your choice later. Privacy policy.