Skip to main content

Discoverability

A dataset is published when it can be found without knowing the portal's address by heart. Otherwise the catalog is an internal drive with a pretty cover.

This chapter is about the catalog as a whole: what the platform must do by itself for hundreds and thousands of pages, where to register the catalog, and how to check that search engines see it. How to make a single page or a repository findable without a portal is in the chapter "Making a Publication Findable".

Search engines​

Google Dataset Search and similar systems read schema.org/Dataset markup on the dataset page. Next to it you need a sitemap, through which the search engine walks all dataset pages: according to the dataportals-registry export (release 1.27.0 of 15 October 2026) this is the most common machine address of catalogs, about 7,000 records. Titles and descriptions are written in human language. A dataset called "Export 12" cannot be found.

On a portal, markup and the sitemap are not written by hand: the platform must produce them for every page.

Platformschema.org markup on the dataset pageSitemap
CKANthe ckanext-dcat extension, structured data modulenot in the core; third-party ckanext-sitemap extensions, not maintained by the CKAN community, the most recent updated in 2024
Dataversein the corein the core: generated by an administrator command, a large catalog is split into several files
InvenioRDMin the corein the core since v13: generated and updated automatically, listed in robots.txt
GeoNetworkin the core: schema.org JSON-LD on the record page; Croissant fields for the ISO 19115-3 schema are in the branch of the upcoming 4.4.13 release, not in 4.4.12in the core, but the robots.txt linking to it sits inside the GeoNetwork address, not at the site root; the sitemap is submitted to search engines by hand
Cloud servicescheck the dataset page: markup is sometimes trimmedcheck the /sitemap.xml address

Google advises putting markup on one canonical dataset page, and on copies (for example in search results or on a portal that harvested the dataset from its source) linking to it with the sameAs property. This matters for federation: if a regional portal repeats a city's record with full markup and no link to the original, the search engine sees two different datasets.

What a search engine does not see​

A search robot reads an HTML page and follows links. Anything that cannot be reached that way does not exist for it, even if a person sees it easily in a browser.

An interface drawn by a script. The page arrives as an empty shell, and a script loads the list of datasets and descriptions from an API. A robot that does not run scripts sees an empty page; so does a web archive. This is how the "Compendia" section of the Rosstat website works: the page HTML contains not a single link to a document, and the list is loaded by a separate request (this is how it looked in the web archive in February 2026). Rosstat documents are still found in search, but mostly because other sites link to them. The dataset page on data.europa.eu is built the same way: the source HTML has neither a description nor markup, and everything is drawn by a script (October 2026). The archive of Accounts Chamber bulletins serves documents through a search API, and robots never reach the PDFs themselves; this case is analysed in the chapter on archiving.

The cure is not to abandon the API but to give every dataset an ordinary page: the server returns HTML straight away with the description, file links and markup, and dataset lists are paged with ordinary links. The API is in addition to pages, not instead of them.

A site closed to foreign addresses. Search engines, web archives, aggregators and AI agents come from cloud data centres in various countries. If a site does not respond to requests from abroad, it drops out of all of them at once. On 12 October 2026 neither rosstat.gov.ru nor ach.gov.ru opened from a foreign server. What this means for a catalog and how to check it is analysed in the anti-pattern "The door is closed to programs" and in the chapter on archiving.

A sitemap the search engine does not know about. A search engine looks for robots.txt only at the site root. GeoNetwork puts its robots.txt with a sitemap link inside its own address, for example /geonetwork/srv/api/robots.txt, and the documentation explicitly advises registering the sitemap with search engines by hand. In October 2026 geodata.nz, geocat.ch and the Estonian geodata catalog had no robots.txt at the root. GeoNetwork 4.4 serves schema.org markup on record pages, but without a sitemap a robot may never reach those pages. So GeoNetwork catalogs are found mainly through aggregators (data.europa.eu, which replaced the INSPIRE geoportal from 1 July 2026, and national geocatalogs), not through ordinary search.

Spam has markup too​

schema.org/Dataset markup is added by the page owner, and nobody checks it. The Web Data Commons project extracts such markup from the open Common Crawl web index. In the 2024 release, the Dataset type was found on 694,000 pages from about 2,000 sites. By I. Begtin's manual estimate (June 2024), of 1.4 million objects marked up this way, about a third are junk (for example, ordinary reference site articles marked up as datasets), a third are data vendors' pages, and only a third are data that can be downloaded.

The DataCatalog type, used to mark up a whole catalog, is so rare that Web Data Commons did not break it out as a separate subset in either 2023 or 2024. A check of the catalog registry in 2024 found such markup on about one catalog in a thousand.

Large portals mark up differently too. In 2024 data.gov served trimmed markup: the name and keywords. In October 2026, on the new catalog.data.gov, dataset markup includes both the licence and a list of files. data.europa.eu has no markup in its HTML at all.

The practical conclusion: markup by itself does not distinguish your dataset from spam. Fill in what spam usually lacks: the licence, the publisher, a list of files with direct links (distribution with contentUrl) and a link to the catalog (includedInDataCatalog), as in the example in Google's documentation.

Aggregators and registries​

A higher-level portal will find you only through a dump you have agreed on. How this works and how to get harvested is analysed in the chapter "Federation and Harvesting".

Independent aggregators and dataset search engines (Dateno, find.data.gov.scot, research aggregators) do not crawl the whole internet either. They keep a list of sources and maintain a harvester for each platform type. Hence two reliable ways to reach them:

  • Publish where harvesting already happens. A national or agency portal, Zenodo, a domain repository: their harvesters are already set up, and a new dataset reaches aggregators without any effort on your part.
  • Install standard software with a known interface. CKAN serves its API and DCAT, DKAN a data.json file, JKAN a common data.json, InvenioRDM a REST API and OAI-PMH. Harvesters for them are already written; it is enough to report the catalog's address. A custom-built portal with its own interface needs a separate harvester, and an aggregator gets to it last or not at all.

How much this matters can be seen in a Russian example. In 2024, of 154 Russian catalogs in the Dateno registry, about 15 were indexed. The reasons given were: government sites are closed to foreign addresses, lists under the methodological recommendations are awkward to harvest, and 40–50 custom-built portals each need their own harvester. Finland, where catalogs run on standard software, was much easier to harvest.

In addition, the catalog is registered where its audience looks for the list of portals itself:

  • the national registry or the country's higher-level portal;
  • re3data for a research repository: the repository itself submits a request through the site form, and after review a record appears with subjects, conditions and certificates;
  • FAIRsharing for a database or a standard;
  • the dataportals-registry, which describes catalogs for those looking for a portal rather than a dataset.

Registration does not replace a dump. A registry entry with a broken address is worse than no entry: update it after the portal moves.

Programs​

A program needs a stable API or a file with all descriptions, and documentation without a captcha. An llms.txt file is a short note for a language model about what is on the site; it is a proposed format, not an adopted standard. MCP is a way to give a program the list of a catalog's queries. Both make sense when such programs already use the catalog. They do not fix the absence of DCAT. More in the chapter on trends.

How to check that the catalog is found​

Substitute the portal's address and the address of one dataset page:

PORTAL=https://portal.example
PAGE=https://portal.example/dataset/budget-2026

# the sitemap responds and is listed in robots.txt
curl -s "$PORTAL/robots.txt" | grep -i sitemap
curl -s -o /dev/null -w "%{http_code}\n" "$PORTAL/sitemap.xml"

# the dataset page has Dataset markup
curl -s "$PAGE" | grep -o '"@type": *"Dataset"' | head -1

Then three steps that a command cannot check:

  1. Run the dataset page through Google's Rich Results Test: it will show errors in the Dataset markup.
  2. Add the site to Google Search Console and submit the sitemap. It shows how many pages the search engine has indexed and why it skipped the rest.
  3. A few weeks later, search for five to ten datasets by title in Google Dataset Search. If copies from aggregators come up rather than your pages, check sameAs and that the dataset page opens without a login.

A short order of work​

The address, schema.org, the sitemap and a dump in the right profile are already in the requirements checklist. Here they are joined by registering the catalog in a relevant registry, submitting the sitemap to Search Console and a check a month later: the dataset is found by search, the dump opens.

Sources​