Skip to main content

Quality Levels

In a task, "quality" means a level. There are four, and a high score on one leaves the others as they were. Before fixing or reporting, ask which level the assessment touches, and assign the work to whoever owns that level.

The number of datasets, a country index and a record score answer different questions. How to calculate shares is in the chapter on metrics. What each known methodology measures is in the chapter on methodologies. How to walk through your catalog in one go is in the chapter on self-audit. This chapter is about how to split an assessment into levels.

Four levels​

LevelQuestionWhat it is usually measured byWho fixes itWhat remains unchecked
Datacan the file or layer be usedformat, completeness, agreement with the source, reproducibility of a calculationthe data ownerrecord completeness, site availability, country index
Descriptionscan the dataset be found and its terms understoodfield completeness, conformance to a profile, a live linkthe publisher and catalog editornumber of rows in the file, country policy
The catalog as a servicedoes it open and do machines receive itavailability, API, dump, licences in the dump, persistent addressesthe platform administratorfile contents, country index
The system of catalogsis there a chain, and do the licence and source travel along itopenness indices, the share of a source's datasets on the upper portal, duplicatesthe open data coordinatora single city file

An empty file with a filled-in record is good descriptions and empty data. A live site without a dump is a good showcase and a weak service. A country with a strong national portal and dead city ones is a strong top and a weak system. The reverse happens too: a city's file and record are in order, but on the upper portal the licence is lost. This is how a dataset from the city of Malmö went in 2024; the chain is analysed in the chapter on federation.

One object, different scores​

The portal looks alive, the files are empty. In February 2023 data.gov.uz had 6,623 datasets, more than most portals in the region. The monitoring section showed almost all datasets as updated. At the service level it looked alive. At the data level it did not: 96% of datasets had fewer than 200 rows, 42.6% had a single row, and the whole portal in CSV took up 40 MB. The records, meanwhile, could have been filled in. A description score sees filled-in fields and leaves empty rows unnoticed: they become visible once the files are opened. How to calculate the distribution by rows, and a second example, data.gov.ru in 2025, are in the chapter on self-audit.

There is a badge, but the calculation does not reproduce. Reproducibility is the ability to obtain the stated result again from the published data and the recorded steps. An "open data" badge on a journal article says the file has been published. In 2021 Hardwicke and co-authors checked 25 articles in Psychological Science with such a badge from 2014–2015. Without help from the authors, the checked figures matched in 9 articles out of 25. After the authors' explanations, in 15 of 25. The main reason for the discrepancy lay in the work itself: the analysis steps were recorded unclearly, although the badge already confirmed the data was published. Whether the calculation matches the article text is checked separately.

A record score looks at the record. data.gouv.fr shows the publisher a metadata quality score: description, files, frequency, licence, open format, territory, period. An empty file with filled-in fields gets a high score. What goes into the score and how it looked on 14 October 2026 is in the chapter on good practices. This is the descriptions level.

What counts as good​

A good answer depends on the catalog type. The six practices without which a dataset cannot be found or used are in the chapter on good practices. Below is how "data" and "descriptions" differ between neighbouring types and where a level breaks first.

TypeData is good whenDescriptions are good whenWhere it usually breaks first
Open data portalthe file opens in a program and has rows and a column dictionarylicence, publisher, period and territory are on the datasetthe counter grows through empty and sliced tables
Geoportalthe layer downloads as a file or through a WFS servicethe layer has a licence, a date and a descriptionthe municipal map: the layer is visible, there is no download button
Research repositorythe calculation reproduces from the files and recorded stepsthere is a DOI, authors, a licence and a citation for an articlethe badge and DOI are there, the analysis steps are not recorded
Indicator catalogthe series continues and can be taken in fullmethodology sits next to the series, codes come from a shared code listthe API responds, but the series has stopped
Microdata catalogthe file can be read using the variable dictionarythe description is open, the access level is namedthe file was closed together with the record
ML catalogthe file downloads, the training and test split is fixedthe record has a licence and provenancethe record is filled in, the licence is more permissive than the source data's
Metadata cataloga vocabulary or code entry is complete and points to the current versionthe entry can be found by code and topicthe record is alive, but the catalog it points to is switched off
Page, Git or hubthe file opens without a login, encoding and columns are statedthere is a README, a licence, schema.org markup, a date and a contactthere is a repository but no page with a persistent address

For microdata, a closed file with an open record is a normal level if the access procedure is written down. The link breaks when the description is hidden too.

Failures are analysed in the chapters on open data portals, geoportals, research repositories, indicators, microdata, ML catalogs and metadata catalogs.

Checking on entry​

Everything above is measured after publication. It can also be done before: the catalog does not accept a dataset that has not passed a check. Then quality is a condition of entry, not an indicator on a dashboard.

This is how neuroscience data archives work. OpenNeuro accepts only datasets following the BIDS standard: a shared order of folders, file names and descriptions. Upload is possible with validator warnings, but not with errors. In DANDI the publish button becomes active only after all errors in the check of descriptions and files have been fixed. A published version does not change.

Checking on entry works where a field has a shared standard. Otherwise it becomes one portal's list of rules, and publishers get around it with empty fields. For tables on an open data portal the analogue is checking a file against a schema on upload. Schemas for common tables are already collected in the schema.data.gouv.fr catalog. The level such a check holds is data. Fixing the file after it is still the data owner's job, but before publication rather than after a complaint.

Other innovations of the same kind are in the chapter on innovative catalogs.

Where to put someone else's score​

Someone else's assessment is first placed on one level. Only then is it decided whether to fix your catalog by it.

AssessmentWhich level it touches
Five starsdata: the format of one dataset
FAIR and F-UJIdescriptions of one research record and the access procedure
MQA, the data.gouv.fr scoredescriptions of catalogs that already send records upward, and whether links are alive
CoreTrustSealthe catalog as an archive: procedures and preservation
Open Data Maturity, ODIN, Global Data Barometerthe system of catalogs and country policy
Number of datasetsa record counter
The registry trust scorea rough view from outside: is the site alive, is a licence stated

MQA sees catalogs that already send descriptions to data.europa.eu and averages the records the assessment has managed to count: this may be the catalog's share in search. A country index is built from policy and national portals. Formulas and the limits of each methodology are in the chapter on methodologies.

In what order to fix​

From the bottom up, following dependencies. First, files that open and can be read. Then the licence and mandatory fields. Then the dump and persistent addresses: without them descriptions will not reach the upper portal and search engines. A country index moves when the lower levels are fixed in many catalogs at once.

A harvesting deadline from above moves the dump to the front of the queue, for the datasets you actually publish. Filling in records for single-row tables by that deadline means fixing descriptions where the data is broken.

The task is set at the level where things are broken. The record editor is given empty fields. The data owner, the file and its freshness. The platform administrator, the dump, the API and addresses. The country coordinator, the rules by which cities send descriptions, and answers for an index such as Open Data Maturity: that score is collected from policy and national portals. A single CSV stays with the data owner.

Example: an agency was told to raise quality​

Context. A regional open data portal. In monitoring, almost all datasets are marked as updated. The country's national index is middling, and a contribution is expected from the region. A user sent in a budget: a single-row table in PDF. The editorial team is two people.

Options. Fill in empty fields in all records. Rewrite the API and dump. Prepare answers for the country index. Open a sample of files, fix the format and content of the datasets people use, then the licence and mandatory fields, then the dump.

Criteria. A person opens the file and uses it without manual fixes. A program takes the descriptions without a login. The work goes to whoever owns the level. Two editors can manage it in a quarter.

Decision. A month goes on the data of the datasets that appear in emails and in download statistics: open format, rows, column dictionary. The next step is the licence and mandatory fields on the same sample, then a dump of all descriptions. The country index is not in the editorial plan.

Why. Filling in all records raises the description score and leaves the PDF in place: it fails the "the file can be used" criterion. A new API over empty files serves the same files faster. Answers for the index are written at country level and do not fix a regional file. A sample with live files resolves the complaint and fits the work of two people.

Consequences. For another quarter the catalog will look more modest than a neighbour with thousands of records. The dump will appear after the files, and until then the federation will not harvest this portal.

When to review. The sample of files opens in a program, and the national portal has named a harvesting date: then the dump becomes the first task. Or there are no more emails about content, but half the records still have empty fields: then the queue moves on to descriptions.

How to write your own decision record is in the chapter "Justifying a Decision".

If there is no portal​

The same four levels exist for a page, a repository and a dataset in a shared hub.

  • Data: the file itself: it opens, the columns are named.
  • Descriptions: README, licence, datapackage.json or schema.org markup.
  • Service: the page opens without a login, and the address does not change when the title is edited.
  • System: a record in a catalog people already search: a national portal, Zenodo, re3data. GitHub search does not replace this record.

The order is the same: first a file that can be used, then a description, then a persistent address, then a record one level up. The minimum for a page is in the chapter on findability without a portal. How a short pass differs from a portal audit is in the section "If there is no portal".

How to set the task​

The task states the level, the scope and who does it.

A good wording looks like this: "On a sample of 30 datasets from different publishers, the file opens as CSV or GeoPackage without manual fixes. Deadline: one month. Done by the data owner, checked by the editor."

The result of a pass is one page from the chapter on self-audit. The queue on it is already split: what the record editor fixes and what requires a change to the dump or address.

Sources​