1. FAIR Evaluator tool

Recipe Overview
Reading Time
30 minutes
Executable Code
FAIR Evaluator tool
FAIRPlus logo
Recipe Type
Principal Investigator, Data Manager, Data Scientist

1.1. Ingredients




HTTP1.1 protocol

data communication protocol

guidance on persistent resolvable identifiers


Persistent Uniform Resource Locators - PURL

redirection service

Archival Resource Key

identifier minting service; identifier resolution service

Handle system

identifier minting service; identifier resolution service


identifier minting service

based on Handle system


identifier resolution service

EZID resolution service

identifier resolution service

name2things rsolution service

identifier resolution service


FAIR assessment


FAIR assessment

RDF/Linked Data





1.2. Objectives

  • Perform an automatic assessment of a dataset against the FAIR principles 1 expressed as nanopublications using the FAIREvaluator 2.

  • Obtain human and machine readable reports highlighting strengths and weaknesses with respect to FAIR.

1.3. Step by Step Process

1.3.1. Loading FAIREvaluator web application

Navigate the FAIREvaluator tool, which can be accessed via the following 2 addresses:

the FAIREvaluator Home page

Fig. 1.3 the FAIREvaluator Home page

1.3.2. Understanding the FAIR indicators

In order the run the FAIREvaluator, it is important to understand to notion of FAIR indicators (formerly referred to as FAIR metrics). One may browse the list of currently community defined indicators from the Collections page

Select a 'FAIR Maturity Indicator - Collections'

Fig. 1.4 Select a ‘FAIR Maturity Indicator - Collections’

1.3.3. Preparing the input information

To run an evaluation, the FAIREvaluator needs to following 5 inputs from users:

  1. a collection of FAIR indicators, selected from the list described above.

  2. a globally unique, persistent, resolveable identifier for the resource to be evaluated.

  3. a title for the evaluation. Enforce a naming convention to make future searches easiers as these evaluations are saved.

  4. a person identifier in the form of an ORCID

Running the FAIREvaluator - part 1 - setting the input

Fig. 1.5 Running the FAIREvaluator - part 1: setting the input

1.3.4. Running the FAIREvaluator

Hit the ‘Run Evaluation’ button from ‘https://fairsharing.github.io/FAIR-Evaluator-FrontEnd/#!/collections/new/evaluate’ page

Running the FAIREvaluator - part 2 - execution

Fig. 1.6 Running the FAIREvaluator - part 2: execution

1.3.5. Analysing the FAIREvaluator report

Following execution of the FAIREvaluator, a detail report is generated.

FAIREvaluator report - overall report

Fig. 1.7 FAIREvaluator report - overall report

Time to dig into the details and figure out the reasons why some indicators are reporting a failure:

FAIREvaluator error report

Fig. 1.8 apparently a problem with identifier persistence if using DOI, which are URN rather than URL stricto-sensu

1.4. Conclusion

Using software tool to assess FAIR maturity constitutes an essential activity to ensure processes and capabilities actually deliver and claims can be checked. Furthermore, only automation is able to cope with the scale and volumes of assets to evaluate. The software-based evaluations are repeatable, reproducible and free of bias (other than those that may be related to definitions of the FAIR indicators themselves). These are also more demanding in terms of technical implementation and knowledge. Services such as the FAIRevaluator are essential to gauge improvements of data management services and for helping developers build FAIR services and data.

1.5. Reference


M. D. Wilkinson, M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J. W. Boiten, L. B. da Silva Santos, P. E. Bourne, J. Bouwman, A. J. Brookes, T. Clark, M. Crosas, I. Dillo, O. Dumon, S. Edmunds, C. T. Evelo, R. Finkers, A. Gonzalez-Beltran, A. J. Gray, P. Groth, C. Goble, J. S. Grethe, J. Heringa, P. A. ‘t Hoen, R. Hooft, T. Kuhn, R. Kok, J. Kok, S. J. Lusher, M. E. Martone, A. Mons, A. L. Packer, B. Persson, P. Rocca-Serra, M. Roos, R. van Schaik, S. A. Sansone, E. Schultes, T. Sengstag, T. Slater, G. Strawn, M. A. Swertz, M. Thompson, J. van der Lei, E. van Mulligen, J. Velterop, A. Waagmeester, P. Wittenburg, K. Wolstencroft, J. Zhao, and B. Mons. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data, 3:160018, Mar 2016.


M. D. Wilkinson, M. Dumontier, S. A. Sansone, L. O. Bonino da Silva Santos, M. Prieto, D. Batista, P. McQuilton, T. Kuhn, P. Rocca-Serra, M. Crosas, and E. Schultes. Evaluating FAIR maturity through a scalable, automated, community-governed framework. Sci Data, 6(1):174, 09 2019.

1.6. Authors







University of Oxford

Writing - Original Draft

University of Oxford

Writing - Review & Editing

1.7. License

This page is released under the Creative Commons 4.0 BY license.