--- author: - affiliation: - id: https://ror.org/027bh9e22 name: Leiden University contributor_roles: [] family: Waltman given: Ludo url: https://orcid.org/0000-0001-8249-1752 - affiliation: - id: https://ror.org/027bh9e22 name: Leiden University contributor_roles: [] family: van Eck given: Nees Jan url: https://orcid.org/0000-0001-8448-4521 - contributor_roles: [] family: Visser given: Martijn - contributor_roles: [] family: Neijssel given: Mark url: https://orcid.org/0000-0001-6252-8503 - contributor_roles: [] family: Montgomery given: Lucy - affiliation: - id: https://ror.org/02n415q13 name: Curtin University contributor_roles: [] family: Neylon given: Cameron url: https://orcid.org/0000-0002-0068-716X - contributor_roles: [] family: Kramer given: Bianca - contributor_roles: [] family: Demes given: Kyle - contributor_roles: [] family: Priem given: Jason blog: authors: - name: Leiden Madtrics community_id: d8304840-75c2-4164-bc37-ec879c4f065b created: 1780876800 current_feed_url: null description: Leiden Madtrics doi: https://doi.org/10.59350/leidenmadtrics favicon: https://rogue-scholar.org/api/communities/d8304840-75c2-4164-bc37-ec879c4f065b/logo feed_format: application/atom+xml feed_url: https://www.leidenmadtrics.nl/atom/ filter: null generator: Other home_page_url: https://www.leidenmadtrics.nl/ issn: null language: eng license: https://creativecommons.org/licenses/by/4.0/legalcode prefix: '10.59350' relative_url: null secure: null slug: leidenmadtrics status: active subfield: '1804' title: Leiden Madtrics updated: 1784193807 use_api: null container: Leiden Madtrics date: '2024-01-30T00:00:00+00:00' date_updated: '2026-01-25T05:39:52+00:00' guid: https://www.leidenmadtrics.nl/articles/introducing-the-leiden-ranking-open-edition identifier: https://doi.org/10.59350/89wpz-hpz32 images: - height: '351' src: https://lh7-us.googleusercontent.com/tHy1PdjuQuLYS4xgQBtRL_vGkbfkfjP4MJxc8PDR3Ji_uHkaxSIWvR7okJ4aaxcoJnhj37sf_6OHphInlgRJfO7JhR57zNxHKxcAFA7Kx0ccdCupskFQSIHakH1Yqga5rMfwUGxn3retA5GrROG_PZQ width: '624' - height: '239' src: https://lh7-us.googleusercontent.com/VjpSp7dpbdBJKP1aRBdINJ4IzoRdB0W9i1Nyb8ZTbyiT_avm6-892DopQpVUKpwoInEXPBI-NVW7rrW9y-5yKLSlIpv7S68l6xfrYUn87rwcPpM0oHcsi8IeWgyjFEUO5H8UqirPzjgMqpbF4Pcec9A width: '624' - height: '445' src: https://lh7-us.googleusercontent.com/GNdAIHrl957GDdRDrD6fzTA8W-MHOBg_HeLyxOv9KpzoXVTVH0BuYhlkOWefvSzX038JqIOoOe9f_qy4WI-BUWTCXt5D0Ji7XzriBYYKWCjLz0CNxZuj_oeIJwWPtW6NHveE8SlaW2Zg8PgGB4rZOF8 width: '624' - height: '448' src: https://lh7-us.googleusercontent.com/HNyLmI0XobkmgZQ_p5JsZqeGVOanxWxUpDeNJ0oMCwX6o2FXx9hgsOKbSpU4k-YRAWn_XSX0fyW7mxiT4smZeTxgpTCY8dmCGK1yoteJz50vB8Ylk8x54_uuxCQcdVXJcPHpWGhE6I3RgBglND9-hAM width: '624' issn: null lang: en license: https://creativecommons.org/licenses/by/4.0/legalcode rid: 0kjpv-99945 rights: https://creativecommons.org/licenses/by/4.0/legalcode summary: When the Shanghai Ranking, also known as the Academic Ranking of World Universities (ARWU), was launched in 2003, Ton van Raan, director of CWTS at the time, sounded the alarm about the problematic way in which the ranking uses bibliometric data, for instance in attributing publications to universities. title: Introducing the Leiden Ranking Open Edition url: https://www.leidenmadtrics.nl/articles/introducing-the-leiden-ranking-open-edition version: v1 --- When the Shanghai Ranking, also known as the Academic Ranking of World Universities (ARWU), was launched in 2003, Ton van Raan, director of CWTS at the time, sounded the alarm about the [problematic way in which the ranking uses bibliometric data](https://doi.org/10.1007/s11192-005-0008-6){target="_blank"}, for instance in attributing publications to universities. In response to the Shanghai Ranking, CWTS decided to launch the [Leiden Ranking](https://www.leidenranking.com/){target="_blank"}, aiming to demonstrate more appropriate ways to use bibliometric data for comparing universities. While the Leiden Ranking did not gain the same visibility as the Shanghai, THE, and QS rankings, it developed a strong reputation for offering a robust, high-quality approach for comparing universities in terms of bibliometric parameters. Today, after almost two decades, the Leiden Ranking is going to take an ambitious next step in improving bibliometric approaches for comparing universities. The [Open Edition of the Leiden Ranking](https://open.leidenranking.com/){target="_blank"}, launched today by CWTS, addresses one of the most challenging problems of bibliometric indicators: the lack of transparency of these indicators due to their dependence on proprietary data. Together with the [Curtin Open Knowledge Initiative (COKI)](https://openknowledge.community/){target="_blank"}, [Sesame Open Science](https://bmkramer.github.io/SesameOpenScience_site/){target="_blank"}, and [OurResearch](https://ourresearch.org/){target="_blank"}, CWTS has rebuilt the Leiden Ranking, making it fully transparent by working exclusively with open data and open source algorithms. Until now, the Leiden Ranking has always been based on data from [Web of Science](https://webofscience.com/){target="_blank"}, a proprietary data source owned by Clarivate. The Leiden Ranking Open Edition uses data from [OpenAlex](https://openalex.org/){target="_blank"}, a fully open data source created by OurResearch. To the best of our knowledge, the Leiden Ranking Open Edition is the first fully transparent university ranking. The importance of transparency of university rankings is widely recognized. CWTS emphasizes the need to be transparent in [its ten rules for ranking universities](https://www.researchresearch.com/news/article/?articleId=1368350){target="_blank"}. Transparency also features prominently in the [criteria for fair and responsible university rankings](https://inorms.net/wp-content/uploads/2022/07/principles-for-fair-and-responsible-university-assessment-v5.pdf){target="_blank"} developed by the International Network of Research Management Societies (INORMS). These criteria were used in the [rethinking the rankings initiative](https://arma.ac.uk/rethinking-the-rankings/){target="_blank"}. Likewise, transparency and openness are key elements in the [strategy for culture change regarding university rankings](https://www.universiteitenvannederland.nl/en/ranking-the-university){target="_blank"} that is currently being implemented in the Netherlands. More in general, the [UNESCO Recommendation on Open Science](https://doi.org/10.54677/MNMH8546){target="_blank"} highlights the need for \"open bibliometrics and scientometrics systems\". In a similar vein, the [Agreement on Reforming Research Assessment](https://coara.eu/agreement/the-agreement-full-text/){target="_blank"} stresses the importance of \"independence and transparency of the data, infrastructure and criteria necessary for research assessment\". ### **How did we create the Leiden Ranking Open Edition?** The Leiden Ranking Open Edition aims to reproduce the traditional Leiden Ranking as closely as possible, focusing on the [Leiden Ranking 2023](https://www.leidenmadtrics.nl/articles/the-cwts-leiden-ranking-2023){target="_blank"} that was published in June last year. The Open Edition includes the same 1411 universities that are also included in the Leiden Ranking 2023. Instead of proprietary data from Web of Science, the Open Edition uses open data from OpenAlex. This data is harvested by OpenAlex from a [variety of sources](https://help.openalex.org/how-it-works/entities-overview){target="_blank"}, including [Crossref](https://www.crossref.org/){target="_blank"}, [PubMed](https://pubmed.ncbi.nlm.nih.gov/){target="_blank"}, and the websites of publishers. The Open Edition is based on the [OpenAlex snapshot](https://docs.openalex.org/download-all-data/openalex-snapshot){target="_blank"} released on November 21, 2023. Figure 1 summarizes the process of creating the Leiden Ranking Open Edition. We now discuss the different steps in this process.
Figure 1. Summary of the process of creating the Leiden Ranking Open Edition.
\ **Core publications** Web of Science and OpenAlex differ in their coverage of the scientific literature. Most publications covered by Web of Science are also covered by OpenAlex, but OpenAlex covers many publications that Web of Science does not cover. The traditional Leiden Ranking focuses on so-called core publications, which are identified based on a [number of criteria](https://www.leidenranking.com/information/indicators#publications){target="_blank"}. The Open Edition also has a focus on core publications, identified based on [similar criteria](https://open.leidenranking.com/information/indicators#publications){target="_blank"}. However, because of coverage differences between Web of Science and OpenAlex, the set of core publications considered in the Open Edition differs from the set of core publications considered in the traditional Leiden Ranking. This is discussed in more detail in [this blog post](https://www.leidenmadtrics.nl/articles/opening-up-the-cwts-leiden-ranking-toward-a-decentralized-and-open-model-for-data-curation){target="_blank"}. **Publication classification** The Leiden Ranking relies on a detailed algorithmic classification of publications into research areas. This classification is used to assign publications to main fields and to calculate normalized citation impact indicators. For the Open Edition, CWTS created a new classification of publications based on OpenAlex data. [This blog post](https://www.leidenmadtrics.nl/articles/an-open-approach-for-classifying-research-publications){target="_blank"} provides more details. **CWTS organization registry** OpenAlex uses [ROR (Research Organization Registry) identifiers](https://ror.org/){target="_blank"} for organizations, while the Leiden Ranking has traditionally used internal CWTS organization identifiers. For the Open Edition, CWTS created a new organization registry by matching CWTS organization identifiers to ROR identifiers. This was done for the 1411 universities included in the Open Edition and for the [affiliated organizations linked to these universities](https://www.leidenmadtrics.nl/articles/a-team-a-stands-for-very-challenging-job){target="_blank"}. Additional links between universities and affiliated organizations were obtained from the ROR organization registry. Links between universities and their affiliated organizations are presented in a transparent way on the website of the Open Edition (see Figure 2).\
Figure 2. Screenshot of the website of the Leiden Ranking Open Edition showing the links between University of Amsterdam and its affiliated organisations.
**\ Publication-university links** OpenAlex provides ROR identifiers for the affiliations of authors of publications. We linked publications to universities by connecting ROR identifiers obtained from OpenAlex to the ROR identifiers of universities and their affiliated organizations in the CWTS organization registry. In [this blog post](https://www.leidenmadtrics.nl/articles/opening-up-the-cwts-leiden-ranking-toward-a-decentralized-and-open-model-for-data-curation){target="_blank"} the CWTS team presents a comparison between the approach for linking publications to universities used in the traditional Leiden Ranking and the approach used in the Open Edition. **Indicators** The Open Edition includes the same bibliometric indicators as the traditional Leiden Ranking, except for [indicators of gender diversity](https://www.leidenranking.com/information/indicators#gender-indicators){target="_blank"}. These indicators are somewhat more challenging to produce using OpenAlex data because OpenAlex does not make a distinction between first and last names of authors. This distinction is important for algorithmic gender inference. At the moment gender diversity indicators are therefore not included in the Open Edition. We may add them in the future. ### **How to access the data and software** The data used to create the Leiden Ranking Open Edition is [openly available in Zenodo](https://doi.org/10.5281/zenodo.10579113){target="_blank"} under a [CC0 public domain dedication](https://creativecommons.org/public-domain/cc0/){target="_blank"}. The data can also be [accessed through Google BigQuery](https://console.cloud.google.com/bigquery?ws=!1m4!1m3!3m2!1scwts-leiden!2sleiden_ranking_open_edition_2023){target="_blank"} (to run queries you need to have a BigQuery account and your own project). Universities may for instance use the data to check whether publications have been correctly attributed to them. Performing such checks is not possible in the case of the traditional Leiden Ranking, since this ranking is based on proprietary data that cannot be shared openly. The source code of the software used to create the Leiden Ranking Open Edition is [openly available in GitHub](https://github.com/CWTSLeiden/CWTS-Leiden-Ranking-Open-Edition){target="_blank"} under an MIT license. A Microsoft SQL Server database system was used to perform the data processing and the calculations. The source code therefore consists mostly of SQL scripts.**\** ### **How to compare the Open Edition with the traditional Leiden Ranking** To what extent does the Open Edition provide results that are similar to those provided by the traditional Leiden Ranking? To address this question, we have created an [interactive dashboard](https://app.powerbi.com/view?r=eyJrIjoiODg0YWM2MzEtMmVlNS00NzEyLWE2YjMtMTIxM2Y3ZjMxYTJkIiwidCI6ImNhMmE3Zjc2LWRiZDctNGVjMC05MTA4LTZiM2Q1MjRmYjdjOCIsImMiOjh9){target="_blank"} in which the values of the indicators in the two editions of the Leiden Ranking can be easily compared. The interactive dashboard can for instance be used to explore the correlation between the number of publications of a university in the traditional Leiden Ranking and in the Open Edition (see Figure 3), or the correlation between a university\'s proportion of highly cited publications in the two editions of the Leiden Ranking (see Figure 4).\ \
Figure 3. Correlation between the number of publications of a university in the traditional Leiden Ranking and in the Open Edition (fractional counting).
\
Figure 4. Correlation between the proportion of highly cited (top 10%) publications of a university in the traditional Leiden Ranking and in the Open Edition (fractional counting).
\ In general, we consider the correlations between indicator values in the two editions of the Leiden Ranking to be quite strong. However, for some universities there are large differences between the two editions of the Leiden Ranking. In many cases this is probably due to differences in the publications attributed to a university. This issue is analyzed in more detail in [this blog post](https://www.leidenmadtrics.nl/articles/opening-up-the-cwts-leiden-ranking-toward-a-decentralized-and-open-model-for-data-curation){target="_blank"}. ### **What is next?** At the moment the Leiden Ranking Open Edition is still of an experimental nature. Building the ranking was an incredibly useful exercise for our organizations (CWTS, COKI, Sesame Open Science, and OurResearch). We also expect to learn a lot from the feedback we hope to receive from users of the Open Edition. Based on the lessons learned, we will make further improvements to the Open Edition. Within one or two years, we expect the Open Edition to be fully mature and to offer a full replacement for the traditional Leiden Ranking. While our organisations are making further improvements to the Leiden Ranking Open Edition, CWTS will continue to release annual updates of the traditional Leiden Ranking based on Web of Science data. In the next one or two years, we expect the traditional Leiden Ranking and the Open Edition to co-exist. In the somewhat longer term, CWTS will make a [full transition to open research information](https://www.leidenmadtrics.nl/articles/toward-open-research-information-introducing-the-information-openness-focal-area-at-cwts){target="_blank"}. Within the next few years, all bibliometric indicators produced by CWTS, including those in the Leiden Ranking, will be based on open data. The transition of the Leiden Ranking toward open data fits into broader discussions about openness of research information, responsible use of bibliometric indicators, and reform of research assessment. Critical reflection on the value of university rankings should be part of these important discussions.\ ### **How to get in touch**\ {#how-to-get-in-touch} Would you like to know more about the Leiden Ranking Open Edition? CWTS is organizing two webinars about the Open Edition, providing an opportunity to discuss the ranking with members of the CWTS team. The [first webinar](https://universiteitleiden.zoom.us/meeting/register/u5cuce-vrD0iGNZUZREryhJ0Ogvh7zaZOILs#/registration){target="_blank"} takes place on February 6 at 10h CET, and the [second webinar](https://universiteitleiden.zoom.us/meeting/register/u50ode-sqDgrHd2RGWtphvap5qPoZ5bB2ord#/registration){target="_blank"} on February 9 at 15h CET. COKI, in collaboration with Future Campus, is offering a [webinar with an Australian focus](https://futurecampus.com.au/event/your-university-ranked/){target="_blank"} on January 31 at 11h AEDT. OpenAlex is organising a [webinar about institutional profiles and affiliation curation](https://us02web.zoom.us/webinar/register/WN_ggI4M2mwQF-Ce-pp947nUA){target="_blank"} on February 8 at 12h EST. We also invite you to reach out to us using the [contact form](https://open.leidenranking.com/contact){target="_blank"} on the website of the Leiden Ranking Open Edition. We appreciate your comments and feedback on the Open Edition. Do you want to start working yourself with open bibliometric data sources such as OpenAlex? You may then consider signing up for the [course *Scientometrics Using Open Data*](https://www.cwts.nl/education/cwts-course-program/scientometrics-using-open-data){target="_blank"} organised by CWTS, COKI, and Sesame Open Science. The next edition of this online course takes place from March 25 to March 28.\ [Header image from ]{.caption dir="ltr"}