![\"mountain](\"https://tzovar.as/assets/images/2026-10-17-git-pages.jpg\"/)
\nAt the end of the year, Codeberg will shut down its old Codeberg Pages server, in favor of the already deployed, git-pages-based alternative.1.\nFor many deployments, migration will be quite straight-forward, requiring little more than using a webhook, as outlined in the migration guide.\nAnd this page has already for a few months now been served by the git-pages server, without any problems.
There is only one little but for the migration path, which affects pages that so far relied on git lfs for storing larger asset files like images and that are trying to directly serve from a repository without a build stage:\ngit-pages does not (and probably never will) support git lfs.\nI ran into this problem (without noticing it) when I migrated my talks, which are all just HTML files generated locally by marpit, and for which I like keeping the images in the lfs. \nWhen I was preparing a new slide deck for a conference, I noticed that the images didn't show up when I wanted to share a draft with a friend for review.
Luckily, the at least temporary hot fix for that is quite easy:\nWhile git-pages doesn't support git lfs when serving directly from a repository via the webhook, one can use a CI action and the git-pages-cli to push a total folder. \nWhile normally more used for deploying the output of a static site generator (like this page), it can also be paired with a simple checkout action to grab a copy of the repository including the lfs objects and then push that whole folder via the git-pages-cli.
The simplest way of doing that is via writing a small Forgejo Action that triggers on a push, and is how I did it for my talks:
\nname: Publish\non:\n push:\n branches: [pages]\njobs:\n publish:\n runs-on: codeberg-tiny\n steps:\n - uses: actions/checkout@v7\n with:\n lfs: true\n - uses: actions/git-pages@v2\n with:\n site: https://talks.tzovar.as/\n server: codeberg.page\n token: $\n source: .\nAfter the checkout of the repository, I let it just immediately use the git-pages action to deploy the folder as is.\nIf for some reason you prefer doing the same with the Woodpecker CI, I made an equivalent plugin that can be used for that.
And of course if your pages already use a CI action to deploy generated artifacts, you'll not have to worry about this either way.\nIn the long term I should probably also just let the CI also build the slides, to ensure that the deployed HTML version reflects the actually committed files, but that's a modification for a different day.
\nIf you're not sure what method you are using, this small tool lets you check\u00a0\u21a9
\nThe LIS-Bibliometrics Committee is inviting expressions of interest from UK institutions interested in hosting the 2027 conference, as well as colleagues who would like to join the organising committee.
\n\n\n\nThe proposed conference dates are between mid-June and mid-July, or in early September 2027. Hosting or helping to organise the event offers an opportunity to connect with colleagues and contribute to the bibliometrics community.
\n\n\n\nThe deadline for expressions of interest is 6 November 2026.
\n\n\n\nWe would love to hear from you and welcome your involvement in making the next conference a success.
\n\n\n\nFind out more about the conference and how to get involved.
\n\n\n\nContact us: https://lisbibliometrics.wordpress.com/contact/
","doi":"https://doi.org/10.59350/g5rq3-kfd25","guid":"https://thebibliomagician.wordpress.com/?p=5687","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1792195200,"rid":"y4844-v1g52","summary":"Help shape the LIS Bibliometrics Conference 2027 The LIS-Bibliometrics Committee is inviting expressions of interest from UK institutions interested in hosting the 2027 conference, as well as colleagues who would like to join the organising committee. The proposed conference dates are between mid-June and mid-July, or in early September 2027.","tags":["Metrics"],"title":"LIS Bibliometrics Conference 2027 \u2013 Call for Expressions of Interest!","updated_at":1792248905,"url":"https://thebibliomagician.wordpress.com/2026/10/17/lis-bibliometrics-conference-2027-call-for-expressions-of-interest/","version":"v1"}},{"document":{"authors":[{"affiliation":[{"name":"Durham University, Department of Psychology"}],"contributor_roles":[],"family":"Rubin","given":"Mark","url":"https://orcid.org/0000-0002-6483-8561"}],"blog":{"authors":[{"name":"Mark Rubin","url":"https://orcid.org/0000-0002-6483-8561"}],"community_id":"3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6","created":1681948800,"current_feed_url":null,"description":"Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.","doi":"https://doi.org/10.59350/markrubin","favicon":"https://rogue-scholar.org/api/communities/3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6/logo","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"markrubin","status":"active","subfield":"1207","title":"Critical Metascience","updated":1792151011,"use_api":true},"blog_name":"Critical Metascience","blog_slug":"markrubin","content_html":"I really enjoyed Mel Andrews' recent essay: \"Philosophy in the Trenches and Laboratory Benches of Science\" and its main point that \"every laboratory needs a philosopher.\" This point is made in the context of a concern about \"the industrialisation of science\":
Not all of the shifts of recent history have constituted improvements to science, in either its truth-seeking capacity or its social responsibility. I allude to the industrialisation of science. To the corporatisation, the commodification, and the militarisation of science\u2026.
The so-called replication crisis has come about, in large part, because generations of behavioural scientists have been instructed in assembly-line procedures rather than modes of questioning and critical analysis.
This concern about the industrialisation of science reminded me of something that statistician Ronald Fisher wrote many years ago when he argued that the Neyman-Pearson approach to hypothesis testing is more suitable for \"acceptance procedures\" used during quality control on industrial production lines than for scientific research:
I am casting no contempt on acceptance procedures, and I am thankful, whenever I travel by air, that the high level of precision can really be achieved by such means. But the logical differences between such an operation and the work of scientific discovery by physical or biological experimentation seem to me so wide that the analogy between them is not helpful, and the identification of the two sorts of operation is decidedly misleading (Fisher, 1955, pp. 69\u201370).
As I've discussed elsewhere, the production line analogy assumes an inappropriate amount of knowledge and certainty on the part of scientists (Rubin, 2020). Unlike quality controllers on a factory production line, scientists don't know the relevant and irrelevant features of their populations and testing conditions. Factors that they theorize to be essential for demonstrating an effect may turn out to be unimportant, and factors that they assume to be unimportant may turn out to be essential! Indeed, one of the key differences between scientific research and quality control is that scientists can learn what's important and unimportant from their replication failures, whereas quality controllers merely consign their \"bad batches\" to the garbage. Hence, like Andrews, I see a clear link between the replication crisis and the industrialisation of science (e.g., Rubin, 2021, p. 5829).
Andrews also describes the industrialisation of science as a:
broad-sweeping prohibition against \"theorising\" \u2013 and by this it is meant any and all forms of sustained critical thought about or questioning of the scientific work
Again, this description rings true for me. In particular, it reminds me of Kerry Chamberlain's (2000) description of \"methodolatory,\" which consists of:
An overemphasis on locating the 'correct' or 'proper' methods
A focus on description at the expense of interpretation
The avoidance of theory, and
Avoidance of the critical
As Chamberlain explained \"methodolatry\u2026works to prevent us looking at the assumptions behind our research.\" Andrews' proposal of a philosopher-in-the-lab should provide an important safeguard against methodolatory:
Every laboratory needs to have someone who understands broadly and intimately what is going on and who is empowered to ask questions, to stand back from the laboratory bench or the telescope or the command line and ask: What are we really trying to determine here, and to what end? Will it be of benefit or detriment to mankind? Are the assumptions and idealisations that we have made along the way warranted? Where do our driving theories, models, and representations of the phenomena under study stand in relation to rival or complementary conceptualisations? How are we best to interpret and contextualise the results of our modelling or experimental work? Are our instruments of measurement and analysis, our conceptual repertoire, up to the task at hand?
Andrews concludes that:
Every laboratory needs a philosopher. The philosopher's role, however, is not only to philosophise on behalf of scientists, but to re-educate scientists in how to philosophise for themselves.
This seems like a good idea. In recent years, statisticians and methodologists have made excellent contributions to re-educating scientists in the wake of the replication crisis. Philosophers should be key members of this educational team!
Andrews, M. (2023). \"Philosophy in the trenches and laboratory benches of science\": An essay by Mel Andrews. The Philosopher, 111(1). https://www.thephilosopher1923.org/post/philosophy-in-the-trenches-and-laboratory-benches-of-science
Sabina Leonelli's new book \u2013 \"Philosophy of Open Science\" \u2013 will be published later this year. However, there is an open access preprint available on the PhilSci Archive here.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!NTT9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png\")
In the book, Leonelli provides \"a constructively critical reading\" of the standard approach to open science which, she argues, is focused on sharing \"objects\" such as data and materials. In Leonelli's view, this object-sharing approach has become an obstacle to promoting open science. Grounded in philosophies of science from Popper, Chang, Longino, and others, she puts forward an alternative, process-oriented view of open science that refers to efforts to establish \"judicious connections among systems of practice.\"
I propose a conception of openness as judicious connection, which is grounded in a process-oriented epistemology of science that recognises the situated, embodied and goal-directed nature of communication and collaboration among researchers.
What does \"connection\" have to do with openness?
The idea of openness is quintessentially linked to that of learning as going beyond one's boundaries. This arguably applies to systems of research practice as much as to research groups and individual learners: establishing new connections often means expanding one's learning, challenging existing assumptions around what is considered external or irrelevant to a given system, and considering whether new boundaries need to be established that incorporate the novel relations.
Importantly, Leonelli does not equate an indiscriminate increase in scientific connections with an increase in open science. Instead,
the building and maintenance of connections need to be judicious: they require skilled deliberation, whereby the new opportunities offered by the connection in question are evaluated within the contexts at hand. Indeed, openness can itself be understood as a dynamic and highly situated mode of valuing the research process and its outputs, which encompasses economic as well as scientific, cultural, political, ethical and social considerations (Levin and Leonelli 2017).
The differences between the object-sharing and \"judicious connection\" approaches to open science also have implications for inclusion. The standard object-sharing approach assumes that greater transparency leads to a more inclusive science. In contrast, Leonelli's \"judicious connection\" approach flips the causal arrow and assumes that more inclusion leads to greater transparency.
The implementation of OS [open science] needs to start from consideration of what it may take to make research more inclusive, diverse and just \u2013 rather than expecting such an outcome to naturally follow from the 'right' choice of software, infrastructures, standards, publishing platforms, or whichever other technological or institutional fix is being devised to facilitate access to resources.
Finally, Leonelli's distinction between the two approaches to open science is important because she believes that the standard, object-sharing approach:
threatens to blindly privilege specific ways of knowing, thus potentially disrupting sophisticated methodologies, inadvertently dismissing well-established research traditions, and exacerbating the already large epistemic and social divides separating research domains and locations As denounced by a number of critics in science and science studies, there is a substantive risk of some OS policies \u2013 despite their good intentions and progressive slant - acting as a reactionary force which reinforces conservatism, discrimination, commodification and inequality in research, thus ultimately closing down opportunities for inquiry in a disastrous reversal of what they set out to achieve.
I won't go any further into the distinction between the object-sharing and judicous connection approaches to open science. Leonelli already does an excellent job of that in her book. Instead, I want to highlight five quotes that are critical of the standard object-sharing approach, because these are motivating towards a consideration of her alternative judicious connection approach.
Many of the more institutionalised OS initiatives tend to privilege a homogenous, universally applicable understanding of the scientific method over a pluralistic and situated one.
Similar points have been made by several others. For example, Malich and Rehmann-Sutter (2022) pointed out that:
Metascience often construes science and psychology from a homogenizing perspective that omits the plurality and complexity of both scientific methods and psychological approaches. This omission entails several epistemic risks because the reforms that metascience initiates or accompanies can reinforce these homogenizing tendencies
Open Methods is not a matter of recording and sharing every detail of a research procedure, but rather a reflection on which research components and techniques are most salient to the outcomes, and should thus be accessible and reproducible.
I totally agree. As I argue near the end of this presentation, I'm opposed to what I call \"mindless transparency.\"
![\"\"](\"https://substackcdn.com/image/fetch/$s_!XECF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png\")
By the way, I think Popper (2002, p. 230) would also agree with this point:
When the judge tells a witness that he should speak 'The truth, the whole truth, and nothing but the truth', then what he looks for is as much of the relevant truth as the witness may be able to offer. A witness who likes to wander off into irrelevancies is unsatisfactory as a witness, even though these irrelevancies may be truisms, and thus part of 'the whole truth'.
Taking a monolithic understanding of reproducibility as a demarcation strategy for the whole of science side-steps the precious plurality of methods developed to suit specific goals, concepts and target objects.
Again, this point has been made by several others (e.g., Devezer et al., 2019, 2021; Feest, 2019; Haig, 2022). Nonetheless, replication is often portrayed as a \"defining\" or \"essential\" aspect of science.
A case in point is the skepticism towards qualitative research often displayed within debates on reproducibility, which sometimes portray qualitative methods as hopelessly subjective and devoid of rigorous forms of data collection and verification....Assuming that qualitative research traditions in anthropology or sociology are incapable of rigorous research means drastically reducing the diversity of systems regarded as exemplars of good practice.
Yes! We need to be very careful that open science does not reopen \"'sectarian' divides between quantitative and qualitative\" research, as Dunleavy (2022) has cautioned.
To date, some parts of the OS movement \u2013 particularly its institutionalized, top-down incarnations - have paid too much attention to designing procedures and technologies for sharing, and this has come at the expense of strategies, training and procedures to assess who is included and excluded from such apparatus, understand why and with which implications, and mitigate eventual instances of epistemic injustice.
Another important point! A good example of this lack of attention to inclusion comes from Prosser et al. (2022) in their article \"When open data closes the door.\" These researchers found surveyed 261 social psychology journals and found that journal guidelines for sharing data were exclusionist with regards to qualitative data:
Guidelines explicitly addressing open qualitative data were very rare, provided by just three journals (2.1% of all relevant journals). All other journals that made reference to 'open data' did so without making a distinction between qualitative and quantitative data.\"
Leonelli, S. (2023). Philosophy of open science. [Preprint] http://philsci-archive.pitt.edu/21986/
Order the published print version here.
The University of Exeter's PHIL_OS project (2021\u20132025): \"Aims to develop an empirically grounded philosophy of Open Science [OS] that emphasises the diversity of research environments around the world and articulates the conditions under which OS can leverage such diversity to promote good research practice.
","doi":"https://doi.org/10.59350/x4g76-zb210","guid":"116187642","image":"https://substack-post-media.s3.amazonaws.com/public/images/564ba545-18df-463c-8bd3-65f469e19f6d_296x446.png","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1682035200,"rid":"pkrx0-kj258","summary":"Sabina Leonelli's new book \u2013 \"Philosophy of Open Science\" \u2013 will be published later this year. However, there is an open access preprint available on the PhilSci Archive here. In the book, Leonelli provides \"a constructively critical reading\" of the standard approach to open science which, she argues, is focused on sharing \"objects\" such as data and materials.","title":"Sabina Leonelli's \"Philosophy of Open Science\"","updated_at":1792238140,"url":"https://markrubin.substack.com/p/sabina-leonellis-philosophy-of-open","version":"v1"}},{"document":{"authors":[{"affiliation":[{"name":"Durham University, Department of Psychology"}],"contributor_roles":[],"family":"Rubin","given":"Mark","url":"https://orcid.org/0000-0002-6483-8561"}],"blog":{"authors":[{"name":"Mark Rubin","url":"https://orcid.org/0000-0002-6483-8561"}],"community_id":"3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6","created":1681948800,"current_feed_url":null,"description":"Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.","doi":"https://doi.org/10.59350/markrubin","favicon":"https://rogue-scholar.org/api/communities/3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6/logo","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"markrubin","status":"active","subfield":"1207","title":"Critical Metascience","updated":1792151011,"use_api":true},"blog_name":"Critical Metascience","blog_slug":"markrubin","content_html":"In this new article, I consider questionable research practices in the field of metascience. A questionable metascience practice (QMP) is a research practice, assumption, or perspective that's been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science. My aim is not to cast aspersions on the field of metascience but to encourage a deeper consideration of its more questionable research practices, assumptions, and perspectives.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!VAlR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg\")
Rejecting or ignoring self-criticism: Rejecting or ignoring criticisms of metascience and/or science reform
Fast 'n' bropen scientific criticism: A quick, superficial, dismissive, and/or mocking style of scientific criticism
Overplaying the role of replication: Assuming that replication is essential to science, and that it indexes \"the truth\"
Unspecified replication rate targets: Assuming a replication rate is \"too low\" without specifying an \"acceptable\" rate
Metabias: An unacknowledged bias towards explaining the replication crisis in terms of researcher bias
A bias reduction assumption: Focusing on selective reporting as the primary form of researcher bias and assuming that it can be reduced without increasing other forms of bias
Devaluing exploratory results: Devaluing an exploratory result as being more \"tentative\" than a confirmatory result without considering other relevant issues (e.g., quality of associated theory, methods, analyses, transparency)
Presuming QRPs are problematic: Presuming that questionable research practices (e.g., HARKing) are always problematic research practices
A focus on knowledge accumulation: Conceiving knowledge accumulation as the primary objective of science without considering (a) the role of specified ignorance or (b) different objectives in other philosophies of science
Homogenizing science: Focusing on specific approaches (e.g., quantitative methods; replicable effects) as \"the scientific method\"
In my article, I stress that only some metascientists engage in some QMPs some of the time, and that these QMPs may not always be problematic. Research is required to estimate the prevalence and impact of QMPs. In the meantime, I think that QMPs should be viewed as invitations to ask questions about how we go about doing better metascience.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!v6xt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg\")
Rubin, M. (2023). Questionable metascience practices. Journal of Trial and Error. https://doi.org/10.36850/mr4
My article is part of a special issue in the Journal of Trial and Error: \"Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction? https://journal.trialanderror.org/pub/callscientificreform/release/4
Many of the articles that I cite in my article are included in this chronological list of critical metascience articles: https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/list-of-critical-metascience-articles
The December 2022 issue of the journal Industrial and Organizational Psychology includes a very interesting discussion of the pros and cons of open science. There's a target article titled \"Open science, closed doors: The perils and potential of open science for research in practice\" (Guzzo et al., 2022) and nine commentaries on this article. You can find all 10 papers here.
In this post, I focus on Andrea Bazzoli's (2022) excellent commentary: \"Open science and epistemic pluralism: A tale of many perils and some opportunities.\" Many thanks to Andrea for drawing my attention to his work on Twitter. I elaborate on some of his points with my own thoughts inspired by Sabina Leonelli's (2022) recent work.
In his article, Bazzoli distinguishes between two research paradigms:
Postpositivism: A research paradigm holding that a knowable, tangible, and measurable reality exists (i.e., na\u00efve or critical realism), knowledge claims about this reality can be developed objectively, and verification/falsification of a priori hypotheses is the most prevalent methodological choice.
Constructionism: A scholarly movement that holds that reality is the result of communicative processes that create a sense of shared reality (i.e., it is locally co-constructed), emphasizes that objectivity is also co-constructed through communicative processes, and aims to examine taken-for-granted realities that might be oppressive or dysfunctional through future-forming, dialogic, hermeneutical, and dialectical research to generate new functional realities.
You can read a bit more about these and other research paradigms in Moin Syed's (2020) blog post.
It's important to distinguish between these different research paradigms and their associated research communities because, as Bazzoli explains,
undeclared and unexamined assumptions that are, by definition, self-evident to a community can be seen as impositions by other communities that do not share them (Romaioli & McNamee, 2021). In turn, a given community and its members will defend as reasonable arguments that logically descend from those assumptions against competing communities.
Bazzoli argues that the open science movement is based on the postpositivist research paradigm. Hence, for example, the open science principles of (a) providing a verifiable distinction between hypothesis generation and hypothesis testing, (b) reducing researcher bias, and (c) increasing reproducibility can be seen \"as reasonable arguments that logically descend from\" a postpositivist epistemology.
Bazzoli argues that:
the open science movement, as a direct offspring of (post)positivist research paradigms, has the potential to stifle epistemological and scientific pluralism and reproduce historical scientific hierarchies it purports to redress.
He's speaking here of postpositivism as a dominant epistemology in the sciences and arguing that advocating open science also advocates postpositivist epistemology, bolstering its dominance and, potentially, distorting and displacing minority research paradigms that are based on non-postpositivist epistemologies such as constructivism. Hence, at least in its postpositivist form, open science has the potential to stifle scientific pluralism. Bazzoli provides a useful practical example:
There is a movement, especially in Northern Europe, that aims to link adherence to open science practices to tenure and promotion.
For example, in their article \"Nudging open science,\" Robson et al. (2021) proposed that:
evaluating candidates for hire and promotion in faculties can also be modified to weight open practices more heavily. Such changes are likely to encourage researchers to reconsider their practices and can increase the perception that open practices are not only normative but also valued (Nosek, 2019). Job listings and interview questions could ask for evidence of open practices or for the candidate to illustrate commitment to transparent research in their cover letter. It may also be fruitful to ask candidates to provide an annotated CV detailing whether their articles have been preregistered or openly accessible, what datasets they have openly shared\u2026
These \"nudges\" may encourage the adoption of open science practices such as preregistration and data sharing. However, they may also encourage the adoption of open science's postpositivist epistemology. Hence, as Bazzoli points out,
open science serves as a tool to reinforce postpositivism's hegemonic position in I-O [industrial and organisational] psychology and beyond, to the detriment of other epistemologies.
To be clear, there's nothing wrong with \"nudging\" postpositivist researchers to adopt open science research practices. The concern is with nudging non-postpositivist researcher to adopt open science practices. Two potential problems with this approach are assimilation and marginalisation (see also Bazzoli, 2022, p. 527):
Assimilation: Researchers import postpostivist epistemology into their non-postpositivist research paradigms in order to justify their use of newly adopted open science practices. Consequently, their paradigm loses some of its distinctive aspects.
Marginalisation: Non-postpositivist paradigms that don't adopt open science principles and practices suffer a reduction of resources as a result (e.g., a reduction in jobs, funding, journal space, etc.).
The case of qualitative research has proven to be an important testing ground for the applicability of standard open science practices. What type of data should qualitative researchers share and not share? Is it useful to preregister qualitative studies? Should we expect qualitative results to replicate? The answers to these questions are controversial (for recent discussions, see Branney et al., 2023; Steltenpohl et al., 2021) and, as Bazzoli observes, the controversy runs deeper than simply identifying whether a researcher's data is represented by numbers or words. It reflects a more fundamental difference in epistemic stances between quantitative and qualitative researchers:
We suggest that it is not qualitative methods, per se, that are at odds with open science principles (as Guzzo et al., 2022, and Pratt, Kaplan, & Whittington, 2020, among others, argued), but the universalistic and all-encompassing claims that are tied to open science are incompatible with epistemic stances most often embraced by qualitative scholars.
In other words, postpositivist open science practices may not fit the non-postpositivist epistemologies employed by qualitative researchers, and indiscriminately encouraging these researchers to adopt such practices may inadvertently compromise their epistemological approach. As Branney et al. (2023) point out,
qualitative researchers, who are increasingly required to engage in open science practices, may feel their research is being judged against quantitative criteria that are inappropriate, irrelevant, or incompatible (Brooks et al., 2018a; Pratt et al., 2020).
![\"\"](\"https://substackcdn.com/image/fetch/$s_!CfA_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42bdabf1-9a71-4969-9450-e5814e96df38_720x246.png)
One response to these sorts of concerns is to reassure researchers that the adoption of open science practices is not mandatory. For example, in their commentary on the Guzzo et al. (2022) article, H\u00fcffmeier et al. (2022) explained that:
it is neither the idea underlying OSPs [open science practices] to force researchers to engage in unwanted research practices, nor is there any movement in such a direction.
Another response is to stress that the principles of open science are flexible and adaptive to different disciplinary approaches. For example, in their commentary, Rudolph and Zacher (2022) explained that:
openness is not a monolithic \"one size fits all\" approach to either science or practice.
Both points are true. Nonetheless, it remains the case that the open science movement is promoting a set of policies and strategies that are designed to reform scientific culture in ways that are consistent with a postpostivist research paradigm. Hence, although researchers are free to choose (a) which open science practices to adopt and (b) how to implement them, they are being encouraged to adopt research practices that are designed for, and carry the assumptions of, a postpositivist research paradigm. Again, this approach is not problematic when researchers are already working within postpositivist research paradigms. The problem only arises when they're working within non-postpositivist paradigms. In this latter case, we might talk about postpositivist \"overreach\" or \"encroachment\" into non-postpositivist research paradigms. (For some examples, please see p. 14 of Beaumont and Coning's (2022) article: \"Coping with complexity: Toward epistemological pluralism in climate\u2013conflict scholarship.\")
The potential for open science's postpositivist encroachment on non-postpositivist research paradigms may be heightened by two factors. First, scientists may be motivated to universalize their scientific approaches. As Feyerabend (1975) noted:
Scientists are not content with running their own playpens in accordance with what they regard as the rules of scientific method, they want to universalize these rules, they want them to become part of society at large and they use every means at their disposal - argument, propaganda, pressure tactics, intimidation, lobbying - to achieve their aims.
In the case of open science, the motive for widespread propagation of open science practices is an honourable one: to improve the quality, rigour, and credibility of science. The concern is that, although these principles may benefit postpositivist research paradigms, they may be detrimental to non-postpositivist paradigms.
Second, because of its dominance and majority acceptance, the postpositivist research paradigm exerts its influence in a relatively subtle and unacknowledged way. As Kincheloe and Tobin (2015) explained:
many of the tenets of positivism are so embedded within Western culture, academia, and the world of education in particular that they are often invisible to researchers and those who consume their research\u2026.The crypto-positivist ideology is not perceived as oppressive and its role in quashing [non-positivist] research is not acknowledged.
Hence, \"crypto-positivism\" may mislead us into believing that open science principles and practices are universally applicable across all of science when, in fact, those principles and practices are limited to the postpositivist paradigm. This assumption of universality may help to explain universalist slogans such as \"open science; just science done right.\"
In summary, despite the fact that open science practices are neither mandatory nor one-size-fits-all, there may be an unconscious motive to encourage the uptake of these practices within non-postpositivist research paradigms because they're believed to be beneficial for science in general. Again, there's no suggestion of any ill-intentions on the part of the open science movement here. On the contrary, the open science community is clearly attempting to improve the quality, rigour, and credibility of science on the basis of \"reasonable arguments that logically descend from\u2026[their postpositivist] assumptions\" (Bazzoli, 2022). The concern is that these assumptions do not necessarily apply in the case of non-postpositivist research paradigms.
One solution to these potential problems is to open up open science to make it more inclusive and applicable to scientists from non-postpositivist research cultures. In other words, instead of adopting an epistemically monist and assimilatory approach to open science, we could adopt a pluralist and multicultural approach. What might pluralist open science look like? Bergmann's (2023) recent post on The Buffet Approach to Open Science provides a good starting point:
Open science practices are not all or none, you can pick and choose, match and mix, and do what's most suitable to your career stage, project, lab, and level of support.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!wtmc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaeebaa4-e37d-4af6-abfe-6b8a6cfb55b3_933x622.png\")
I'd suggest a couple of modifications to the original buffet approach to open science to make it a bit more inclusive. First, we should allow researchers from different research paradigms to create their own open science practices (cook their own dishes!) rather than merely pick and choose from a selection provided by postpositivists. Second, like Bazzoli, I'd argue that researchers from different paradigms should be free to pick and choose (and create) their own open science principles as well as their associated practices. For example, the principle of reproducibility may not apply to non-postpositivist research paradigms. As Bazzoli points out:
constructionist epistemologies are not concerned with replicability of findings at all.
In her new book, \"The Philosophy of Open Science,\" Leonelli (2022) explains that the open science movement's monist view of science is easier to implement but less well suited to address the \"rampant plurality\" that exists on the ground:
Many of the more institutionalised OS [open science] initiatives tend to privilege a homogenous, universally applicable understanding of the scientific method over a pluralistic and situated one. It is much easier to set up OS guidelines when assuming that science consist of a coherent body of knowledge and procedures that can and should conform to common norms \u2013 an assumption that flies in the face of the rampant plurality of research approaches used across domains, locations and contexts.
Certainly, opening up open science within multiple epistemological cultures will be harder than the monist approach because (a) it involves changing multiple scientific cultures, rather than a single scientific culture, and (b) open science principles and practices need to be customised to the needs of particular research paradigms. Opening up open science is also likely to lead to heated discussions between and within different research paradigms about \"what 'open' and 'science' mean\" (Bazzoli, 2022, p. 527). As Leonelli (2022) explains:
a key challenge for OS is to productively manage the clash between different interpretations and operationalisations of openness, which emerge from diverse systems of practice with unequal levels of influence and visibility.
One approach to managing this clash is to be more open about not only our methods and data, but also the previously \"undeclared and unexamined assumptions\" of our research paradigms (Bazzoli, 2022). It's only after a research paradigm's epistemic rationales for open science principles and practices are made explicit that its interpretations and operationalisations of openness can be fully understood and appreciated by scientists from other paradigms. Furthermore, it's only after \"judicious connections\" (Leonelli, 2022) are made across paradigmatic interpretations and operationalisations that we'll arrive at a genuinely pluralist open science.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!pGyB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F416c908b-9c99-475d-b44d-d492d48054de_539x551.png\")
Bazzoli, A. (2022). Open science and epistemic pluralism: A tale of many perils and some opportunities. Industrial and Organizational Psychology, 15(4), 525-528. https://doi.org/10.1017/iop.2022.67
You can download a PDF version of this article here.
The reference for this article is as follows:
Rubin, M. (2023, May 4). Opening up open science to epistemic pluralism: Comment on Bazzoli (2022) and some additional thoughts. Critical Metascience. https://doi.org/10.31222/osf.io/dgzxa
","doi":"https://doi.org/10.59350/b20k8-0sr15","guid":"117317477","image":"https://substack-post-media.s3.amazonaws.com/public/images/5cb4484b-0901-48a0-8ff7-602e98b0f182_539x551.png","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1682899200,"reference":[{"id":"https://doi.org/10.31222/osf.io/dgzxa","unstructured":"Rubin, M. (2023). Opening up open science to epistemic pluralism: Comment on Bazzoli (2022) and some additional thoughts. Center for Open Science."}],"rid":"0qcn3-w2j73","summary":"Comment on Bazzoli (2022)","title":"Opening Up Open Science to Epistemic Pluralism","updated_at":1792238135,"url":"https://markrubin.substack.com/p/opening-up-open-science-to-epistemic","version":"v1"}},{"document":{"authors":[{"affiliation":[{"name":"Durham University, Department of Psychology"}],"contributor_roles":[],"family":"Rubin","given":"Mark","url":"https://orcid.org/0000-0002-6483-8561"}],"blog":{"authors":[{"name":"Mark Rubin","url":"https://orcid.org/0000-0002-6483-8561"}],"community_id":"3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6","created":1681948800,"current_feed_url":null,"description":"Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.","doi":"https://doi.org/10.59350/markrubin","favicon":"https://rogue-scholar.org/api/communities/3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6/logo","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"markrubin","status":"active","subfield":"1207","title":"Critical Metascience","updated":1792151011,"use_api":true},"blog_name":"Critical Metascience","blog_slug":"markrubin","content_html":"Tom Hostler has just published an article titled \"The invisible workload of open research\" in the Journal of Trial and Error's special issue on the \"Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction?\" It's an insightful and thought-provoking piece that uncovers the potential workload costs of open research, considers why these costs may be ignored by university management, and makes some useful recommendations for moving forward. Here's a few quotes to whet your appetite.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!Agiz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png)
The benefits of open research practices have been widely discussed (Munaf\u00f2 et al., 2017), and generally speaking researchers have positive attitudes towards adopting open research practices, finding them worthwhile (Eynden et al., 2016; Lowndes et al., 2017), and recommending their use to others (Sarafoglou et al., 2022). However, high time cost is repeatedly identified as one of the main barriers to the adoption of open practices (Eynden et al., 2016; Gownaris et al., 2022; Tenopir et al., 2011) and time is the main thing that researchers lack, with workloads at capacity across the sector, having already reached \"untenable\" levels (Long et al., 2020; Papadopoulos, 2017).
In short, implementing open science reforms take work, and work takes time, which academics have very little of!
The theory of academic capitalism suggests that the responsibility for addressing potential discrepancies between modelled and actual workload will not be taken up voluntarily by university management, who may at best ignore such issues, or at worse tacitly approve of them as a form of capitalist efficiency (Lyons & Ingersoll, 2010). In other words, if university management can choose not to incorporate open research into workload models, then they won't.
I don't think many working academics need to refer to the theory of academic capitalism to appreciate this point! But to quell any doubt, Hostler refers to empirical work by Kenny and Fluck (2022) and Miller (2019). For example, Kenny and Fluck (2022) found that:
Management knows that academics will do research (and other work) in their own time and exploit that\u2026.What is written on paper in terms of hours re admin or research does not correspond to the actual time spent. It all looks fine on paper but has enormous hidden workloads.
Hence, Hostler concludes:
There is a high potential for expectations to engage in open research to become widespread, but without additional time on academics' workloads dedicated to the activity.
Suggestions that concerns about the workload of open research are an example of a \"myth\" (Bastiaansen, 2019) or a \"misconception\" that can be corrected by \"positive advocacy\" (Hagger, 2022) are unhelpful to having honest conversations about the practical negative consequences of conducting open research.
This is an important point. Some open science advocates engage in a pretty hard sell, and this can include omitting or downplaying some of the costs of open science. However, like Hostler says, it's worth having \"honest conversations\" about the pros and cons of open science in order to set more realistic expectations.
I've just picked out one of Hostler's recommendations below to keep things brief, but note that he makes several more in his article:
When promoting open research reforms in universities or conducting open practices themselves [open research advocates] should advocate not only for investments in open research infrastructure and training but to receive extra time in workload allocations to acknowledge the additional burden of open research.
Getting open research work acknowledged in university workload models is likely to be easier said than done! As Hostler explains, open research work often consists of many small tasks, rather than a few large tasks, and so it's easily overlooked in workload models. Nonetheless, cumulatively, open research workload can be substantial and, as Hostler notes, if it's ignored, we risk \"death by a thousand 10-minute tasks\" (Bozeman et al., 2021).
![\"\"](\"https://substackcdn.com/image/fetch/$s_!NzXv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg\")
Hostler, T. J. (2023). The invisible workload of open research. Journal of Trial & Error. https://doi.org/10.36850/mr5
","doi":"https://doi.org/10.59350/8ch32-n6h54","guid":"119984535","image":"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1683504000,"rid":"bmd6h-jds77","summary":"Comments on Hostler (2023)","tags":["Metascience","Workload Models","Open Science","Academic Workload","Science Reform"],"title":"Open Science and Overwork","updated_at":1792238133,"url":"https://markrubin.substack.com/p/open-science-and-overwork","version":"v1"}},{"document":{"authors":[{"affiliation":[{"name":"Durham University, Department of Psychology"}],"contributor_roles":[],"family":"Rubin","given":"Mark","url":"https://orcid.org/0000-0002-6483-8561"}],"blog":{"authors":[{"name":"Mark Rubin","url":"https://orcid.org/0000-0002-6483-8561"}],"community_id":"3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6","created":1681948800,"current_feed_url":null,"description":"Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.","doi":"https://doi.org/10.59350/markrubin","favicon":"https://rogue-scholar.org/api/communities/3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6/logo","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"markrubin","status":"active","subfield":"1207","title":"Critical Metascience","updated":1792151011,"use_api":true},"blog_name":"Critical Metascience","blog_slug":"markrubin","content_html":"I discuss Van Drimmelen's (2023) Metascience2023 presentation on researchers' decision making during the research process. In particular, I consider his evidence that researchers' discretion over research decisions is unavoidable when they follow research plans that are either overdetermined (i.e., too prescriptive) or underdetermined (i.e., too vague). I argue that this evidence points to a prescriptiveness trade-off when writing preregistered plans: All other things being equal, plans that are more prescriptive are more likely to result in deviations that turn their confirmatory tests into exploratory tests, and plans that are less prescriptive are more likely to result in confirmatory tests that are susceptible to questionable research practices. I also consider Van Drimmelen's idea that researchers may make unconscious, implicit decisions during the research process. I relate these implicit decisions to Rumsfeld's (2002) concept of unknown unknowns: \"the things we don't know we don't know\"! I argue that scientists can report their known knowns (what they know they did and found), and they can be transparent and speculative about their known unknowns (what they know they didn't do and may find), but that they can't say much about their unknown unknowns (including their unconscious, implicit decisions) because, by definition, they don't know what they are! Nonetheless, I think that it's important to acknowledge unknown unknowns in science because doing so helps to contextualise research efforts as being highly tentative and fallible.
The videos of the presentations at the Metascience2023 conference were recently made available here. One that caught my eye is by Tom van Drimmelen, titled \"Researchers' decision making: Navigating ambiguity in research practice\":
Here, I summarise Van Drimmelen's excellent work and share my thoughts on his ideas.
Van Drimmelen (2023) explained that, during the research process, researchers have discretion over numerous decisions that they make about their methodology and analyses. During the data analysis stage, this researcher discretion may be perceived negatively as researcher degrees of freedom that is susceptible to researcher bias and questionable research practices (John et al., 2012; Simmons et al., 2011).
![\"\"](\"https://substackcdn.com/image/fetch/$s_!LVC8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png\")
In theory, it is possible to reduce and/or reveal researcher discretion during data analyses by preregistering one or more methodological and analytical research paths. However, Van Drimmelen (2023) and colleagues were interested in how researcher discretion operates in practice. To investigate, they conducted an ethnographic study in which they observed researcher discretion in two different research groups. They found that researcher discretion emerged when researchers used two categories of research plan.
1. Underdetermined plans: Researcher discretion emerged when research plans were not precise enough to determine specific actions for the researchers. In this case, researchers needed to use their own discretion to fill in the gaps of the research plan and figure out how to proceed. Researchers often find themselves in this situation because preregistered research plans are often vague (e.g., Bakker et al., 2020; Heirene et al., 2021; Van den Akker et al., 2023).
2. Overdetermined plans: Researcher discretion also emerged when there were unforeseen changes in the research situation that made the original research plan undesirable or impossible, necessitating an alternative research approach. Again, this situation is common, because most researchers do not possess \"godlike planning abilities\" (Navarro, 2020). To attempt to address this issue, researchers can preregister \"if\u2026then\" contingencies (i.e., decision trees) that accommodate anticipated changes in the research situation (Nosek et al., 2018, p. 2602). In addition, it may be possible to update a preregistered analysis plan prior to the data analysis in order to take account of unanticipated events (Nosek et al., 2019, p. 817). However, it is often the case that unanticipated events arise during data analyses. As Reinhart (2015, p. 95) explained (paraphrasing Helmuth von Moltke), \"no analysis plan survives contact with the data.\" In these cases, researchers either need to stick to an inadequate preregistered analysis plan or deviate and adapt their approach to address the current situation. The evidence shows that researchers often deviate from their preregistered plans (e.g., Abrams et al., 2020; Claesen et al., 2021; Heirene et al., 2021).
Van Drimmelen's (2023) work shows us that researchers are forced to either deviate from overdetermined plans to overcome unforeseen events or fill in the gaps of vague, underdetermined plans. Hence, researcher discretion emerges in the case of both over- and under-determined plans. As Van Drimmelen (2023) put it, \"researcher discretion is an integral and unavoidable part of the research practice.\"
I think Van Drimmelen's (2023) evidence also points to a trade-off between the prescriptiveness of preregistered research plans and the feasibility of their implementation in an unpredictable world. As preregistered plans become more precise and prescriptive, there is a greater chance that researchers will need to deviate from them in order to accommodate unforeseen and/or uncontrolled events that occur during the implementation of the plan. For example, researchers are more likely to deviate from a plan to test \"300 participants\" than they are to deviate from a less prescriptive plan to test \"around 300 participants,\" because uncontrolled events (e.g., the number of participant exclusions during data analysis) may result in a final sample size that is only close to 300 participants and not exactly 300 participants (e.g., Claesen et al., 2021, pp. 6-7). Consequently, as preregistered plans become more prescriptive, they are more likely to result in deviations that turn their planned confirmatory tests into unplanned exploratory tests.1 Of course, preregistration continues to distinguish between confirmatory and exploratory tests in this situation, and it prevents exploratory tests from being falsely portrayed as confirmatory tests. Nonetheless, it remains the case that, as plans for confirmatory tests become more prescriptive, the actual implementation of these confirmatory tests in the real world becomes less likely.
On the other hand, as preregistered plans become vaguer and less prescriptive, confirmatory hypothesis tests will become not only more feasible to achieve without deviation, but also more susceptible to researcher degrees of freedom and questionable research practices. For example, it is more feasible to test \"around 300 participants\" than to test exactly \"300 participants.\" However, the first sampling plan does not prevent the questionable research practice of optional stopping (John et al., 2021; Simmons et al., 2011), whereas the second, more prescriptive plan does.
In summary, researchers need to consider a prescriptiveness trade-off when writing their preregistered plans: All other things being equal, plans that are more prescriptive are more likely to result in deviations that turn their confirmatory tests into exploratory tests, and plans that are less prescriptive are more likely to result in confirmatory tests that are susceptible to questionable research practices. Hence, researchers face a difficult choice between prescriptive confirmatory tests that they are less likely to carry out in practice and vague confirmatory tests that are more prone to questionable research practices.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!KKeL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg\")
Van Drimmelen and colleagues (2023) also identified another interesting issue in their work. At times, they found it difficult to identify researcher discretion because there were multiple potential methodological or analytical approaches available that researchers did not consider. In other words, sometimes researchers were not aware that they were making decisions!
This idea of unconscious and implicit research decisions reminds me of something that Donald Rumsfeld, the United States Secretary of Defence, said at a press briefing in 2002:
There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns\u2014the ones we don't know we don't know. (Wikipedia, 2023)
In the context of Figure 1, our known knowns are the path or paths that we reported following in our research (e.g., the green paths). In these cases, we know what we did, and we know what we found as a result. In contrast, our known unknowns are the paths that we actively decided not to follow (e.g., all the other paths in Figure 1). We know these paths exist, but we don't yet know what results lie at the end of them.
But what about our unknown unknowns? These paths don't appear in Figure 1! They relate to decisions that we didn't even consider, at least not at a conscious level. For example, we may not have considered the time of day of testing, the ambient temperature, or a host of other unspecified variables that may be influential on our results. Instead, we unconsciously relegated these unknown unknowns to a ceteris paribus clause (i.e., \"all other things being equal\"), which assumes that no other influential factors are at play (Meehl, 1990). The problem is that our unconscious and implicit decision to assume the irrelevance of these factors may be wrong! Indeed, the history of science is full of cases in which factors that were not initially considered to be important subsequently turned out to be crucial moderators or boundary conditions of effects (for some examples, see Firestein, 2016).
![\"\"](\"https://substackcdn.com/image/fetch/$s_!0Dff!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg\")
Van Drimmelen (2023) concluded that \"we might not be able to be effectively transparent because we don't know which decisions we've actually made\" (my emphasis). In other words, our lack of awareness about our potentially important implicit decisions means that we can't be transparent about them: We can report our known knowns, and we can be transparent and speculative about our known unknowns, but we can't say much about our unknown unknowns because we don't know what they are! By definition, as soon as we start to speculate on a potentially influential factor in our research, it becomes a known unknown about which we need to make conscious decisions!
Despite our inability to specify our unknown unknowns, Van Drimmelen's (2023) work reminds us that it's important to acknowledge this type of ignorance in science because it helps to contextualise our research efforts as being highly tentative and fallible steps in a collective process of discovery and theory development. Acknowledging our unknown unknowns also helps to address recent calls for greater intellectual humility and modesty in science (Hoekstra & Vazire, 2021; Ramsey, 2021).
I'll end with a nice example of scientists acknowledging unknown unknowns. Here's Errington et al. (2021, p. 12) discussing the challenges of assessing replicability in preclinical cancer biology:
It might also be that, in some cases, a failure to replicate was caused by the replication team deviating from the protocol in some way that was not recognized, or that a key part of the procedure was left out of the protocol inadvertently. It is also possible that the effect reported in the original paper depended on methodological factors that were not identified by original authors, the replication team, or any other experts involved in the peer review of the original paper or the Registered Report.
Abrams, E., Libgober, J., & List, J. A. (2020). Research registries: Facts, myths, and possible improvements (No. w27250). National Bureau of Economic Research. https://doi.org/10.3386/w27250
Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A.,\u2026& Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12), e3000937. https://doi.org/10.1371/journal.pbio.3000937
Claesen, A., Gomes, S., Tuerlinckx, F., Vanpaemel, W., & Leuven, K. U. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), 1\u201311. https://doi.org/10.1098/rsos.211037
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Reproducibility in cancer biology: Challenges for assessing replicability in preclinical cancer biology. Elife, 10, Article e67995. https://doi.org/10.7554/eLife.67995
Fife, D., & Rodgers, J. L. (2019). Moving beyond the \"replication crisis\": Understanding the exploratory/confirmatory data analysis continuum. PsyArXiv. https://doi.org/10.31234/osf.io/5vfq6
Firestein, S. (2016, February 14). Why failure to replicate findings can actually be good for science. LA Times. https://www.latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html
Heirene, R., LaPlante, D., Louderback, E. R., Keen, B., Bakker, M., Serafimovska, A., & Gainsbury, S. M. (2021). Preregistration specificity & adherence: A review of preregistered gambling studies & cross-disciplinary comparison. PsyArXiv. https://psyarxiv.com/nj4es/
Hoekstra, R., & Vazire, S. (2021). Aspiring to greater intellectual humility in science. Nature Human Behaviour, 5(12), 1602-1607. https://doi.org/10.1038/s41562-021-01203-8
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524\u2013532. https://doi.org/10.1177/0956797611430953
Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108-141. https://doi.org/10.1207/s15327965pli0102_1
Navarro, D. (2020, September 23). Paths in strange spaces: A comment on preregistration. PsycArXiv. https://doi.org/10.31234/osf.io/wxn58
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van 't Veer, A. E., & Vazire, S. (2019). Preregistration is hard, and worthwhile. Trends in Cognitive Sciences, 23, 815\u2013818. https://doi.org/10.1016/j.tics.2019.07.009
Ramsey, R. (2021). A call for greater modesty in psychology and cognitive neuroscience. Collabra: Psychology, 7(1), 24091. https://doi.org/10.1525/collabra.24091
Reinhart, A. (2015). Statistics done wrong: The woefully complete guide. No Starch Press.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359\u20131366. https://doi.org/10.1177 /0956797611417632
Van den Akker, O., Bakker, M., van Assen, M. A. L. M., Pennington, C. R., Verweij, L., Elsherif, M. M.,\u2026Wicherts, J. M. (2023, May 10). The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency. PsyArXiv. https://doi.org/10.31222/osf.io/h8xjw
Van Drimmelen, T., Slagboom, N., Reis, R., Bouter, L., Van der Steen, J. (2023, May 9). Researchers' decision making: Navigating ambiguity in research practice. Metascience2023, Washington, DC. Abstract: https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice/ Preregistration: https://osf.io/tqwgp/
Wikipedia. (2023, March 16). There are unknown unknowns. Wikipedia. https://en.wikipedia.org/wiki/There_are_unknown_unknowns
1. Here, I define confirmatory tests as planned tests, and exploratory tests as unplanned tests. This conceptualization is consistent with the frequentist rationale for preregistration, according to which unplanned tests undermine assumptions regarding Type I error rates (Nosek et al., 2019, p. 816). I acknowledge that there are other definitions for \"confirmatory\" and \"exploratory\", including definitions that view them as existing at opposite ends of a continuum rather than as discrete categories (e.g., Fife & Rodgers, 2019). According to this continuum perspective, larger deviations from preregistered plans result in \"more exploratory\" tests than smaller deviations. I don't subscribe to this continuum perspective, because it is unclear how to formally assess some deviations as being \"larger\" or \"smaller\" than others. I agree that research studies can be described as being more confirmatory or exploratory, because they can contain a greater proportion of planned or unplanned tests. However, in my view, the tests themselves can only be either confirmatory or exploratory, because they can only be either planned or unplanned.
I'm grateful to Tom van Drimmelen for his comments on an earlier version of this article.
You can download a PDF version of this article here.
The reference for this article is:
Rubin, M. (2023, June 7). The preregistration prescriptiveness trade-off and unknown unknowns in science: Comments on Van Drimmelen (2023). Critical Metascience: MetaArXiv. https://doi.org/10.31222/osf.io/3t7pc
Felipe Romero presented his work - \"The conceptual origins of metascience: Fashion or revolution?\" - in The Popper Seminar at the London School of Economics on 30th May 2023. Here, I highlight a few points that he raised and provide some comments along the way.
Metascience is often described as the \"science of science.\" However, as Peterson and Panofsky (2023) recently observed, contemporary metascience is something more than this. To illustrate, they used a Venn diagram to show how metascience may be conceived as falling at the intersection of science of science, open science, and methodological activism.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!MheW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1726018f-1f04-464b-b963-4e43855339ec_689x629.png)
After providing some examples of the various aspects of metascience, Romero considers some different ways of distinguishing metascience from related disciplines such as (a) the history and philosophy of science, (b) sociology, and (c) science, technology, and society (STS) studies. He notes that metascientists see themselves as quite separate from these other disciplines. Indeed, he notes that some metascientists seem to show a resistance to engaging with these disciplines by (a) failing to engage in interdisciplinary discussions, (b) not citing interdisciplinary work, and (c) generally adopting \"an adversarial attitude.\"
But what distinguishes metascience from these other disciplines? Romero runs through several possibilities and considers the pros and cons in each case.
Romero notes that metascientists tend to stress the empirical nature of their work as a distinctive feature. As Jonathon Schooler explained in Peterson and Panofsky's (2023) article, metascientists champion \"less armchair, less qualitative distillation of impressions and more hard-nosed empiricism\" (p. 167).
![\"\"](\"https://substackcdn.com/image/fetch/$s_!dp41!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c2cfabd-987b-4917-b7fd-173f40bbc1ff_680x445.jpeg\")
However, as Romero points out, one problem with emphasising the empirical character of metascience is that it runs the risk of concealing and/or overlooking important philosophical issues. What is a replication? What counts as evidence? What is progress? Empirical work must adopt philosophical positions on these questions, even if they're only implicit positions. As Romero explains, \"you might think that the work is empirical, but you are already making commitments about all of these notions.\"
The philosopher Daniel Dennett (2013) famously argued that \"there is no such thing as philosophy-free science, just science that has been conducted without any consideration of its underlying philosophical assumptions\" (p. 20). Similarly, it might be argued that there is no such thing as a philosophy-free metascience, and greater discussion of the philosophy of metascience may help to build mutually beneficial bridges between metascience and the philosophy of science.
Romero accepts that metascientists do conceptual work around issues such as replication, evidence, and progress. However, he suggests that, without paying closer attention to previous work that's been done in other disciplines, \"there is a risk of reinventing the wheel and repeating mistakes.\" To be fair, some metascientists couch their work within a Popperian philosophy of science (Derksen, 2019). Nonetheless, there remain concerns about the lack of integration with other positions in the philosophy of science and in other related fields (e.g., Gervais, 2021; Morawski, 2019, p. 234; Wiggins & Christopherson, 2019). For example, Gervais (2021) explained that:
Metascience and methodological reform movements\u2014despite noble intentions\u2014risk needlessly reinventing other fields' time-tested wheels or, worse, reproducing the problematic norms, institutions, and incentives of status quo science (p. 827).
To illustrate his point, Romero notes that some metascientists focus on \"methodological individualism,\" according to which \"(bad) statistics and methodology are the primary factors affecting science.\" From this perspective, many of the problems lie with individuals, and many of the proposed solutions are correspondingly individualistic in nature. Hence, as Field (2022) observed, \"the current crisis prominently features finger-pointing and ad-hominem attacks between researchers, where direct accusations of incompetence and data fabrication are hurled around between people and fields\" (p. 13).
![\"\"](\"https://substackcdn.com/image/fetch/$s_!WlHf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F904b4ff4-863c-4924-8539-1f9d80cc8e92_377x375.jpeg\")
However, Romero observes that other metascientists are rediscovering the idea that we need to look at social structures and organization to affect scientific progress (for further discussions, see Flis, 2019, p. 175; Flis, 2022; Wiggins & Christopherson, 2019, p. 210). I think this broader and more abstract level of analysis is more befitting of a genuinely metascientific approach. In my view, scientists should critique the work of other scientists, and metascientists should critique scientific processes and approaches.
Romero also notes that the \"dominant narrative\u2026that metascience is good because it is empirical\u2026could lead to methodological fetishism.\" In my view, \"methodological fetishism\" occurs when people use methodological checklists and heuristics in ways that ignore the local goals of specific research studies (see also Devezer et al., 2021, p. 3). For example, depending on the function(s) that you think it serves, preregistration may not be a useful approach in certain research situations (e.g., some qualitative studies; Steltenpohl et al., 2023). Nonetheless, Spitzer and Mueller (2023) found that some psychological researchers suggest making preregistration mandatory, and Aguinis et al. (2020, p. 29) have argued that we should \"require\" preregistration for both quantitative and qualitative studies. In these cases, the overapplication of preregistration may represent a form of methodological fetishism.
Furthermore, the use of checklists and heuristics may mislead researchers, reviewers, and their readers into a false sense of security about their approach when a deeper, more philosophical consideration would reveal important uncertainties and limitations. As Chamberlain (2000) explained, \"it is much less worrying to concern oneself with the correct details of method and methodology than to tangle with the philosophy of science\" (p. 293).
![\"\"](\"https://substackcdn.com/image/fetch/$s_!eSQc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97e7d7a5-924c-4aa5-b9c2-1a413765fbe1_571x437.jpeg\")
Metascience has seen a remarkable rise in popularity in the wake of the replication crisis. Hence, Romero asks whether metascience may be a \"fashion,\" although he makes it clear that this term is not used in a pejorative manner.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!Sn_9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3579bb03-e536-4464-9830-7982b2236841_252x252.png\")
To illustrate, Romero considers how metascience and open science might fit with a fashion model that has been applied previously in science (Crane, 1969; Starbuck, 2009). Consistent with this model, he notes that metascience contains:
Elites: Small groups of scholars who set fashions
Conformity: Scholars who adopt practices they don't endorse
Performativeness: Scientists who pursue a practice/theory to elicit approval from others
\"What's the problem?\" asks Romero. \"I think that some scientists are adopting open science as a fashion and this leads to performative open science.\"
As an example, Romero considers \"performative preregistration,\" which might manifest in a couple of ways: First, the \"underspecification problem\" is that researchers might go through the motions of preregistration, but preregister plans that are vague and, therefore, relatively useless (Bakker et al., 2020). Second, the \"undisclosed deviations problem\" is that researchers may not indicate deviations from their preregistered plan (Akker et al., 2022). Again, this approach ticks the preregistration box, but it makes the potential benefits of preregistration relatively inaccessible.
Romero concludes that a risk associated with metascience being a fashion is that fashions can have poor epistemic benefits and, when scientists undertake a cost-benefit analysis, this may result in a decline in the uptake of research practices like preregistration.
According to Romero, the best way to demarcate metascience is to refer to its normative, activist component: Metascientists tend to be activists who are trying to change scientific systems and culture (see also Peterson & Panofsky, 2023, p. 147).
![\"\"](\"https://substackcdn.com/image/fetch/$s_!T5GH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50a3716f-fa23-4efb-926a-f4171b742ac6_1198x1074.png\")
In Romero's view, it is the activism aspect of metascience that provides the sharpest distinction with the history and philosophy of science, sociology, and STS, because these other disciplines focus on describing and understanding science, but not necessarily improving it. Importantly, however, if metascience is to be distinguished by its goal of \"improving science,\" then I think the metascience community should include an adequate representation of viewpoints about what \"science\" is and how it can best be \"improved.\" In this sense, the current overrepresentation of psychologists in this community may be problematic (see also Flis, 2019).
Final point from me: As I've noted elsewhere (Rubin, 2023), there's a danger of homogenizing metascience in these discussions (Field, 2022). Like science, metascience may be many things to many people, and it may be done in many different ways, with different metascientists occupying different positions in Peterson and Panofsky's (2023) Venn diagram as well as having different interdisciplinary overlap with the history and philosophy of science, sociology, and STS. There is strength in diversity here, and we should be careful not to lose sight of this diversity during efforts to categorize and demarcate metascience.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!UZJZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdaa42441-ca6b-41fd-8b7d-0c15e1861362_718x800.jpeg\")
Website: https://www.feliperomero.org/
Publications: https://www.feliperomero.org/publications.html
Aguinis, H., Banks, G. C., Rogelberg, S. G., & Cascio, W. F. (2020). Actionable recommendations for narrowing the science-practice gap in open science. Organizational Behavior and Human Decision Processes, 158, 27-35. https://doi.org/10.1016/j.obhdp.2020.02.007
Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A.,\u2026& Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12), e3000937. https://doi.org/10.1371/journal.pbio.3000937
Chamberlain, K. (2000). Methodolatry and qualitative health research. Journal of Health Psychology, 5(3), 285-296. https://doi.org/10.1177/135910530000500306
Crane, D. (1969). Fashion in science: Does it exist? Social Problems, 16(4), 433-441. https://doi.org/10.2307/799952
Derksen, M. (2019). Putting Popper to work. Theory & Psychology, 29(4), 449-465. https://doi.org/10.1177/0959354319838343
Dennett, D. C. (2013). Intuition pumps and other tools for thinking. WW Norton & Company.
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (2021). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3), Article 200805. https://doi.org/10.1098/rsos.200805
Field, S. M. (2022, July 13). Charting the constellation of science reform. https://doi.org/10.31219/osf.io/udfw4
Flis, I. (2019). Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology, 29(2), 158-181. https://doi.org/10.1177/0959354319835322
Flis, I. (2022). The function of literature in psychological science. Review of General Psychology, 26(2), 146-156. https://doi.org/10.1177/10892680211066466
Gervais, W. M. (2021). Practical methodological reform needs good theory. Perspectives on Psychological Science, 16(4), 827-843. https://doi.org/10.1177/1745691620977471
Morawski, J. (2019). The replication crisis: How might philosophy and theory of psychology be of use? Journal of Theoretical and Philosophical Psychology, 39(4), 218\u2013238. https://doi.org/10.1037/teo0000129
Peterson, D., & Panofsky, A. (2023). Metascience as a scientific social movement. Minerva https://doi.org/10.1007/s11024-023-09490-3
Romero, F. (2023, May 30) The conceptual origins of metascience: fashion or revolution? YouTube. https://youtu.be/AKWCCdK-jgc
Rubin, M. (2023). Questionable metascience practices. Journal of Trial & Error. https://doi.org/10.36850/mr4
Spitzer, L., & Mueller, S. (2023). Registered report: Survey on attitudes and experiences regarding preregistration in psychological research. Plos One, 18(3), e0281086. https://doi.org/10.1371/journal.pone.0281086
Starbuck, W. H. (2009). The constant causes of never-ending faddishness in the behavioral and social sciences. Scandinavian Journal of Management, 25(1), 108-116. https://doi.org/10.1016/j.scaman.2008.11.005
Steltenpohl, C. N., Lustick, H., Meyer, M. S., Lee, L. E., Stegenga, S. M., Reyes, L. S., & Renbarger, R. L. (2023). Rethinking transparency and rigor from a qualitative open science perspective. Journal of Trial & Error. https://doi.org/10.36850/mr7
Van den Akker, O., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H., R\u00fcffer, F. F.,\u2026Bakker, M. (2022, October 18). Selective hypothesis reporting in psychology: comparing preregistrations and corresponding publications. PsyArXiv. https://doi.org/10.31222/osf.io/nf6mq
Wiggins, B. J., & Christopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202\u2013217. https://doi.org/10.1037/teo0000137
Rubin, M. (2023, June 13). What's special about metascience: Comments on Romero (2023). Critical Metascience. https://markrubin.substack.com/p/whats-special-about-metascience
Subscribe to this Blog and/or Share this Article
","doi":"https://doi.org/10.59350/8rscj-bgy43","guid":"126312825","image":"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1726018f-1f04-464b-b963-4e43855339ec_689x629.png","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1686614400,"reference":[{"id":"https://doi.org/10.1016/j.obhdp.2020.02.007","unstructured":"Aguinis, H., Banks, G. C., Rogelberg, S. G., & Cascio, W. F. (2020). Actionable recommendations for narrowing the science-practice gap in open science. Organizational Behavior and Human Decision Processes, 158, 27\u201335."},{"id":"https://doi.org/10.1371/journal.pbio.3000937","unstructured":"Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., Wicherts, J. M., & Bero, L. (2020). Ensuring the quality and specificity of preregistrations. PLOS Biology, 18(12), e3000937."},{"id":"https://doi.org/10.1177/135910530000500306","unstructured":"Chamberlain, K. (2000). Methodolatry and Qualitative Health Research. Journal of Health Psychology, 5(3), 285\u2013296."},{"id":"https://doi.org/10.2307/799952","unstructured":"Crane, D. (1969). Fashion in Science: Does It Exist?. Social Problems, 16(4), 433\u2013441."},{"id":"https://doi.org/10.1177/0959354319838343","unstructured":"Derksen, M. (2019). Putting Popper to work. Theory & Psychology, 29(4), 449\u2013465."},{"id":"https://doi.org/10.1098/rsos.200805","unstructured":"Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (2021). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3)."},{"id":"https://doi.org/10.31219/osf.io/udfw4","unstructured":"Field, S. M. (2022). Charting the Constellation of Science Reform. Center for Open Science."},{"id":"https://doi.org/10.1177/0959354319835322","unstructured":"Flis, I. (2019). Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology, 29(2), 158\u2013181."},{"id":"https://doi.org/10.1177/10892680211066466","unstructured":"Flis, I. (2022). The Function of Literature in Psychological Science. Review of General Psychology, 26(2), 146\u2013156."},{"id":"https://doi.org/10.1177/1745691620977471","unstructured":"Gervais, W. M. (2021). Practical Methodological Reform Needs Good Theory. Perspectives on Psychological Science, 16(4), 827\u2013843."},{"id":"https://doi.org/10.1037/teo0000129","unstructured":"Morawski, J. (2019). The replication crisis: How might philosophy and theory of psychology be of use?. Journal of Theoretical and Philosophical Psychology, 39(4), 218\u2013238."},{"id":"https://doi.org/10.1007/s11024-023-09490-3","unstructured":"Peterson, D., & Panofsky, A. (2023). Metascience as a Scientific Social Movement. Minerva, 61(2), 147\u2013174."},{"id":"https://youtu.be/akwccdk-jgc","unstructured":"Unknown title"},{"id":"https://doi.org/10.36850/mr4","unstructured":"Rubin, M. (2023). Questionable Metascience Practices. Journal of Trial and Error, 4(1), 5\u201320."},{"id":"https://doi.org/10.1371/journal.pone.0281086","unstructured":"Spitzer, L., Mueller, S., & Naudet, F. (2023). Registered report: Survey on attitudes and experiences regarding preregistration in psychological research. PLOS ONE, 18(3), e0281086."},{"id":"https://doi.org/10.1016/j.scaman.2008.11.005","unstructured":"Starbuck, W. H. (2009). The constant causes of never-ending faddishness in the behavioral and social sciences. Scandinavian Journal of Management, 25(1), 108\u2013116."},{"id":"https://doi.org/10.36850/mr7","unstructured":"Steltenpohl, C. N., Lustick, H., Meyer, M. S., Lee, L. E., Stegenga, S. M., Standiford Reyes, L., & Renbarger, R. L. (2023). Rethinking Transparency and Rigor from a Qualitative Open Science Perspective. Journal of Trial and Error, 4(1), 47\u201359."},{"id":"https://doi.org/10.31222/osf.io/nf6mq","unstructured":"Van den Akker, O., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H. H., R\u00fcffer, F. F., Schoenmakers, M., Stoevenbelt, A. H., Wicherts, J. M., & Bakker, M. (2022). Selective Hypothesis Reporting in Psychology: Comparing Preregistrations and Corresponding Publications. Center for Open Science."},{"id":"https://doi.org/10.1037/teo0000137","unstructured":"Wiggins, B. J., & Christopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202\u2013217."}],"rid":"g14t5-fsf37","summary":"Comments on Romero (2023)","tags":["Metascience","Methodolatory","Science, Technology, And Society","Methodologism","Open Science"],"title":"What's Special About Metascience?","updated_at":1792238127,"url":"https://markrubin.substack.com/p/whats-special-about-metascience","version":"v1"}},{"document":{"authors":[{"contributor_roles":[],"family":"Hostler","given":"Tom"}],"blog":{"authors":[{"name":"Mark Rubin","url":"https://orcid.org/0000-0002-6483-8561"}],"community_id":"3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6","created":1681948800,"current_feed_url":null,"description":"Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.","doi":"https://doi.org/10.59350/markrubin","favicon":"https://rogue-scholar.org/api/communities/3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6/logo","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"markrubin","status":"active","subfield":"1207","title":"Critical Metascience","updated":1792151011,"use_api":true},"blog_name":"Critical Metascience","blog_slug":"markrubin","content_html":"In his book \"The Soul of a University\" (2018), Chris Brink describes the story of G.H. Hardy, a Cambridge Mathematician whose principled stance on his academic research was that it had no practical use whatsoever: \"No discovery of mine\" Hardy proudly wrote, \"has made, or is likely to make, directly or indirectly, for good or ill, the least difference to the amenity of the world\" (Hardy, 1945/2005). Unfortunately, he was incorrect. In a rare and reluctant foray outside his own work, he wrote a short letter to Science (Hardy, 1908), explaining and correcting a mathematical mistake pointed out to him by a geneticist friend who he played cricket with. The proof he provided is now known as the Hardy-Weinberg Principle and is a foundational law of the study of population genetics.
Hardy's belief in the uselessness of research was rooted in the idea that non-instrumental research was the purest expression of intellectual curiosity. Although Hardy was an extreme example, the idea of producing knowledge for its own sake \u2013 rather than for an instrumental purpose - is closely wrapped up in the traditional stereotypes and ideals of \"academia\". Universities from this view are seen as 'protected' spaces in society where intellectual work can be undertaken by those with the aptitude and ability to do so, as represented by the metaphor of the disconnected \"ivory tower\", the residence of \"intellectuals\", \"eggheads\", and \"boffins\"1. These individuals' vocation is to focus on esoteric but often impracticable concerns, compared to the rest of society mired in problems of the real world. This stereotype is captured in the phrase \"that's academic\", often used to dismiss things as hypothetical concerns which are practically meaningless.
Academia's disconnect from practical reality and curiosity-driven mission help to provide a basis for its role in society. These stereotypes help to support the myth or 'legend' of academic research (especially scientific research) as a legitimate authority on defining the \"truth\" of an issue and resolving factual disputes. Academia is traditionally seen as a politically neutral institution of independent thinkers, who are trusted to provide disinterested 'facts' to inform moral or political decisions, even if these are unpalatable or relate to controversial topics.
The academic institution that these stereotypes and myths are based on has changed significantly over the last few decades, and the changes look to be accelerating, with the increasingly influential scientific reform movement advocating for a 'revolution' in how research is conducted, published, consumed, and assessed (Spellman, 2015). This essay will focus on how these reforms are linked to existing changes in academia and what the effects might be on our culture and traditions.
The cornerstone of the current piece is the work of John Ziman, and his book \"Real Science \u2013 What it is, and what it means\" (Ziman, 2000), which covers his attempt to investigate the 'legend' of science, and the legitimacy of its authority. Ziman takes a refreshing perspective. Instead of looking for connections between the philosophies or methods of different academic disciplines to find a common definition of what research 'is', he adopts a naturalistic approach: he describes what he sees. The result is a fuzzy yet recognisable picture of 'traditional' academia, based not on shared methodologies or philosophies of science but of shared community practices and values. Ziman suggests that academic research is best described as an institution based around a culture of shared traditions and citizenship. Whilst there are significant differences between academic disciplines in the type and form of knowledge produced, there is a common principle of sharing, discussing, and collectively evaluating codified contributions to knowledge from the academic community.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!PI-A!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F571e70ca-f07a-470b-aa34-eddcc9ef08f9_667x1000.jpeg\")
Ziman describes traditional academia as a pre-modern institution without clear written rules, operating according to shared values and ethos of Communalism, Universalism, Disinterestedness, Originality, and Scepticism (CUDOS) that differentiate academic research from other kinds of knowledge generation in society. These values \u2013 linked to Merton's norms - are closely tied to the non-instrumental nature of much traditional academic research: being driven by intellectual curiosity alone is what grants academics the autonomy and freedom to share knowledge freely, universally accept contributions of merit, remain disinterested to biasing incentives, pursue originality, and express scepticism. This autonomy applies to both individual academics (to pursue careers in their topics of interest), but also to the collective to exercise 'academic freedom' and critically evaluate society. Academics are traditionally seen not to follow any ideology other than the search for truth, and so can generally be trusted to advise society from a position as neutral experts, even if those truths may be uncomfortable (e.g., the need to divest from fossil fuels to prevent climate change or reduce interpersonal contact to control the spread of a pandemic).
The autonomous nature of academic research is reflected in the way that research is typically organized and rewarded in a largely individualistic way. Careers are mainly built on the qualitative prestige of individual contributions to the scientific record, meaning that any individual could aspire to pursue their own research program and achieve tenure and field-wide recognition. Researchers have always been aware that academia is competitive but traditionally there was no upper cap on the number of 'boffins' allowed: If your contributions were valuable, there was a place for you. But Ziman argues that this model was never going to be sustainable: In 1961, Derek de la Solla Price published a graph showing that the number of publications (and therefore researchers) was growing exponentially, as new discoveries produced more new questions and professors developed more prot\u00e9g\u00e9s to answer them. If this continued, we would eventually end up with more scientists than people on the planet! The limiting factor became money: eventually, academic research became too expensive to society to be allowed to keep growing. But in an age of perpetual 'progress', knowledge generation wasn't allowed to slow down. The answer is that the funders of academic research (primarily governments) needed the process to become more efficient.
As a consequence, Ziman's naturalistic description observes a change in the institution of research. He argues that we have now entered a stage of \"post-academic\" research, in which the culture and structure are morphing to a new model designed to achieve more efficiency in the process of turning funding into knowledge that is useful to society. A key part of this change is the collectivisation of research. No longer do individual researchers primarily pursue their own creativity and curiosity-driven academic interests, but instead work together on an industrial scale to tackle 'problems' or 'challenges' commissioned by research funders relevant to society's needs. Academics are strategically employed as a workforce of expert problem solvers, rather than appointed to a university as autonomous intellectuals driven by a thirst for knowledge.
Researchers are required to hype up and justify the potential impact and applications of their work to these problems (Hyland & Jiang, 2021). Pursing a deliberately 'useless' line of research like G.H. Hardy seems unfathomable in the hypercompetitive world of grant applications and would be seen as antithetical to producing knowledge 'efficiently'. The collectivisation of research also leads to specialisation and selects for different academic abilities. Researchers who have the organizational and social skills to act as effective managers of grant-winning and publication-generating teams of specialist post-docs are more likely to succeed than awkward 'eggheads' who don't know how to 'play the game'. Ziman summarises the post-academic ethos with the acronym 'PLACE': a research culture that increasingly produces Proprietary knowledge, is focused on Local problems rather than contributing to general understanding, is managed by a hierarchical Authority, works on Commissioned projects rather than long-term issues of curiosity, and is staffed with technical Experts rather than independent thinkers.
From his naturalistic perspective, Ziman refrains from speculating on the causes of these changes. The application of neoliberal government policy to the funding and organization of research and universities is surely a large contributor and has been the subject of significant study in the theory of \"academic capitalism\" (Jessop, 2018). However, the development of technology, the internet, and the growth of neoliberalism, legislation, and quantification in society more generally (which research exists as a part of) are also important facets. Ultimately there is no one single cause, but the accumulated changes are undeniable. But how does the scientific reform movement relate to this?
The science reform movement will no doubt be familiar to readers of this essay, and here I use the term broadly to refer to advocates for the widespread promotion and adoption of open research practices. For many seasoned open research advocates, entry into the reform movement began with an interest in methodological innovations (such as preregistration, placing datasets online and so on) to discourage questionable research practices (QRPs), which had become common tricks employed to survive the cycle of publish or perish and competition in academia (Bakker et al., 2012). Explanation and promotion of open research practices subsequently became the topic of dozens of papers, workshops, conferences, and editorials across nearly all academic disciplines.
But open research advocates realized that widespread adoption of these practices would require more than grassroots methods workshops: it would require changes to the structure and systems of academia itself (Nosek, 2019). There was increasing debate about the incentives of academia, and how they shaped researcher behaviour. If advocates really wanted researchers to conduct open research, then rewards would need to be introduced higher up the system to funders, journals, and institutions (Nosek, Spies, & Motyl, 2012). To integrate open research into the system, appropriate infrastructure would need to be built to accommodate data, materials, code, and preregistrations. Policies and procedures would need to be developed to record and codify the use of open practices to be able to reward them appropriately. Smashing the pi\u00f1ata of 'closed' research meant academia would have to catch all the valuable open research sweets that tumbled out and this meant developing the appropriate baskets and buckets to do so.
These sweets have also turned out to be chewier than anticipated. Formatting data, writing analysis code, using version control, and writing detailed pre-registrations are often complex skills that require knowledge and training beyond the scope of many researchers, particularly those already stretched to breaking point with untenable workloads (Hostler, 2023). As the science reform movement germinates into new disciplines, standard practices developed in the context of combatting QRPs in quantitative experimental social psychology studies require reworking for new epistemologies and methodologies (E. A. Bennett, 2021; Leonelli, 2023). Sophisticated open research, such as making research materials interoperable across disciplines and questioning the theoretical basis of preregistration decisions are difficult challenges for many academics \u2013 especially those who during the decades of 'replication crises' have been advised or trained to generate 'easy' publications rather than be critically reflective methodologists.
An increasingly common solution to this issue is to propose further specialization and collectivisation: if researchers cannot be expected to accommodate the significant demands of open research themselves, then they should be supported by colleagues who can help them with this. Accordingly, there is an increased focus on the 'team infrastructure roles' that support research (Carter et al., 2019) and increased prominence for the role of open research specialists such as data stewards, methodologists, and software developers (A. J. Stewart et al., 2021). Support for this shift comes from the very top: a recent high-level report by the UK Government science and technology committee on reproducible research recommended increased funding for these roles (UK Science, Innovation, and Technology Committee, 2023). Relatedly, many science reform commentators have argued that the way that research is assessed should also focus more on team contributions, and that funding should reward configured teams, rather than individuals, to reflect the added work that goes into conducting open research (A. Bennett et al., 2022; Tiokhin et al., 2021). Positions such as 'open research officer' are increasingly common in universities to assist academics with the demands of open research, and researchers are encouraged to build in support for open research from specialists into grant applications (e.g., University of Bristol, n.d.).
![\"\"](\"https://substackcdn.com/image/fetch/$s_!KlQK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca884cb1-e149-497b-aeb7-7b2b46fdaaef_1132x600.png\")
There are clear parallels between the scientific reform movement's discourse on policy and infrastructure, and the existing observable changes in the shift to post-academic research: Both advocate for a focus on teams and collectives rather than individuals; for specialization in methods, especially with technical skills; and for an increased role for 'research professionals' as team members to support the logistics of open practices for specific research projects. A quote from Stewart et al. (2021) clearly captures this sentiment: \"The days of the 'lone genius' as the model of a researcher are fading fast, if not gone already. Therefore, there should be wider support, recognition and reward of team-based research\" (p.3). Open research and team research are increasingly linked and promoted as complementary: open research facilitates team research and collaboration by enabling participation through transparency; team research facilitates open research by dividing the potentially significant extra workload and technical demands of openness between many individuals.
Of course, there is not perfect overlap between the ideas of scientific reform and the post-academic research ethos. Open research ideals are antithetical to another aspect post-academic research identified by Ziman, which is the increased collaboration with industry and greater generation of proprietary knowledge. Ziman noted that increasing collaboration with industry led to greater control over knowledge generated through academic research and a rise in 'spin off' academic companies designed to monetize findings. This mindset is certainly incompatible with open research ideals of free and open access to research data and materials (which may lead to tension, e.g. Fern\u00e1ndez Pinto, 2020). Ziman's description of post-academic research as focussing on solving interdisciplinary 'local' problems is also tangential to advocating for greater collaboration and team-based research \u2013 logically, teams of specialists could be directed towards solving fundamental, curiosity driven questions rather than practical problems. The large particle accelerator research at CERN is an example of a team-based project that performs curiosity driven research, such as seeking to test theoretical predictions of the existence of subatomic particles2.
![\"\"](\"https://substackcdn.com/image/fetch/$s_!5W7r!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ead46e6-bd55-4ba4-9b01-ebb0173b0d20_1000x401.jpeg\")
However, given a move away from autonomous individualized research is already a key element of post-academic research, then further collectivization and specialization (accelerated by science reformers) may contribute to a continued shift towards post-academic norms more broadly. Science reform advocates have called for adaptations to infrastructure and policy to accommodate open research practices and increased collectivization. But much of the groundwork for these changes has been laid already in the broader shift to post-academic research, with greater prominence for research professionals in the research process, as well as increases in research administration, and tracking and monitoring of research bids, outputs, and research data (Shepherd, 2008). Science reformers should be aware that utilizing these systems may lead to greater alignment with post-academic research norms in a broader sense, as well as facilitating open methodological practices.
Team-based, method-focused research is fundamentally different in many ways to traditional academic work, particularly when it comes to autonomy. There is the basic matter of compromise when working in larger teams that means it is more difficult for an individual to pursue their own ideas, and there is a difficulty in finding collaborators for ideas that may be questionable or controversial. Collaborating with non-academics also changes the dynamic of research. Existing research professionals (such as grant application managers) often lack autonomy in their jobs (A. Bennett et al., 2022) as their role is often tied more closely to supporting universities' 'research strategy', which typically supports specific types and topics of research and has a large focus on generating research income (Rees, 2015). As such, research professionals are often managed to prioritize supporting researchers whose work aligns with post-academic values, rather than quirky side projects. Expansion of these roles into open research may follow suit: employing core-funded technical research specialists to support academics to conduct research is a progressive move compared to the current precarious 'gig economy' of overworked temporary research positions (Nelson et al., 2020). However, such positions mean that universities may reserve greater control over how such specialists are deployed to support particular strategy-aligned projects, potentially limiting autonomy of researchers to pursue projects alone outside of this remit.
More broadly, open research reforms foreground issues centred around the research process. Several scholars have warned that a focus on methodological problems may neglect deeper philosophical issues in disciplines such as psychology (e.g., Haeffel, 2022; Muthukrishna & Henrich, 2019). However, a focus on methodology and producing transferable, reproducible knowledge is more amenable to a post-academic ethos focused on providing specialist technical solutions to specific local 'problems': It is not surprising that governments and funders are supportive of openness reforms to help improve the 'efficiency' of research and maximise value for money from research funding via open materials and data. Funding lone academics to indulge the whims of their curiosity and pursue careers musing on abstruse theoretical issues with questionable practical significance is less desirable or defensible in a neoliberal political climate.
As a naturalistic descriptor, 'post-academic research' is a valueless term and does not prescribe negative connotations: many aspects of post-academic research may be considered progressive. For example, there are arguments that the 'collectivisation' of research does not constitute a rethinking of the research process, but rather acknowledges the roles and effort that have always existed: It just makes their contributions visible and rewarded. There is certainly a substantial amount of truth in this. To a certain extent the idea of the 'lone genius' was always a myth: behind every Nobel Prize winner was a team of researcher assistants, lab managers, technicians, as well as individuals providing the 'soft' skills which smooth the process and contribute to the success of any piece of research. Recognising these roles is a progressive move to a more equitable research culture (Brand et al., 2015). Yet there is evidence that research teams have for a long time been getting larger, and single-author publications are increasingly rare (Thelwall & Maflahi, 2022). There has also been an undeniable growth in \"third space\" professionals in universities who academics are required to engage with to conduct research (Whitchurch, 2008). These research professional roles largely exist to support academics to navigate and conform to existing policy, legislation, technology, and administration controlled by non-academic entities, rather than to challenge them.
A second defence of post-academic and instrumental research is that it is problematic to use research funding (especially government funding indirectly paid for by tax-payers) to pay for research which does not have any clear benefit. Research is expected to have some articulable impact outside of academia to provide value for money and justify its cost (Doyle, 2018). Similar arguments have been used to justify open data, open materials, and open access (S. L. K. Stewart et al., 2022; UKRI, 2023): If tax-payers are funding our research, then we should aim to maximise the benefit from it and make the products of it available for re-use and scrutiny. Again, this argument has merit. Certainly, public funding should not be used to fund potentially fraudulent research, and open data and materials are often a necessary step for detecting this. However, the 'value for money' argument also relies on a utilitarian logic that prioritizes utility (or potential utility) above all else. The distinction between making data and publications open for scrutiny for bias and errors, and making them open on the basis of 'potential benefit' can become blurred in this discourse. The latter can subtly transform from an advantage of open research to a justifiable imperative, infused with the definitions of 'benefit' from non-academic sources such as funders and policymakers. In other words, it encourages and normalizes assessing research projects primarily on the benefit and impact generated from the researchers' use of resources.
A utilitarian discourse encourages applying neoliberal cost-benefit analysis to research which can lead to decisions to slash funding from non-instrumental research streams and disciplines, such as the Arts and Humanities, which in the UK has recently had its budget reduced by 50% (Weale, 2021). Conducting research for curiosity's sake alone, with no clear practical benefit, cannot be justified in such terms. Alternative human values, such as creativity, enlightenment, heritage, and tradition need to be invoked. But in a dangerous feedback loop, the disciplines and projects most likely to research and advocate for these values are the ones that are finding it harder to justify and find funding.
Reforms to methodological practices have challenged the way that researchers design studies and collect and analyse data, and these challenges have sometimes resulted in pushback from researchers. However, reforms that support the collectivisation of research and greater alignment with post academic norms impact not only the technical skills that researchers employ on a specific project, but their broader way of working and professional practice. More fundamentally, changes in academics' roles challenge academic identities.
There is a significant amount of scholarship of academic identity, particularly within an educational context. The industrialization and collectivization of academic research mirrors the industrialization and collectivization of academic teaching: with the 'massification' of higher education in many countries, the identity of the academic and the relationship between academics and students has changed considerably (Macfarlane, 2011). Many of the tasks that academics used to be responsible for, such as supporting students' career development and 'study skills', have been outsourced to professional services roles3. Macfarlane (2011) has argued that this \"unbundling\" of academic practice has led to a \"hollowing out\" of academic life and further propagates an instrumental and utilitarian mindset towards academic practice (e.g., students should be supported by professionals because the university needs to provide support as a contractual obligation, rather than as a humanistic value of academic citizenship). This hollowing out impacts the traditional identity of the academic as an 'all-round' intellectual and their role in a collegial culture. It raises questions about what it means to be an academic, the answers of which are strongly tied to academics' values and beliefs about the role of a university education (and universities more generally) in society (Henkel, 2005).
The impact of collectivization on research may be similar. In the shift to post-academic research, the definition of 'research' itself has changed (Macfarlane, 2021). Current discourse commonly describes \"research\" as a specific performative activity with a definable output. But traditionally, \"research\" referred to a culture of \"broader intellectual engagement\" (MacFarlane, 2021), encompassing everything from \"scholarly investigation, appreciation, creative and textual criticism, re-interpretation and a critical treatment of contemporary thought\" (Truscot, 1943). This broader definition supported the autonomy of researchers and an academic identity tied to the role of holding society account, birthed in the 'de-Nazification' of science after the second world war (Macfarlane, 2021). In a post-academic research culture, where teams of specialists compete for funding for research on specific pre-defined 'strategic priorities', academic identities are often in conflict. Researchers still espouse the importance of intellectual curiosity, collegiality, autonomy, and criticality, but often find these in conflict with a performative research practice centred around managing small contributions to multiple short-term entrepreneurial projects (T\u00fcl\u00fcba\u015f & G\u00f6kt\u00fcrk, 2023).
For Henkel (2005), autonomy and academic identity are \"integrally related\", but the meaning of autonomy has changed in a post-academic environment. The autonomy derived from sitting in a protected 'ivory tower' is harder to sustain as the boundaries of academia with industry, society, and funders have dissolved. Instead, in an increasingly collectivist research culture, autonomy is generated by managing relationships with those outside of academia to maintain a collective reputation which grants academics the authority to be critical. This model is likely to become more relevant as post-academic research progresses further.
Ziman believed that the shift to post-academic research had significant implications for the legitimacy of academia's authority on knowledge production and its critical role in society. He predicted certain aspects of the stereotypical 'legend' of science \u2013 that it somehow created knowledge that was more 'factual' than other means \u2013 would continue to come under increasing indefensible scrutiny \u2013 a prescient observation considering the growing \"decolonization of knowledge\" agenda, and criticisms of Western hegemony of knowledge production (Santos, 2008). Indeed, open research reforms are likely to accelerate this scrutiny further as the 'black box' of what researchers actually do to generate knowledge is revealed in preregistrations, open data, and open materials. High profile cases of errors and fraud are likely to become more common with increasing transparency. But Ziman discouraged promoting the idea that \"science\" had authority of knowledge on epistemic grounds. Rather, the culture and norms of academia supported an institution which generated knowledge in a way which was defensible on the grounds of being reasonable \u2013 researchers could articulate how their practice and culture reduced or acknowledged bias and errors, and therefore could argue that they produced a uniquely reliable kind of knowledge with value to society.
This view is what made Ziman concerned about the impact of post-academic norms on the cultural fabric of science. He was worried that the transformation of curiosity-driven independent academics into teams of expert problem-solvers working on commissioned projects threatened their autonomy. It was autonomy and the freedom to express intellectual curiosity that was the true basis of academia's authority on knowledge and for holding society to account. The autonomy was embedded into the culture and environment of academia that supported academics to uphold academic values. In post-academic research, the traditions and culture which support these values are continuing to change in numerous ways, accelerated by the science reform movement. It is clear that academic traditions of 'peer review', 'authorship', and the currency of research 'papers' are increasingly not fit for purpose in an open, digital, collectivist research culture. For example, open research reforms significantly increase the amount of 'research material' for scholars to consume - already so excessive that few can keep up \u2013 challenging traditional peer review mechanisms of scepticism and quality control (Hosseini et al., 2022). Larger 'team science' projects and CREDIT taxonomies (Holcombe, 2019) challenge traditional norms of who 'deserves' authorship and how credit is apportioned, disrupting old metrics and reward systems (Hosseini et al., 2022). A traditional focus on 'outputs' as indicators of prestige is increasingly replaced by a focus on the quality of the research process, reconfiguring how and when researchers should be assessed (Chambers & Tzavella, 2021). The values that these new traditions and cultures support \u2013 and whether they will enable the autonomy that is crucial to both academic success and identity \u2013 are yet to be established.
Ziman's naturalistic description of 'post-academic research' is a useful perspective from which to consider the scientific reform movement. The concept transcends the traditional level of analysis of meta-science to describe the inevitable and irreversible changes of an entire culture, and the identity and values that accompany it. The structural changes advocated for by the scientific reform movement are but a small tide in a larger sea change. However, they nevertheless play an important role, especially in accelerating the collectivisation of research through the increase in workload for open research, and in directly advocating for team-based science. Practical changes to the traditions of academic work impact on the identity, culture, and values of the institution. This in turn impacts how the institution is perceived by the society it exists in, raising existential questions about the role and purpose of academia itself. Post-academic research, as the 'post-' moniker implies, is a description of the present, irreversible reality. There is no 'going back' to the traditional model of academia. Ziman's argument was that those of us who go further into the new reality would have to reckon with how to sustain the role of intellectual curiosity, to defend academia's autonomy and role in society. \"The maintenance of the non-instrumental roles of science should be a primary consideration in debating every aspect of its future.\" (Ziman, 2003). In proposing changes to the structure of academia, this is something that science reformers should bear in mind.
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7(6), 543\u2013554. https://doi.org/10.1177/1745691612459060
Bennett, A., Garside, D., Gould van Pragg, C., Hostler, T. J., Kherroubi Garcia, I., Plomp, E., Schettino, A., Teplitzky, S., & Ye, H. (2022). A manifesto for rewarding and recognising Team Infrastructure Roles. Research Equals. https://doi.org/10.53962/knm3-bnvx
Bennett, E. A. (2021). Open Science From a Qualitative, Feminist Perspective: Epistemological Dogmas and a Call for Critical Examination. Psychology of Women Quarterly, 45(4), 448\u2013456. https://doi.org/10.1177/03616843211036460
Brand, A., Allen, L., Altman, M., Hlava, M., & Scott, J. (2015). Beyond authorship: Attribution, contribution, collaboration, and credit. Learned Publishing, 28, 151\u2013155. https://doi.org/doi:10.1087/20150211
Brink, C. (2018). The soul of a university: Why excellence is not enough. Bristol University Press.
Carter, S., Carlson, S., Crockett, J., Falk-Krzesinski, H. J., Lewis, K., & Walker, B. E. (2019). The Role of Research Development Professionals in Supporting Team Science. In K. L. Hall, A. L. Vogel, & R. T. Croyle (Eds.), Strategies for Team Science Success (pp. 375\u2013388). Springer International Publishing. https://doi.org/10.1007/978-3-030-20992-6_28
CERN. (n.d.). The CERN KT Forum: Definition and Mandate. CERN. https://kt.cern/sites/default/files/resource/other/ktforummandate/ktforummandate_2.pdf
Chambers, C. D., & Tzavella, L. (2021). The Past, Present, & Future of Registered Reports. Nature Human Behaviour, 6, 29-42. https://doi.org/10.1038/s41562-021-01193-7
de la Solla Price, D. (1961). Science since Babylon. Yale University Press.
Doyle, J. (2018). Reconceptualising research impact: Reflections on the real-world impact of research in an Australian context. Higher Education Research & Development, 37(7), 1366\u20131379. https://doi.org/10.1080/07294360.2018.1504005
Fern\u00e1ndez Pinto, M. (2020). Open Science for private Interests? How the Logic of Open Science Contributes to the Commercialization of Research. Frontiers in Research Metrics and Analytics, 5, 588331. https://doi.org/10.3389/frma.2020.588331
Haeffel, G. J. (2022). Psychology needs to get tired of winning. Royal Society Open Science, 9(6), 220099. https://doi.org/10.1098/rsos.220099
Hardy, G. H. (1908). Mendelian Proportions in a Mixed Population. Science, XXVIII, 49\u201350.
Hardy, G. H. (1945, 2005). A Mathematician's Apology. http://www.arvindguptatoys.com/arvindgupta/mathsapology-hardy.pdf
Henkel, M. (2005). Academic identity and autonomy in a changing policy environment. Higher Education, 49(1\u20132), 155\u2013176. https://doi.org/10.1007/s10734-004-2919-1
Hogan, J. (2011). Is higher education spending more on administration and, if so, why? Perspectives: Policy and Practice in Higher Education, 15(1), 7\u201313. https://doi.org/10.1080/13603108.2010.532316
Holcombe, A. O. (2019). Contributorship, Not Authorship: Use CRediT to Indicate Who Did What. Publications, 7, 48. https://doi.org/10.3390/publications7030048
Hostler, T. J. (2023). The Invisible Workload of Open Research. Journal of Trial & Error. http://doi.org/10.36850/mr5
Hosseini, M., Senabre Hidalgo, E., Horbach, S. P. J. M., G\u00fcttinger, S., & Penders, B. (2022). Messing with Merton: The intersection between open science practices and Mertonian values. Accountability in Research, 1\u201328. https://doi.org/10.1080/08989621.2022.2141625
Hyland, K., & Jiang, F. (Kevin). (2021). 'Our striking results demonstrate \u2026': Persuasion and the growth of academic hype. Journal of Pragmatics, 182, 189\u2013202. https://doi.org/10.1016/j.pragma.2021.06.018
Institute of Physics. (2023). Bin the Boffin. Institute of Physics. https://www.iop.org/sites/default/files/2023-05/Bin-the-Boffin-pamphlet-v2.pdf
Jessop, B. (2018). On academic capitalism. Critical Policy Studies, 12(1), 104\u2013109. https://doi.org/10.1080/19460171.2017.1403342
Leonelli, S. (2023). Philosophy of Open Science (1st ed.). Cambridge University Press. https://doi.org/10.1017/9781009416368
Macfarlane, B. (2011). The Morphing of Academic Practice: Unbundling and the Rise of the Para-academic: The Morphing of Academic Practice. Higher Education Quarterly, 65(1), 59\u201373. https://doi.org/10.1111/j.1468-2273.2010.00467.x
Macfarlane, B. (2021). The spirit of research. Oxford Review of Education, 47(6), 737\u2013751. https://doi.org/10.1080/03054985.2021.1884058
Muthukrishna, M., & Henrich, J. (2019). A problem in theory. Nature Human Behaviour, 3(3), 221\u2013229. https://doi.org/10.1038/s41562-018-0522-1
Nelson, G., Monson, M. J., & Adibifar, K. (2020). The gig economy comes to academia: Job satisfaction among adjunct faculty. Cogent Education, 7(1), 1786338. https://doi.org/10.1080/2331186X.2020.1786338
Rees, T. (2015). Developing a Research Strategy at a Research Intensive University: A Pro Vice Chancellor's Perspective. In R. Dingwall & M. McDonnell (Eds.), The SAGE Handbook of Research Management (pp. 565\u2013580). SAGE Publications Ltd.
Santos, B. de S. (Ed.). (2008). Another knowledge is possible: Beyond northern epistemologies. Verso.
Shepherd, S. (2008). Managerialism: An ideal type. Studies in Higher Education, 43(9), 1668\u20131678. https://doi.org/10.1080/03075079.2017.1281239
Spellman, B. A. (2015). A Short (Personal) Future History of Revolution 2.0. Perspectives on Psychological Science, 10(6), 886\u2013899. https://doi.org/10.1177/1745691615609918
Stewart, A. J., Farran, E. K., Grange, J. A., Macleod, M., Munaf\u00f2, M., Newton, P., Shanks, D. R., & the UKRN Institutional Leads. (2021). Improving research quality: The view from the UK Reproducibility Network institutional leads for research improvement. BMC Research Notes, 14(1), 458. https://doi.org/10.1186/s13104-021-05883-3
Stewart, S. L. K., Pennington, C. R., da Silva, G. R., Ballou, N., Butler, J., Dienes, Z., Jay, C., Rossit, S., Samara, A., & U. K. Reproducibility Network (UKRN) Local Network Leads. (2022). Reforms to improve reproducibility and quality must be coordinated across the research ecosystem: The view from the UKRN Local Network Leads. BMC Research Notes, 15(1), 58. https://doi.org/10.1186/s13104-022-05949-w
Thelwall, M., & Maflahi, N. (2022). Research coauthorship 1900\u20132020: Continuous, universal, and ongoing expansion. Quantitative Science Studies, 3(2), 331\u2013344. https://doi.org/10.1162/qss_a_00188
Tiokhin, L., Panchanathan, K., Smaldino, P. E., & Lakens, D. (2021). Shifting the level of selection in science [Preprint]. MetaArXiv. https://doi.org/10.31222/osf.io/juwck
Truscot. (1943). Red Brick University. Faber & Faber.
T\u00fcl\u00fcba\u015f, T., & G\u00f6kt\u00fcrk, \u015e. (2023). A meta-synthesis on academic identity in the neoliberal context of academy. Higher Education, 86(1), 139\u2013161. https://doi.org/10.1007/s10734-022-00906-9
UK Science, Innovation, and Technology Committee. (2023). Reproducibility and Research Integrity Report (Sixth report of the session 2022-2023). https://committees.parliament.uk/publications/39343/documents/194466/default/
UKRI. (2023). Open Research. https://www.ukri.org/what-we-do/good-research-resource-hub/open-research/
University of Bristol. (n.d.). How to Fund Research Software or Cost It into Grants. https://www.bristol.ac.uk/acrc/research-software-engineering/software-howtos/how-to-fund-software/
Weale, S. (2021, July 20). Funding cuts to go ahead for university arts courses in England despite opposition. The Guardian. https://www.theguardian.com/education/2021/jul/20/funding-cuts-to-go-ahead-for-university-arts-courses-in-england-despite-opposition
Whitchurch, C. (2008). Shifting Identities and Blurring Boundaries: The Emergence of Third Space Professionals in UK Higher Education. Higher Education Quarterly, 62(4), 377\u2013396. https://doi.org/10.1111/j.1468-2273.2008.00387.x
Ziman, J. (2003). Non-instrumental roles of science. Science and Engineering Ethics, 9(1), 17\u201327. https://doi.org/10.1007/s11948-003-0016-y
Ziman, J. M. (2000). Real Science: What It Is and What It Means. Cambridge University Press.
Hostler, T. (2023, August 8). Scientific reform, post-academic research, and academic identity. Critical Metascience. https://markrubin.substack.com/p/scientific-reform-post-academic-research
Tom Hostler is a lecturer at Manchester Metropolitan Univeristy. For more information about his work, please visit: https://scholar.google.com/citations?user=ZKMS8m0AAAAJ&hl=en To contact Tom, please email: t.hostler@mmu.ac.uk
![\"\"](\"https://substackcdn.com/image/fetch/$s_!m5TV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd074ff41-90e6-4a88-9421-162cb79141fa_256x243.jpeg\")
Not a universally admired term, as highlighted by the Institute of Physics recent \"Bin the Boffin\" campaign (Institute of Physics, 2023).
Although it should be noted that project has a dedicated \"knowledge transfer forum\", with the mandate to \"[ensure] that new technology disclosures presented by CERN\u2026 are brought to the attention of potentially interested industrial and academic partners in their country\", showing that funding is partly based on potential applied utility. (CERN, n.d.)
Yet paradoxically, this has not reduced workload: academics are still required to engage with these professionals, which takes significant time (Hogan, 2011).
At the end of the year, Codeberg will shut down its old Codeberg Pages server, in favor of the already deployed, git-pages-based alternative.1.\nFor many deployments, migration will be quite straight-forward, requiring little more than using a webhook, as outlined in the migration guide.\nAnd this page has already for a few months now been served by the git-pages server, without any problems.
There is only one little but for the migration path, which affects pages that so far relied on git lfs for storing larger asset files like images and that are trying to directly serve from a repository without a build stage:\ngit-pages does not (and probably never will) support git lfs.\nI ran into this problem (without noticing it) when I migrated my talks, which are all just HTML files generated locally by marpit, and for which I like keeping the images in the lfs. \nWhen I was preparing a new slide deck for a conference, I noticed that the images didn't show up when I wanted to share a draft with a friend for review.
Luckily, the at least temporary hot fix for that is quite easy:\nWhile git-pages doesn't support git lfs when serving directly from a repository via the webhook, one can use a CI action and the git-pages-cli to push a total folder. \nWhile normally more used for deploying the output of a static site generator (like this page), it can also be paired with a simple checkout action to grab a copy of the repository including the lfs objects and then push that whole folder via the git-pages-cli.
The simplest way of doing that is via writing a small Forgejo Action that triggers on a push, and is how I did it for my talks:
\n![\"\"](\"https://substackcdn.com/image/fetch/$s_!MULU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg\")
name: Publish\non:\n push:\n branches: [pages]\njobs:\n publish:\n runs-on: codeberg-tiny\n steps:\n - uses: actions/checkout@v7\n with:\n lfs: true\n - uses: actions/git-pages@v2\n with:\n site: https://talks.tzovar.as/\n server: codeberg.page\n token: $\n source: .\nAfter the checkout of the repository, I let it just immediately use the git-pages action to deploy the folder as is.\nIf for some reason you prefer doing the same with the Woodpecker CI, I made an equivalent plugin that can be used for that.
And of course if your pages already use a CI action to deploy generated artifacts, you'll not have to worry about this either way.\nIn the long term I should probably also just let the CI also build the slides, to ensure that the deployed HTML version reflects the actually committed files, but that's a modification for a different day.
\nIf you're not sure what method you are using, this small tool lets you check\u00a0\u21a9
\nThe LIS-Bibliometrics Committee is inviting expressions of interest from UK institutions interested in hosting the 2027 conference, as well as colleagues who would like to join the organising committee.
\n\n\n\nThe proposed conference dates are between mid-June and mid-July, or in early September 2027. Hosting or helping to organise the event offers an opportunity to connect with colleagues and contribute to the bibliometrics community.
\n\n\n\nThe deadline for expressions of interest is 6 November 2026.
\n\n\n\nWe would love to hear from you and welcome your involvement in making the next conference a success.
\n\n\n\nFind out more about the conference and how to get involved.
\n\n\n\nContact us: https://lisbibliometrics.wordpress.com/contact/
","doi":"https://doi.org/10.59350/g5rq3-kfd25","guid":"https://thebibliomagician.wordpress.com/?p=5687","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1792195200,"rid":"y4844-v1g52","summary":"Help shape the LIS Bibliometrics Conference 2027 The LIS-Bibliometrics Committee is inviting expressions of interest from UK institutions interested in hosting the 2027 conference, as well as colleagues who would like to join the organising committee. The proposed conference dates are between mid-June and mid-July, or in early September 2027.","tags":["Metrics"],"title":"LIS Bibliometrics Conference 2027 \u2013 Call for Expressions of Interest!","updated_at":1792248905,"url":"https://thebibliomagician.wordpress.com/2026/10/17/lis-bibliometrics-conference-2027-call-for-expressions-of-interest/","version":"v1"},{"authors":[{"affiliation":[{"name":"Durham University, Department of Psychology"}],"contributor_roles":[],"family":"Rubin","given":"Mark","url":"https://orcid.org/0000-0002-6483-8561"}],"blog":{"authors":[{"name":"Mark Rubin","url":"https://orcid.org/0000-0002-6483-8561"}],"community_id":"3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6","created":1681948800,"current_feed_url":null,"description":"Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.","doi":"https://doi.org/10.59350/markrubin","favicon":"https://rogue-scholar.org/api/communities/3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6/logo","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"markrubin","status":"active","subfield":"1207","title":"Critical Metascience","updated":1792151011,"use_api":true},"blog_name":"Critical Metascience","blog_slug":"markrubin","content_html":"I really enjoyed Mel Andrews' recent essay: \"Philosophy in the Trenches and Laboratory Benches of Science\" and its main point that \"every laboratory needs a philosopher.\" This point is made in the context of a concern about \"the industrialisation of science\":
Not all of the shifts of recent history have constituted improvements to science, in either its truth-seeking capacity or its social responsibility. I allude to the industrialisation of science. To the corporatisation, the commodification, and the militarisation of science\u2026.
The so-called replication crisis has come about, in large part, because generations of behavioural scientists have been instructed in assembly-line procedures rather than modes of questioning and critical analysis.
This concern about the industrialisation of science reminded me of something that statistician Ronald Fisher wrote many years ago when he argued that the Neyman-Pearson approach to hypothesis testing is more suitable for \"acceptance procedures\" used during quality control on industrial production lines than for scientific research:
I am casting no contempt on acceptance procedures, and I am thankful, whenever I travel by air, that the high level of precision can really be achieved by such means. But the logical differences between such an operation and the work of scientific discovery by physical or biological experimentation seem to me so wide that the analogy between them is not helpful, and the identification of the two sorts of operation is decidedly misleading (Fisher, 1955, pp. 69\u201370).
As I've discussed elsewhere, the production line analogy assumes an inappropriate amount of knowledge and certainty on the part of scientists (Rubin, 2020). Unlike quality controllers on a factory production line, scientists don't know the relevant and irrelevant features of their populations and testing conditions. Factors that they theorize to be essential for demonstrating an effect may turn out to be unimportant, and factors that they assume to be unimportant may turn out to be essential! Indeed, one of the key differences between scientific research and quality control is that scientists can learn what's important and unimportant from their replication failures, whereas quality controllers merely consign their \"bad batches\" to the garbage. Hence, like Andrews, I see a clear link between the replication crisis and the industrialisation of science (e.g., Rubin, 2021, p. 5829).
Andrews also describes the industrialisation of science as a:
broad-sweeping prohibition against \"theorising\" \u2013 and by this it is meant any and all forms of sustained critical thought about or questioning of the scientific work
Again, this description rings true for me. In particular, it reminds me of Kerry Chamberlain's (2000) description of \"methodolatory,\" which consists of:
An overemphasis on locating the 'correct' or 'proper' methods
A focus on description at the expense of interpretation
The avoidance of theory, and
Avoidance of the critical
As Chamberlain explained \"methodolatry\u2026works to prevent us looking at the assumptions behind our research.\" Andrews' proposal of a philosopher-in-the-lab should provide an important safeguard against methodolatory:
Every laboratory needs to have someone who understands broadly and intimately what is going on and who is empowered to ask questions, to stand back from the laboratory bench or the telescope or the command line and ask: What are we really trying to determine here, and to what end? Will it be of benefit or detriment to mankind? Are the assumptions and idealisations that we have made along the way warranted? Where do our driving theories, models, and representations of the phenomena under study stand in relation to rival or complementary conceptualisations? How are we best to interpret and contextualise the results of our modelling or experimental work? Are our instruments of measurement and analysis, our conceptual repertoire, up to the task at hand?
Andrews concludes that:
Every laboratory needs a philosopher. The philosopher's role, however, is not only to philosophise on behalf of scientists, but to re-educate scientists in how to philosophise for themselves.
This seems like a good idea. In recent years, statisticians and methodologists have made excellent contributions to re-educating scientists in the wake of the replication crisis. Philosophers should be key members of this educational team!
Andrews, M. (2023). \"Philosophy in the trenches and laboratory benches of science\": An essay by Mel Andrews. The Philosopher, 111(1). https://www.thephilosopher1923.org/post/philosophy-in-the-trenches-and-laboratory-benches-of-science
Sabina Leonelli's new book \u2013 \"Philosophy of Open Science\" \u2013 will be published later this year. However, there is an open access preprint available on the PhilSci Archive here.
In the book, Leonelli provides \"a constructively critical reading\" of the standard approach to open science which, she argues, is focused on sharing \"objects\" such as data and materials. In Leonelli's view, this object-sharing approach has become an obstacle to promoting open science. Grounded in philosophies of science from Popper, Chang, Longino, and others, she puts forward an alternative, process-oriented view of open science that refers to efforts to establish \"judicious connections among systems of practice.\"
I propose a conception of openness as judicious connection, which is grounded in a process-oriented epistemology of science that recognises the situated, embodied and goal-directed nature of communication and collaboration among researchers.
What does \"connection\" have to do with openness?
The idea of openness is quintessentially linked to that of learning as going beyond one's boundaries. This arguably applies to systems of research practice as much as to research groups and individual learners: establishing new connections often means expanding one's learning, challenging existing assumptions around what is considered external or irrelevant to a given system, and considering whether new boundaries need to be established that incorporate the novel relations.
Importantly, Leonelli does not equate an indiscriminate increase in scientific connections with an increase in open science. Instead,
the building and maintenance of connections need to be judicious: they require skilled deliberation, whereby the new opportunities offered by the connection in question are evaluated within the contexts at hand. Indeed, openness can itself be understood as a dynamic and highly situated mode of valuing the research process and its outputs, which encompasses economic as well as scientific, cultural, political, ethical and social considerations (Levin and Leonelli 2017).
The differences between the object-sharing and \"judicious connection\" approaches to open science also have implications for inclusion. The standard object-sharing approach assumes that greater transparency leads to a more inclusive science. In contrast, Leonelli's \"judicious connection\" approach flips the causal arrow and assumes that more inclusion leads to greater transparency.
The implementation of OS [open science] needs to start from consideration of what it may take to make research more inclusive, diverse and just \u2013 rather than expecting such an outcome to naturally follow from the 'right' choice of software, infrastructures, standards, publishing platforms, or whichever other technological or institutional fix is being devised to facilitate access to resources.
Finally, Leonelli's distinction between the two approaches to open science is important because she believes that the standard, object-sharing approach:
threatens to blindly privilege specific ways of knowing, thus potentially disrupting sophisticated methodologies, inadvertently dismissing well-established research traditions, and exacerbating the already large epistemic and social divides separating research domains and locations As denounced by a number of critics in science and science studies, there is a substantive risk of some OS policies \u2013 despite their good intentions and progressive slant - acting as a reactionary force which reinforces conservatism, discrimination, commodification and inequality in research, thus ultimately closing down opportunities for inquiry in a disastrous reversal of what they set out to achieve.
I won't go any further into the distinction between the object-sharing and judicous connection approaches to open science. Leonelli already does an excellent job of that in her book. Instead, I want to highlight five quotes that are critical of the standard object-sharing approach, because these are motivating towards a consideration of her alternative judicious connection approach.
Many of the more institutionalised OS initiatives tend to privilege a homogenous, universally applicable understanding of the scientific method over a pluralistic and situated one.
Similar points have been made by several others. For example, Malich and Rehmann-Sutter (2022) pointed out that:
Metascience often construes science and psychology from a homogenizing perspective that omits the plurality and complexity of both scientific methods and psychological approaches. This omission entails several epistemic risks because the reforms that metascience initiates or accompanies can reinforce these homogenizing tendencies
Open Methods is not a matter of recording and sharing every detail of a research procedure, but rather a reflection on which research components and techniques are most salient to the outcomes, and should thus be accessible and reproducible.
I totally agree. As I argue near the end of this presentation, I'm opposed to what I call \"mindless transparency.\"
By the way, I think Popper (2002, p. 230) would also agree with this point:
When the judge tells a witness that he should speak 'The truth, the whole truth, and nothing but the truth', then what he looks for is as much of the relevant truth as the witness may be able to offer. A witness who likes to wander off into irrelevancies is unsatisfactory as a witness, even though these irrelevancies may be truisms, and thus part of 'the whole truth'.
Taking a monolithic understanding of reproducibility as a demarcation strategy for the whole of science side-steps the precious plurality of methods developed to suit specific goals, concepts and target objects.
Again, this point has been made by several others (e.g., Devezer et al., 2019, 2021; Feest, 2019; Haig, 2022). Nonetheless, replication is often portrayed as a \"defining\" or \"essential\" aspect of science.
A case in point is the skepticism towards qualitative research often displayed within debates on reproducibility, which sometimes portray qualitative methods as hopelessly subjective and devoid of rigorous forms of data collection and verification....Assuming that qualitative research traditions in anthropology or sociology are incapable of rigorous research means drastically reducing the diversity of systems regarded as exemplars of good practice.
Yes! We need to be very careful that open science does not reopen \"'sectarian' divides between quantitative and qualitative\" research, as Dunleavy (2022) has cautioned.
To date, some parts of the OS movement \u2013 particularly its institutionalized, top-down incarnations - have paid too much attention to designing procedures and technologies for sharing, and this has come at the expense of strategies, training and procedures to assess who is included and excluded from such apparatus, understand why and with which implications, and mitigate eventual instances of epistemic injustice.
Another important point! A good example of this lack of attention to inclusion comes from Prosser et al. (2022) in their article \"When open data closes the door.\" These researchers found surveyed 261 social psychology journals and found that journal guidelines for sharing data were exclusionist with regards to qualitative data:
Guidelines explicitly addressing open qualitative data were very rare, provided by just three journals (2.1% of all relevant journals). All other journals that made reference to 'open data' did so without making a distinction between qualitative and quantitative data.\"
Leonelli, S. (2023). Philosophy of open science. [Preprint] http://philsci-archive.pitt.edu/21986/
Order the published print version here.
The University of Exeter's PHIL_OS project (2021\u20132025): \"Aims to develop an empirically grounded philosophy of Open Science [OS] that emphasises the diversity of research environments around the world and articulates the conditions under which OS can leverage such diversity to promote good research practice.
","doi":"https://doi.org/10.59350/x4g76-zb210","guid":"116187642","image":"https://substack-post-media.s3.amazonaws.com/public/images/564ba545-18df-463c-8bd3-65f469e19f6d_296x446.png","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1682035200,"rid":"pkrx0-kj258","summary":"Sabina Leonelli's new book \u2013 \"Philosophy of Open Science\" \u2013 will be published later this year. However, there is an open access preprint available on the PhilSci Archive here. In the book, Leonelli provides \"a constructively critical reading\" of the standard approach to open science which, she argues, is focused on sharing \"objects\" such as data and materials.","title":"Sabina Leonelli's \"Philosophy of Open Science\"","updated_at":1792238140,"url":"https://markrubin.substack.com/p/sabina-leonellis-philosophy-of-open","version":"v1"},{"authors":[{"affiliation":[{"name":"Durham University, Department of Psychology"}],"contributor_roles":[],"family":"Rubin","given":"Mark","url":"https://orcid.org/0000-0002-6483-8561"}],"blog":{"authors":[{"name":"Mark Rubin","url":"https://orcid.org/0000-0002-6483-8561"}],"community_id":"3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6","created":1681948800,"current_feed_url":null,"description":"Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.","doi":"https://doi.org/10.59350/markrubin","favicon":"https://rogue-scholar.org/api/communities/3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6/logo","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"markrubin","status":"active","subfield":"1207","title":"Critical Metascience","updated":1792151011,"use_api":true},"blog_name":"Critical Metascience","blog_slug":"markrubin","content_html":"In this new article, I consider questionable research practices in the field of metascience. A questionable metascience practice (QMP) is a research practice, assumption, or perspective that's been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science. My aim is not to cast aspersions on the field of metascience but to encourage a deeper consideration of its more questionable research practices, assumptions, and perspectives.
Rejecting or ignoring self-criticism: Rejecting or ignoring criticisms of metascience and/or science reform
Fast 'n' bropen scientific criticism: A quick, superficial, dismissive, and/or mocking style of scientific criticism
Overplaying the role of replication: Assuming that replication is essential to science, and that it indexes \"the truth\"
Unspecified replication rate targets: Assuming a replication rate is \"too low\" without specifying an \"acceptable\" rate
Metabias: An unacknowledged bias towards explaining the replication crisis in terms of researcher bias
A bias reduction assumption: Focusing on selective reporting as the primary form of researcher bias and assuming that it can be reduced without increasing other forms of bias
Devaluing exploratory results: Devaluing an exploratory result as being more \"tentative\" than a confirmatory result without considering other relevant issues (e.g., quality of associated theory, methods, analyses, transparency)
Presuming QRPs are problematic: Presuming that questionable research practices (e.g., HARKing) are always problematic research practices
A focus on knowledge accumulation: Conceiving knowledge accumulation as the primary objective of science without considering (a) the role of specified ignorance or (b) different objectives in other philosophies of science
Homogenizing science: Focusing on specific approaches (e.g., quantitative methods; replicable effects) as \"the scientific method\"
In my article, I stress that only some metascientists engage in some QMPs some of the time, and that these QMPs may not always be problematic. Research is required to estimate the prevalence and impact of QMPs. In the meantime, I think that QMPs should be viewed as invitations to ask questions about how we go about doing better metascience.
Rubin, M. (2023). Questionable metascience practices. Journal of Trial and Error. https://doi.org/10.36850/mr4
My article is part of a special issue in the Journal of Trial and Error: \"Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction? https://journal.trialanderror.org/pub/callscientificreform/release/4
Many of the articles that I cite in my article are included in this chronological list of critical metascience articles: https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/list-of-critical-metascience-articles
The December 2022 issue of the journal Industrial and Organizational Psychology includes a very interesting discussion of the pros and cons of open science. There's a target article titled \"Open science, closed doors: The perils and potential of open science for research in practice\" (Guzzo et al., 2022) and nine commentaries on this article. You can find all 10 papers here.
In this post, I focus on Andrea Bazzoli's (2022) excellent commentary: \"Open science and epistemic pluralism: A tale of many perils and some opportunities.\" Many thanks to Andrea for drawing my attention to his work on Twitter. I elaborate on some of his points with my own thoughts inspired by Sabina Leonelli's (2022) recent work.
In his article, Bazzoli distinguishes between two research paradigms:
Postpositivism: A research paradigm holding that a knowable, tangible, and measurable reality exists (i.e., na\u00efve or critical realism), knowledge claims about this reality can be developed objectively, and verification/falsification of a priori hypotheses is the most prevalent methodological choice.
Constructionism: A scholarly movement that holds that reality is the result of communicative processes that create a sense of shared reality (i.e., it is locally co-constructed), emphasizes that objectivity is also co-constructed through communicative processes, and aims to examine taken-for-granted realities that might be oppressive or dysfunctional through future-forming, dialogic, hermeneutical, and dialectical research to generate new functional realities.
You can read a bit more about these and other research paradigms in Moin Syed's (2020) blog post.
It's important to distinguish between these different research paradigms and their associated research communities because, as Bazzoli explains,
undeclared and unexamined assumptions that are, by definition, self-evident to a community can be seen as impositions by other communities that do not share them (Romaioli & McNamee, 2021). In turn, a given community and its members will defend as reasonable arguments that logically descend from those assumptions against competing communities.
Bazzoli argues that the open science movement is based on the postpositivist research paradigm. Hence, for example, the open science principles of (a) providing a verifiable distinction between hypothesis generation and hypothesis testing, (b) reducing researcher bias, and (c) increasing reproducibility can be seen \"as reasonable arguments that logically descend from\" a postpositivist epistemology.
Bazzoli argues that:
the open science movement, as a direct offspring of (post)positivist research paradigms, has the potential to stifle epistemological and scientific pluralism and reproduce historical scientific hierarchies it purports to redress.
He's speaking here of postpositivism as a dominant epistemology in the sciences and arguing that advocating open science also advocates postpositivist epistemology, bolstering its dominance and, potentially, distorting and displacing minority research paradigms that are based on non-postpositivist epistemologies such as constructivism. Hence, at least in its postpositivist form, open science has the potential to stifle scientific pluralism. Bazzoli provides a useful practical example:
There is a movement, especially in Northern Europe, that aims to link adherence to open science practices to tenure and promotion.
For example, in their article \"Nudging open science,\" Robson et al. (2021) proposed that:
evaluating candidates for hire and promotion in faculties can also be modified to weight open practices more heavily. Such changes are likely to encourage researchers to reconsider their practices and can increase the perception that open practices are not only normative but also valued (Nosek, 2019). Job listings and interview questions could ask for evidence of open practices or for the candidate to illustrate commitment to transparent research in their cover letter. It may also be fruitful to ask candidates to provide an annotated CV detailing whether their articles have been preregistered or openly accessible, what datasets they have openly shared\u2026
These \"nudges\" may encourage the adoption of open science practices such as preregistration and data sharing. However, they may also encourage the adoption of open science's postpositivist epistemology. Hence, as Bazzoli points out,
open science serves as a tool to reinforce postpositivism's hegemonic position in I-O [industrial and organisational] psychology and beyond, to the detriment of other epistemologies.
To be clear, there's nothing wrong with \"nudging\" postpositivist researchers to adopt open science research practices. The concern is with nudging non-postpositivist researcher to adopt open science practices. Two potential problems with this approach are assimilation and marginalisation (see also Bazzoli, 2022, p. 527):
Assimilation: Researchers import postpostivist epistemology into their non-postpositivist research paradigms in order to justify their use of newly adopted open science practices. Consequently, their paradigm loses some of its distinctive aspects.
Marginalisation: Non-postpositivist paradigms that don't adopt open science principles and practices suffer a reduction of resources as a result (e.g., a reduction in jobs, funding, journal space, etc.).
The case of qualitative research has proven to be an important testing ground for the applicability of standard open science practices. What type of data should qualitative researchers share and not share? Is it useful to preregister qualitative studies? Should we expect qualitative results to replicate? The answers to these questions are controversial (for recent discussions, see Branney et al., 2023; Steltenpohl et al., 2021) and, as Bazzoli observes, the controversy runs deeper than simply identifying whether a researcher's data is represented by numbers or words. It reflects a more fundamental difference in epistemic stances between quantitative and qualitative researchers:
We suggest that it is not qualitative methods, per se, that are at odds with open science principles (as Guzzo et al., 2022, and Pratt, Kaplan, & Whittington, 2020, among others, argued), but the universalistic and all-encompassing claims that are tied to open science are incompatible with epistemic stances most often embraced by qualitative scholars.
In other words, postpositivist open science practices may not fit the non-postpositivist epistemologies employed by qualitative researchers, and indiscriminately encouraging these researchers to adopt such practices may inadvertently compromise their epistemological approach. As Branney et al. (2023) point out,
qualitative researchers, who are increasingly required to engage in open science practices, may feel their research is being judged against quantitative criteria that are inappropriate, irrelevant, or incompatible (Brooks et al., 2018a; Pratt et al., 2020).
One response to these sorts of concerns is to reassure researchers that the adoption of open science practices is not mandatory. For example, in their commentary on the Guzzo et al. (2022) article, H\u00fcffmeier et al. (2022) explained that:
it is neither the idea underlying OSPs [open science practices] to force researchers to engage in unwanted research practices, nor is there any movement in such a direction.
Another response is to stress that the principles of open science are flexible and adaptive to different disciplinary approaches. For example, in their commentary, Rudolph and Zacher (2022) explained that:
openness is not a monolithic \"one size fits all\" approach to either science or practice.
Both points are true. Nonetheless, it remains the case that the open science movement is promoting a set of policies and strategies that are designed to reform scientific culture in ways that are consistent with a postpostivist research paradigm. Hence, although researchers are free to choose (a) which open science practices to adopt and (b) how to implement them, they are being encouraged to adopt research practices that are designed for, and carry the assumptions of, a postpositivist research paradigm. Again, this approach is not problematic when researchers are already working within postpositivist research paradigms. The problem only arises when they're working within non-postpositivist paradigms. In this latter case, we might talk about postpositivist \"overreach\" or \"encroachment\" into non-postpositivist research paradigms. (For some examples, please see p. 14 of Beaumont and Coning's (2022) article: \"Coping with complexity: Toward epistemological pluralism in climate\u2013conflict scholarship.\")
The potential for open science's postpositivist encroachment on non-postpositivist research paradigms may be heightened by two factors. First, scientists may be motivated to universalize their scientific approaches. As Feyerabend (1975) noted:
Scientists are not content with running their own playpens in accordance with what they regard as the rules of scientific method, they want to universalize these rules, they want them to become part of society at large and they use every means at their disposal - argument, propaganda, pressure tactics, intimidation, lobbying - to achieve their aims.
In the case of open science, the motive for widespread propagation of open science practices is an honourable one: to improve the quality, rigour, and credibility of science. The concern is that, although these principles may benefit postpositivist research paradigms, they may be detrimental to non-postpositivist paradigms.
Second, because of its dominance and majority acceptance, the postpositivist research paradigm exerts its influence in a relatively subtle and unacknowledged way. As Kincheloe and Tobin (2015) explained:
many of the tenets of positivism are so embedded within Western culture, academia, and the world of education in particular that they are often invisible to researchers and those who consume their research\u2026.The crypto-positivist ideology is not perceived as oppressive and its role in quashing [non-positivist] research is not acknowledged.
Hence, \"crypto-positivism\" may mislead us into believing that open science principles and practices are universally applicable across all of science when, in fact, those principles and practices are limited to the postpositivist paradigm. This assumption of universality may help to explain universalist slogans such as \"open science; just science done right.\"
In summary, despite the fact that open science practices are neither mandatory nor one-size-fits-all, there may be an unconscious motive to encourage the uptake of these practices within non-postpositivist research paradigms because they're believed to be beneficial for science in general. Again, there's no suggestion of any ill-intentions on the part of the open science movement here. On the contrary, the open science community is clearly attempting to improve the quality, rigour, and credibility of science on the basis of \"reasonable arguments that logically descend from\u2026[their postpositivist] assumptions\" (Bazzoli, 2022). The concern is that these assumptions do not necessarily apply in the case of non-postpositivist research paradigms.
One solution to these potential problems is to open up open science to make it more inclusive and applicable to scientists from non-postpositivist research cultures. In other words, instead of adopting an epistemically monist and assimilatory approach to open science, we could adopt a pluralist and multicultural approach. What might pluralist open science look like? Bergmann's (2023) recent post on The Buffet Approach to Open Science provides a good starting point:
Open science practices are not all or none, you can pick and choose, match and mix, and do what's most suitable to your career stage, project, lab, and level of support.
I'd suggest a couple of modifications to the original buffet approach to open science to make it a bit more inclusive. First, we should allow researchers from different research paradigms to create their own open science practices (cook their own dishes!) rather than merely pick and choose from a selection provided by postpositivists. Second, like Bazzoli, I'd argue that researchers from different paradigms should be free to pick and choose (and create) their own open science principles as well as their associated practices. For example, the principle of reproducibility may not apply to non-postpositivist research paradigms. As Bazzoli points out:
constructionist epistemologies are not concerned with replicability of findings at all.
In her new book, \"The Philosophy of Open Science,\" Leonelli (2022) explains that the open science movement's monist view of science is easier to implement but less well suited to address the \"rampant plurality\" that exists on the ground:
Many of the more institutionalised OS [open science] initiatives tend to privilege a homogenous, universally applicable understanding of the scientific method over a pluralistic and situated one. It is much easier to set up OS guidelines when assuming that science consist of a coherent body of knowledge and procedures that can and should conform to common norms \u2013 an assumption that flies in the face of the rampant plurality of research approaches used across domains, locations and contexts.
Certainly, opening up open science within multiple epistemological cultures will be harder than the monist approach because (a) it involves changing multiple scientific cultures, rather than a single scientific culture, and (b) open science principles and practices need to be customised to the needs of particular research paradigms. Opening up open science is also likely to lead to heated discussions between and within different research paradigms about \"what 'open' and 'science' mean\" (Bazzoli, 2022, p. 527). As Leonelli (2022) explains:
a key challenge for OS is to productively manage the clash between different interpretations and operationalisations of openness, which emerge from diverse systems of practice with unequal levels of influence and visibility.
One approach to managing this clash is to be more open about not only our methods and data, but also the previously \"undeclared and unexamined assumptions\" of our research paradigms (Bazzoli, 2022). It's only after a research paradigm's epistemic rationales for open science principles and practices are made explicit that its interpretations and operationalisations of openness can be fully understood and appreciated by scientists from other paradigms. Furthermore, it's only after \"judicious connections\" (Leonelli, 2022) are made across paradigmatic interpretations and operationalisations that we'll arrive at a genuinely pluralist open science.
Bazzoli, A. (2022). Open science and epistemic pluralism: A tale of many perils and some opportunities. Industrial and Organizational Psychology, 15(4), 525-528. https://doi.org/10.1017/iop.2022.67
You can download a PDF version of this article here.
The reference for this article is as follows:
Rubin, M. (2023, May 4). Opening up open science to epistemic pluralism: Comment on Bazzoli (2022) and some additional thoughts. Critical Metascience. https://doi.org/10.31222/osf.io/dgzxa
","doi":"https://doi.org/10.59350/b20k8-0sr15","guid":"117317477","image":"https://substack-post-media.s3.amazonaws.com/public/images/5cb4484b-0901-48a0-8ff7-602e98b0f182_539x551.png","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1682899200,"reference":[{"id":"https://doi.org/10.31222/osf.io/dgzxa","unstructured":"Rubin, M. (2023). Opening up open science to epistemic pluralism: Comment on Bazzoli (2022) and some additional thoughts. Center for Open Science."}],"rid":"0qcn3-w2j73","summary":"Comment on Bazzoli (2022)","title":"Opening Up Open Science to Epistemic Pluralism","updated_at":1792238135,"url":"https://markrubin.substack.com/p/opening-up-open-science-to-epistemic","version":"v1"},{"authors":[{"affiliation":[{"name":"Durham University, Department of Psychology"}],"contributor_roles":[],"family":"Rubin","given":"Mark","url":"https://orcid.org/0000-0002-6483-8561"}],"blog":{"authors":[{"name":"Mark Rubin","url":"https://orcid.org/0000-0002-6483-8561"}],"community_id":"3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6","created":1681948800,"current_feed_url":null,"description":"Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.","doi":"https://doi.org/10.59350/markrubin","favicon":"https://rogue-scholar.org/api/communities/3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6/logo","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"markrubin","status":"active","subfield":"1207","title":"Critical Metascience","updated":1792151011,"use_api":true},"blog_name":"Critical Metascience","blog_slug":"markrubin","content_html":"Tom Hostler has just published an article titled \"The invisible workload of open research\" in the Journal of Trial and Error's special issue on the \"Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction?\" It's an insightful and thought-provoking piece that uncovers the potential workload costs of open research, considers why these costs may be ignored by university management, and makes some useful recommendations for moving forward. Here's a few quotes to whet your appetite.
The benefits of open research practices have been widely discussed (Munaf\u00f2 et al., 2017), and generally speaking researchers have positive attitudes towards adopting open research practices, finding them worthwhile (Eynden et al., 2016; Lowndes et al., 2017), and recommending their use to others (Sarafoglou et al., 2022). However, high time cost is repeatedly identified as one of the main barriers to the adoption of open practices (Eynden et al., 2016; Gownaris et al., 2022; Tenopir et al., 2011) and time is the main thing that researchers lack, with workloads at capacity across the sector, having already reached \"untenable\" levels (Long et al., 2020; Papadopoulos, 2017).
In short, implementing open science reforms take work, and work takes time, which academics have very little of!
The theory of academic capitalism suggests that the responsibility for addressing potential discrepancies between modelled and actual workload will not be taken up voluntarily by university management, who may at best ignore such issues, or at worse tacitly approve of them as a form of capitalist efficiency (Lyons & Ingersoll, 2010). In other words, if university management can choose not to incorporate open research into workload models, then they won't.
I don't think many working academics need to refer to the theory of academic capitalism to appreciate this point! But to quell any doubt, Hostler refers to empirical work by Kenny and Fluck (2022) and Miller (2019). For example, Kenny and Fluck (2022) found that:
Management knows that academics will do research (and other work) in their own time and exploit that\u2026.What is written on paper in terms of hours re admin or research does not correspond to the actual time spent. It all looks fine on paper but has enormous hidden workloads.
Hence, Hostler concludes:
There is a high potential for expectations to engage in open research to become widespread, but without additional time on academics' workloads dedicated to the activity.
Suggestions that concerns about the workload of open research are an example of a \"myth\" (Bastiaansen, 2019) or a \"misconception\" that can be corrected by \"positive advocacy\" (Hagger, 2022) are unhelpful to having honest conversations about the practical negative consequences of conducting open research.
This is an important point. Some open science advocates engage in a pretty hard sell, and this can include omitting or downplaying some of the costs of open science. However, like Hostler says, it's worth having \"honest conversations\" about the pros and cons of open science in order to set more realistic expectations.
I've just picked out one of Hostler's recommendations below to keep things brief, but note that he makes several more in his article:
When promoting open research reforms in universities or conducting open practices themselves [open research advocates] should advocate not only for investments in open research infrastructure and training but to receive extra time in workload allocations to acknowledge the additional burden of open research.
Getting open research work acknowledged in university workload models is likely to be easier said than done! As Hostler explains, open research work often consists of many small tasks, rather than a few large tasks, and so it's easily overlooked in workload models. Nonetheless, cumulatively, open research workload can be substantial and, as Hostler notes, if it's ignored, we risk \"death by a thousand 10-minute tasks\" (Bozeman et al., 2021).
Hostler, T. J. (2023). The invisible workload of open research. Journal of Trial & Error. https://doi.org/10.36850/mr5
","doi":"https://doi.org/10.59350/8ch32-n6h54","guid":"119984535","image":"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1683504000,"rid":"bmd6h-jds77","summary":"Comments on Hostler (2023)","tags":["Metascience","Workload Models","Open Science","Academic Workload","Science Reform"],"title":"Open Science and Overwork","updated_at":1792238133,"url":"https://markrubin.substack.com/p/open-science-and-overwork","version":"v1"},{"authors":[{"affiliation":[{"name":"Durham University, Department of Psychology"}],"contributor_roles":[],"family":"Rubin","given":"Mark","url":"https://orcid.org/0000-0002-6483-8561"}],"blog":{"authors":[{"name":"Mark Rubin","url":"https://orcid.org/0000-0002-6483-8561"}],"community_id":"3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6","created":1681948800,"current_feed_url":null,"description":"Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.","doi":"https://doi.org/10.59350/markrubin","favicon":"https://rogue-scholar.org/api/communities/3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6/logo","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"markrubin","status":"active","subfield":"1207","title":"Critical Metascience","updated":1792151011,"use_api":true},"blog_name":"Critical Metascience","blog_slug":"markrubin","content_html":"I discuss Van Drimmelen's (2023) Metascience2023 presentation on researchers' decision making during the research process. In particular, I consider his evidence that researchers' discretion over research decisions is unavoidable when they follow research plans that are either overdetermined (i.e., too prescriptive) or underdetermined (i.e., too vague). I argue that this evidence points to a prescriptiveness trade-off when writing preregistered plans: All other things being equal, plans that are more prescriptive are more likely to result in deviations that turn their confirmatory tests into exploratory tests, and plans that are less prescriptive are more likely to result in confirmatory tests that are susceptible to questionable research practices. I also consider Van Drimmelen's idea that researchers may make unconscious, implicit decisions during the research process. I relate these implicit decisions to Rumsfeld's (2002) concept of unknown unknowns: \"the things we don't know we don't know\"! I argue that scientists can report their known knowns (what they know they did and found), and they can be transparent and speculative about their known unknowns (what they know they didn't do and may find), but that they can't say much about their unknown unknowns (including their unconscious, implicit decisions) because, by definition, they don't know what they are! Nonetheless, I think that it's important to acknowledge unknown unknowns in science because doing so helps to contextualise research efforts as being highly tentative and fallible.
The videos of the presentations at the Metascience2023 conference were recently made available here. One that caught my eye is by Tom van Drimmelen, titled \"Researchers' decision making: Navigating ambiguity in research practice\":
Here, I summarise Van Drimmelen's excellent work and share my thoughts on his ideas.
Van Drimmelen (2023) explained that, during the research process, researchers have discretion over numerous decisions that they make about their methodology and analyses. During the data analysis stage, this researcher discretion may be perceived negatively as researcher degrees of freedom that is susceptible to researcher bias and questionable research practices (John et al., 2012; Simmons et al., 2011).
In theory, it is possible to reduce and/or reveal researcher discretion during data analyses by preregistering one or more methodological and analytical research paths. However, Van Drimmelen (2023) and colleagues were interested in how researcher discretion operates in practice. To investigate, they conducted an ethnographic study in which they observed researcher discretion in two different research groups. They found that researcher discretion emerged when researchers used two categories of research plan.
1. Underdetermined plans: Researcher discretion emerged when research plans were not precise enough to determine specific actions for the researchers. In this case, researchers needed to use their own discretion to fill in the gaps of the research plan and figure out how to proceed. Researchers often find themselves in this situation because preregistered research plans are often vague (e.g., Bakker et al., 2020; Heirene et al., 2021; Van den Akker et al., 2023).
2. Overdetermined plans: Researcher discretion also emerged when there were unforeseen changes in the research situation that made the original research plan undesirable or impossible, necessitating an alternative research approach. Again, this situation is common, because most researchers do not possess \"godlike planning abilities\" (Navarro, 2020). To attempt to address this issue, researchers can preregister \"if\u2026then\" contingencies (i.e., decision trees) that accommodate anticipated changes in the research situation (Nosek et al., 2018, p. 2602). In addition, it may be possible to update a preregistered analysis plan prior to the data analysis in order to take account of unanticipated events (Nosek et al., 2019, p. 817). However, it is often the case that unanticipated events arise during data analyses. As Reinhart (2015, p. 95) explained (paraphrasing Helmuth von Moltke), \"no analysis plan survives contact with the data.\" In these cases, researchers either need to stick to an inadequate preregistered analysis plan or deviate and adapt their approach to address the current situation. The evidence shows that researchers often deviate from their preregistered plans (e.g., Abrams et al., 2020; Claesen et al., 2021; Heirene et al., 2021).
Van Drimmelen's (2023) work shows us that researchers are forced to either deviate from overdetermined plans to overcome unforeseen events or fill in the gaps of vague, underdetermined plans. Hence, researcher discretion emerges in the case of both over- and under-determined plans. As Van Drimmelen (2023) put it, \"researcher discretion is an integral and unavoidable part of the research practice.\"
I think Van Drimmelen's (2023) evidence also points to a trade-off between the prescriptiveness of preregistered research plans and the feasibility of their implementation in an unpredictable world. As preregistered plans become more precise and prescriptive, there is a greater chance that researchers will need to deviate from them in order to accommodate unforeseen and/or uncontrolled events that occur during the implementation of the plan. For example, researchers are more likely to deviate from a plan to test \"300 participants\" than they are to deviate from a less prescriptive plan to test \"around 300 participants,\" because uncontrolled events (e.g., the number of participant exclusions during data analysis) may result in a final sample size that is only close to 300 participants and not exactly 300 participants (e.g., Claesen et al., 2021, pp. 6-7). Consequently, as preregistered plans become more prescriptive, they are more likely to result in deviations that turn their planned confirmatory tests into unplanned exploratory tests.1 Of course, preregistration continues to distinguish between confirmatory and exploratory tests in this situation, and it prevents exploratory tests from being falsely portrayed as confirmatory tests. Nonetheless, it remains the case that, as plans for confirmatory tests become more prescriptive, the actual implementation of these confirmatory tests in the real world becomes less likely.
On the other hand, as preregistered plans become vaguer and less prescriptive, confirmatory hypothesis tests will become not only more feasible to achieve without deviation, but also more susceptible to researcher degrees of freedom and questionable research practices. For example, it is more feasible to test \"around 300 participants\" than to test exactly \"300 participants.\" However, the first sampling plan does not prevent the questionable research practice of optional stopping (John et al., 2021; Simmons et al., 2011), whereas the second, more prescriptive plan does.
In summary, researchers need to consider a prescriptiveness trade-off when writing their preregistered plans: All other things being equal, plans that are more prescriptive are more likely to result in deviations that turn their confirmatory tests into exploratory tests, and plans that are less prescriptive are more likely to result in confirmatory tests that are susceptible to questionable research practices. Hence, researchers face a difficult choice between prescriptive confirmatory tests that they are less likely to carry out in practice and vague confirmatory tests that are more prone to questionable research practices.
Van Drimmelen and colleagues (2023) also identified another interesting issue in their work. At times, they found it difficult to identify researcher discretion because there were multiple potential methodological or analytical approaches available that researchers did not consider. In other words, sometimes researchers were not aware that they were making decisions!
This idea of unconscious and implicit research decisions reminds me of something that Donald Rumsfeld, the United States Secretary of Defence, said at a press briefing in 2002:
There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns\u2014the ones we don't know we don't know. (Wikipedia, 2023)
In the context of Figure 1, our known knowns are the path or paths that we reported following in our research (e.g., the green paths). In these cases, we know what we did, and we know what we found as a result. In contrast, our known unknowns are the paths that we actively decided not to follow (e.g., all the other paths in Figure 1). We know these paths exist, but we don't yet know what results lie at the end of them.
But what about our unknown unknowns? These paths don't appear in Figure 1! They relate to decisions that we didn't even consider, at least not at a conscious level. For example, we may not have considered the time of day of testing, the ambient temperature, or a host of other unspecified variables that may be influential on our results. Instead, we unconsciously relegated these unknown unknowns to a ceteris paribus clause (i.e., \"all other things being equal\"), which assumes that no other influential factors are at play (Meehl, 1990). The problem is that our unconscious and implicit decision to assume the irrelevance of these factors may be wrong! Indeed, the history of science is full of cases in which factors that were not initially considered to be important subsequently turned out to be crucial moderators or boundary conditions of effects (for some examples, see Firestein, 2016).
Van Drimmelen (2023) concluded that \"we might not be able to be effectively transparent because we don't know which decisions we've actually made\" (my emphasis). In other words, our lack of awareness about our potentially important implicit decisions means that we can't be transparent about them: We can report our known knowns, and we can be transparent and speculative about our known unknowns, but we can't say much about our unknown unknowns because we don't know what they are! By definition, as soon as we start to speculate on a potentially influential factor in our research, it becomes a known unknown about which we need to make conscious decisions!
Despite our inability to specify our unknown unknowns, Van Drimmelen's (2023) work reminds us that it's important to acknowledge this type of ignorance in science because it helps to contextualise our research efforts as being highly tentative and fallible steps in a collective process of discovery and theory development. Acknowledging our unknown unknowns also helps to address recent calls for greater intellectual humility and modesty in science (Hoekstra & Vazire, 2021; Ramsey, 2021).
I'll end with a nice example of scientists acknowledging unknown unknowns. Here's Errington et al. (2021, p. 12) discussing the challenges of assessing replicability in preclinical cancer biology:
It might also be that, in some cases, a failure to replicate was caused by the replication team deviating from the protocol in some way that was not recognized, or that a key part of the procedure was left out of the protocol inadvertently. It is also possible that the effect reported in the original paper depended on methodological factors that were not identified by original authors, the replication team, or any other experts involved in the peer review of the original paper or the Registered Report.
Abrams, E., Libgober, J., & List, J. A. (2020). Research registries: Facts, myths, and possible improvements (No. w27250). National Bureau of Economic Research. https://doi.org/10.3386/w27250
Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A.,\u2026& Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12), e3000937. https://doi.org/10.1371/journal.pbio.3000937
Claesen, A., Gomes, S., Tuerlinckx, F., Vanpaemel, W., & Leuven, K. U. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), 1\u201311. https://doi.org/10.1098/rsos.211037
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Reproducibility in cancer biology: Challenges for assessing replicability in preclinical cancer biology. Elife, 10, Article e67995. https://doi.org/10.7554/eLife.67995
Fife, D., & Rodgers, J. L. (2019). Moving beyond the \"replication crisis\": Understanding the exploratory/confirmatory data analysis continuum. PsyArXiv. https://doi.org/10.31234/osf.io/5vfq6
Firestein, S. (2016, February 14). Why failure to replicate findings can actually be good for science. LA Times. https://www.latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html
Heirene, R., LaPlante, D., Louderback, E. R., Keen, B., Bakker, M., Serafimovska, A., & Gainsbury, S. M. (2021). Preregistration specificity & adherence: A review of preregistered gambling studies & cross-disciplinary comparison. PsyArXiv. https://psyarxiv.com/nj4es/
Hoekstra, R., & Vazire, S. (2021). Aspiring to greater intellectual humility in science. Nature Human Behaviour, 5(12), 1602-1607. https://doi.org/10.1038/s41562-021-01203-8
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524\u2013532. https://doi.org/10.1177/0956797611430953
Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108-141. https://doi.org/10.1207/s15327965pli0102_1
Navarro, D. (2020, September 23). Paths in strange spaces: A comment on preregistration. PsycArXiv. https://doi.org/10.31234/osf.io/wxn58
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van 't Veer, A. E., & Vazire, S. (2019). Preregistration is hard, and worthwhile. Trends in Cognitive Sciences, 23, 815\u2013818. https://doi.org/10.1016/j.tics.2019.07.009
Ramsey, R. (2021). A call for greater modesty in psychology and cognitive neuroscience. Collabra: Psychology, 7(1), 24091. https://doi.org/10.1525/collabra.24091
Reinhart, A. (2015). Statistics done wrong: The woefully complete guide. No Starch Press.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359\u20131366. https://doi.org/10.1177 /0956797611417632
Van den Akker, O., Bakker, M., van Assen, M. A. L. M., Pennington, C. R., Verweij, L., Elsherif, M. M.,\u2026Wicherts, J. M. (2023, May 10). The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency. PsyArXiv. https://doi.org/10.31222/osf.io/h8xjw
Van Drimmelen, T., Slagboom, N., Reis, R., Bouter, L., Van der Steen, J. (2023, May 9). Researchers' decision making: Navigating ambiguity in research practice. Metascience2023, Washington, DC. Abstract: https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice/ Preregistration: https://osf.io/tqwgp/
Wikipedia. (2023, March 16). There are unknown unknowns. Wikipedia. https://en.wikipedia.org/wiki/There_are_unknown_unknowns
1. Here, I define confirmatory tests as planned tests, and exploratory tests as unplanned tests. This conceptualization is consistent with the frequentist rationale for preregistration, according to which unplanned tests undermine assumptions regarding Type I error rates (Nosek et al., 2019, p. 816). I acknowledge that there are other definitions for \"confirmatory\" and \"exploratory\", including definitions that view them as existing at opposite ends of a continuum rather than as discrete categories (e.g., Fife & Rodgers, 2019). According to this continuum perspective, larger deviations from preregistered plans result in \"more exploratory\" tests than smaller deviations. I don't subscribe to this continuum perspective, because it is unclear how to formally assess some deviations as being \"larger\" or \"smaller\" than others. I agree that research studies can be described as being more confirmatory or exploratory, because they can contain a greater proportion of planned or unplanned tests. However, in my view, the tests themselves can only be either confirmatory or exploratory, because they can only be either planned or unplanned.
I'm grateful to Tom van Drimmelen for his comments on an earlier version of this article.
You can download a PDF version of this article here.
The reference for this article is:
Rubin, M. (2023, June 7). The preregistration prescriptiveness trade-off and unknown unknowns in science: Comments on Van Drimmelen (2023). Critical Metascience: MetaArXiv. https://doi.org/10.31222/osf.io/3t7pc
Felipe Romero presented his work - \"The conceptual origins of metascience: Fashion or revolution?\" - in The Popper Seminar at the London School of Economics on 30th May 2023. Here, I highlight a few points that he raised and provide some comments along the way.
Metascience is often described as the \"science of science.\" However, as Peterson and Panofsky (2023) recently observed, contemporary metascience is something more than this. To illustrate, they used a Venn diagram to show how metascience may be conceived as falling at the intersection of science of science, open science, and methodological activism.
After providing some examples of the various aspects of metascience, Romero considers some different ways of distinguishing metascience from related disciplines such as (a) the history and philosophy of science, (b) sociology, and (c) science, technology, and society (STS) studies. He notes that metascientists see themselves as quite separate from these other disciplines. Indeed, he notes that some metascientists seem to show a resistance to engaging with these disciplines by (a) failing to engage in interdisciplinary discussions, (b) not citing interdisciplinary work, and (c) generally adopting \"an adversarial attitude.\"
But what distinguishes metascience from these other disciplines? Romero runs through several possibilities and considers the pros and cons in each case.
Romero notes that metascientists tend to stress the empirical nature of their work as a distinctive feature. As Jonathon Schooler explained in Peterson and Panofsky's (2023) article, metascientists champion \"less armchair, less qualitative distillation of impressions and more hard-nosed empiricism\" (p. 167).
However, as Romero points out, one problem with emphasising the empirical character of metascience is that it runs the risk of concealing and/or overlooking important philosophical issues. What is a replication? What counts as evidence? What is progress? Empirical work must adopt philosophical positions on these questions, even if they're only implicit positions. As Romero explains, \"you might think that the work is empirical, but you are already making commitments about all of these notions.\"
The philosopher Daniel Dennett (2013) famously argued that \"there is no such thing as philosophy-free science, just science that has been conducted without any consideration of its underlying philosophical assumptions\" (p. 20). Similarly, it might be argued that there is no such thing as a philosophy-free metascience, and greater discussion of the philosophy of metascience may help to build mutually beneficial bridges between metascience and the philosophy of science.
Romero accepts that metascientists do conceptual work around issues such as replication, evidence, and progress. However, he suggests that, without paying closer attention to previous work that's been done in other disciplines, \"there is a risk of reinventing the wheel and repeating mistakes.\" To be fair, some metascientists couch their work within a Popperian philosophy of science (Derksen, 2019). Nonetheless, there remain concerns about the lack of integration with other positions in the philosophy of science and in other related fields (e.g., Gervais, 2021; Morawski, 2019, p. 234; Wiggins & Christopherson, 2019). For example, Gervais (2021) explained that:
Metascience and methodological reform movements\u2014despite noble intentions\u2014risk needlessly reinventing other fields' time-tested wheels or, worse, reproducing the problematic norms, institutions, and incentives of status quo science (p. 827).
To illustrate his point, Romero notes that some metascientists focus on \"methodological individualism,\" according to which \"(bad) statistics and methodology are the primary factors affecting science.\" From this perspective, many of the problems lie with individuals, and many of the proposed solutions are correspondingly individualistic in nature. Hence, as Field (2022) observed, \"the current crisis prominently features finger-pointing and ad-hominem attacks between researchers, where direct accusations of incompetence and data fabrication are hurled around between people and fields\" (p. 13).
However, Romero observes that other metascientists are rediscovering the idea that we need to look at social structures and organization to affect scientific progress (for further discussions, see Flis, 2019, p. 175; Flis, 2022; Wiggins & Christopherson, 2019, p. 210). I think this broader and more abstract level of analysis is more befitting of a genuinely metascientific approach. In my view, scientists should critique the work of other scientists, and metascientists should critique scientific processes and approaches.
Romero also notes that the \"dominant narrative\u2026that metascience is good because it is empirical\u2026could lead to methodological fetishism.\" In my view, \"methodological fetishism\" occurs when people use methodological checklists and heuristics in ways that ignore the local goals of specific research studies (see also Devezer et al., 2021, p. 3). For example, depending on the function(s) that you think it serves, preregistration may not be a useful approach in certain research situations (e.g., some qualitative studies; Steltenpohl et al., 2023). Nonetheless, Spitzer and Mueller (2023) found that some psychological researchers suggest making preregistration mandatory, and Aguinis et al. (2020, p. 29) have argued that we should \"require\" preregistration for both quantitative and qualitative studies. In these cases, the overapplication of preregistration may represent a form of methodological fetishism.
Furthermore, the use of checklists and heuristics may mislead researchers, reviewers, and their readers into a false sense of security about their approach when a deeper, more philosophical consideration would reveal important uncertainties and limitations. As Chamberlain (2000) explained, \"it is much less worrying to concern oneself with the correct details of method and methodology than to tangle with the philosophy of science\" (p. 293).
Metascience has seen a remarkable rise in popularity in the wake of the replication crisis. Hence, Romero asks whether metascience may be a \"fashion,\" although he makes it clear that this term is not used in a pejorative manner.
To illustrate, Romero considers how metascience and open science might fit with a fashion model that has been applied previously in science (Crane, 1969; Starbuck, 2009). Consistent with this model, he notes that metascience contains:
Elites: Small groups of scholars who set fashions
Conformity: Scholars who adopt practices they don't endorse
Performativeness: Scientists who pursue a practice/theory to elicit approval from others
\"What's the problem?\" asks Romero. \"I think that some scientists are adopting open science as a fashion and this leads to performative open science.\"
As an example, Romero considers \"performative preregistration,\" which might manifest in a couple of ways: First, the \"underspecification problem\" is that researchers might go through the motions of preregistration, but preregister plans that are vague and, therefore, relatively useless (Bakker et al., 2020). Second, the \"undisclosed deviations problem\" is that researchers may not indicate deviations from their preregistered plan (Akker et al., 2022). Again, this approach ticks the preregistration box, but it makes the potential benefits of preregistration relatively inaccessible.
Romero concludes that a risk associated with metascience being a fashion is that fashions can have poor epistemic benefits and, when scientists undertake a cost-benefit analysis, this may result in a decline in the uptake of research practices like preregistration.
According to Romero, the best way to demarcate metascience is to refer to its normative, activist component: Metascientists tend to be activists who are trying to change scientific systems and culture (see also Peterson & Panofsky, 2023, p. 147).
In Romero's view, it is the activism aspect of metascience that provides the sharpest distinction with the history and philosophy of science, sociology, and STS, because these other disciplines focus on describing and understanding science, but not necessarily improving it. Importantly, however, if metascience is to be distinguished by its goal of \"improving science,\" then I think the metascience community should include an adequate representation of viewpoints about what \"science\" is and how it can best be \"improved.\" In this sense, the current overrepresentation of psychologists in this community may be problematic (see also Flis, 2019).
Final point from me: As I've noted elsewhere (Rubin, 2023), there's a danger of homogenizing metascience in these discussions (Field, 2022). Like science, metascience may be many things to many people, and it may be done in many different ways, with different metascientists occupying different positions in Peterson and Panofsky's (2023) Venn diagram as well as having different interdisciplinary overlap with the history and philosophy of science, sociology, and STS. There is strength in diversity here, and we should be careful not to lose sight of this diversity during efforts to categorize and demarcate metascience.
Website: https://www.feliperomero.org/
Publications: https://www.feliperomero.org/publications.html
Aguinis, H., Banks, G. C., Rogelberg, S. G., & Cascio, W. F. (2020). Actionable recommendations for narrowing the science-practice gap in open science. Organizational Behavior and Human Decision Processes, 158, 27-35. https://doi.org/10.1016/j.obhdp.2020.02.007
Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A.,\u2026& Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12), e3000937. https://doi.org/10.1371/journal.pbio.3000937
Chamberlain, K. (2000). Methodolatry and qualitative health research. Journal of Health Psychology, 5(3), 285-296. https://doi.org/10.1177/135910530000500306
Crane, D. (1969). Fashion in science: Does it exist? Social Problems, 16(4), 433-441. https://doi.org/10.2307/799952
Derksen, M. (2019). Putting Popper to work. Theory & Psychology, 29(4), 449-465. https://doi.org/10.1177/0959354319838343
Dennett, D. C. (2013). Intuition pumps and other tools for thinking. WW Norton & Company.
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (2021). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3), Article 200805. https://doi.org/10.1098/rsos.200805
Field, S. M. (2022, July 13). Charting the constellation of science reform. https://doi.org/10.31219/osf.io/udfw4
Flis, I. (2019). Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology, 29(2), 158-181. https://doi.org/10.1177/0959354319835322
Flis, I. (2022). The function of literature in psychological science. Review of General Psychology, 26(2), 146-156. https://doi.org/10.1177/10892680211066466
Gervais, W. M. (2021). Practical methodological reform needs good theory. Perspectives on Psychological Science, 16(4), 827-843. https://doi.org/10.1177/1745691620977471
Morawski, J. (2019). The replication crisis: How might philosophy and theory of psychology be of use? Journal of Theoretical and Philosophical Psychology, 39(4), 218\u2013238. https://doi.org/10.1037/teo0000129
Peterson, D., & Panofsky, A. (2023). Metascience as a scientific social movement. Minerva https://doi.org/10.1007/s11024-023-09490-3
Romero, F. (2023, May 30) The conceptual origins of metascience: fashion or revolution? YouTube. https://youtu.be/AKWCCdK-jgc
Rubin, M. (2023). Questionable metascience practices. Journal of Trial & Error. https://doi.org/10.36850/mr4
Spitzer, L., & Mueller, S. (2023). Registered report: Survey on attitudes and experiences regarding preregistration in psychological research. Plos One, 18(3), e0281086. https://doi.org/10.1371/journal.pone.0281086
Starbuck, W. H. (2009). The constant causes of never-ending faddishness in the behavioral and social sciences. Scandinavian Journal of Management, 25(1), 108-116. https://doi.org/10.1016/j.scaman.2008.11.005
Steltenpohl, C. N., Lustick, H., Meyer, M. S., Lee, L. E., Stegenga, S. M., Reyes, L. S., & Renbarger, R. L. (2023). Rethinking transparency and rigor from a qualitative open science perspective. Journal of Trial & Error. https://doi.org/10.36850/mr7
Van den Akker, O., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H., R\u00fcffer, F. F.,\u2026Bakker, M. (2022, October 18). Selective hypothesis reporting in psychology: comparing preregistrations and corresponding publications. PsyArXiv. https://doi.org/10.31222/osf.io/nf6mq
Wiggins, B. J., & Christopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202\u2013217. https://doi.org/10.1037/teo0000137
Rubin, M. (2023, June 13). What's special about metascience: Comments on Romero (2023). Critical Metascience. https://markrubin.substack.com/p/whats-special-about-metascience
Subscribe to this Blog and/or Share this Article
","doi":"https://doi.org/10.59350/8rscj-bgy43","guid":"126312825","image":"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1726018f-1f04-464b-b963-4e43855339ec_689x629.png","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1686614400,"reference":[{"id":"https://doi.org/10.1016/j.obhdp.2020.02.007","unstructured":"Aguinis, H., Banks, G. C., Rogelberg, S. G., & Cascio, W. F. (2020). Actionable recommendations for narrowing the science-practice gap in open science. Organizational Behavior and Human Decision Processes, 158, 27\u201335."},{"id":"https://doi.org/10.1371/journal.pbio.3000937","unstructured":"Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., Wicherts, J. M., & Bero, L. (2020). Ensuring the quality and specificity of preregistrations. PLOS Biology, 18(12), e3000937."},{"id":"https://doi.org/10.1177/135910530000500306","unstructured":"Chamberlain, K. (2000). Methodolatry and Qualitative Health Research. Journal of Health Psychology, 5(3), 285\u2013296."},{"id":"https://doi.org/10.2307/799952","unstructured":"Crane, D. (1969). Fashion in Science: Does It Exist?. Social Problems, 16(4), 433\u2013441."},{"id":"https://doi.org/10.1177/0959354319838343","unstructured":"Derksen, M. (2019). Putting Popper to work. Theory & Psychology, 29(4), 449\u2013465."},{"id":"https://doi.org/10.1098/rsos.200805","unstructured":"Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (2021). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3)."},{"id":"https://doi.org/10.31219/osf.io/udfw4","unstructured":"Field, S. M. (2022). Charting the Constellation of Science Reform. Center for Open Science."},{"id":"https://doi.org/10.1177/0959354319835322","unstructured":"Flis, I. (2019). Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology, 29(2), 158\u2013181."},{"id":"https://doi.org/10.1177/10892680211066466","unstructured":"Flis, I. (2022). The Function of Literature in Psychological Science. Review of General Psychology, 26(2), 146\u2013156."},{"id":"https://doi.org/10.1177/1745691620977471","unstructured":"Gervais, W. M. (2021). Practical Methodological Reform Needs Good Theory. Perspectives on Psychological Science, 16(4), 827\u2013843."},{"id":"https://doi.org/10.1037/teo0000129","unstructured":"Morawski, J. (2019). The replication crisis: How might philosophy and theory of psychology be of use?. Journal of Theoretical and Philosophical Psychology, 39(4), 218\u2013238."},{"id":"https://doi.org/10.1007/s11024-023-09490-3","unstructured":"Peterson, D., & Panofsky, A. (2023). Metascience as a Scientific Social Movement. Minerva, 61(2), 147\u2013174."},{"id":"https://youtu.be/akwccdk-jgc","unstructured":"Unknown title"},{"id":"https://doi.org/10.36850/mr4","unstructured":"Rubin, M. (2023). Questionable Metascience Practices. Journal of Trial and Error, 4(1), 5\u201320."},{"id":"https://doi.org/10.1371/journal.pone.0281086","unstructured":"Spitzer, L., Mueller, S., & Naudet, F. (2023). Registered report: Survey on attitudes and experiences regarding preregistration in psychological research. PLOS ONE, 18(3), e0281086."},{"id":"https://doi.org/10.1016/j.scaman.2008.11.005","unstructured":"Starbuck, W. H. (2009). The constant causes of never-ending faddishness in the behavioral and social sciences. Scandinavian Journal of Management, 25(1), 108\u2013116."},{"id":"https://doi.org/10.36850/mr7","unstructured":"Steltenpohl, C. N., Lustick, H., Meyer, M. S., Lee, L. E., Stegenga, S. M., Standiford Reyes, L., & Renbarger, R. L. (2023). Rethinking Transparency and Rigor from a Qualitative Open Science Perspective. Journal of Trial and Error, 4(1), 47\u201359."},{"id":"https://doi.org/10.31222/osf.io/nf6mq","unstructured":"Van den Akker, O., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H. H., R\u00fcffer, F. F., Schoenmakers, M., Stoevenbelt, A. H., Wicherts, J. M., & Bakker, M. (2022). Selective Hypothesis Reporting in Psychology: Comparing Preregistrations and Corresponding Publications. Center for Open Science."},{"id":"https://doi.org/10.1037/teo0000137","unstructured":"Wiggins, B. J., & Christopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202\u2013217."}],"rid":"g14t5-fsf37","summary":"Comments on Romero (2023)","tags":["Metascience","Methodolatory","Science, Technology, And Society","Methodologism","Open Science"],"title":"What's Special About Metascience?","updated_at":1792238127,"url":"https://markrubin.substack.com/p/whats-special-about-metascience","version":"v1"},{"authors":[{"contributor_roles":[],"family":"Hostler","given":"Tom"}],"blog":{"authors":[{"name":"Mark Rubin","url":"https://orcid.org/0000-0002-6483-8561"}],"community_id":"3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6","created":1681948800,"current_feed_url":null,"description":"Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.","doi":"https://doi.org/10.59350/markrubin","favicon":"https://rogue-scholar.org/api/communities/3bd0dcf5-7d43-47f1-b260-87d9f30bdcd6/logo","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"markrubin","status":"active","subfield":"1207","title":"Critical Metascience","updated":1792151011,"use_api":true},"blog_name":"Critical Metascience","blog_slug":"markrubin","content_html":"In his book \"The Soul of a University\" (2018), Chris Brink describes the story of G.H. Hardy, a Cambridge Mathematician whose principled stance on his academic research was that it had no practical use whatsoever: \"No discovery of mine\" Hardy proudly wrote, \"has made, or is likely to make, directly or indirectly, for good or ill, the least difference to the amenity of the world\" (Hardy, 1945/2005). Unfortunately, he was incorrect. In a rare and reluctant foray outside his own work, he wrote a short letter to Science (Hardy, 1908), explaining and correcting a mathematical mistake pointed out to him by a geneticist friend who he played cricket with. The proof he provided is now known as the Hardy-Weinberg Principle and is a foundational law of the study of population genetics.
Hardy's belief in the uselessness of research was rooted in the idea that non-instrumental research was the purest expression of intellectual curiosity. Although Hardy was an extreme example, the idea of producing knowledge for its own sake \u2013 rather than for an instrumental purpose - is closely wrapped up in the traditional stereotypes and ideals of \"academia\". Universities from this view are seen as 'protected' spaces in society where intellectual work can be undertaken by those with the aptitude and ability to do so, as represented by the metaphor of the disconnected \"ivory tower\", the residence of \"intellectuals\", \"eggheads\", and \"boffins\"1. These individuals' vocation is to focus on esoteric but often impracticable concerns, compared to the rest of society mired in problems of the real world. This stereotype is captured in the phrase \"that's academic\", often used to dismiss things as hypothetical concerns which are practically meaningless.
Academia's disconnect from practical reality and curiosity-driven mission help to provide a basis for its role in society. These stereotypes help to support the myth or 'legend' of academic research (especially scientific research) as a legitimate authority on defining the \"truth\" of an issue and resolving factual disputes. Academia is traditionally seen as a politically neutral institution of independent thinkers, who are trusted to provide disinterested 'facts' to inform moral or political decisions, even if these are unpalatable or relate to controversial topics.
The academic institution that these stereotypes and myths are based on has changed significantly over the last few decades, and the changes look to be accelerating, with the increasingly influential scientific reform movement advocating for a 'revolution' in how research is conducted, published, consumed, and assessed (Spellman, 2015). This essay will focus on how these reforms are linked to existing changes in academia and what the effects might be on our culture and traditions.
The cornerstone of the current piece is the work of John Ziman, and his book \"Real Science \u2013 What it is, and what it means\" (Ziman, 2000), which covers his attempt to investigate the 'legend' of science, and the legitimacy of its authority. Ziman takes a refreshing perspective. Instead of looking for connections between the philosophies or methods of different academic disciplines to find a common definition of what research 'is', he adopts a naturalistic approach: he describes what he sees. The result is a fuzzy yet recognisable picture of 'traditional' academia, based not on shared methodologies or philosophies of science but of shared community practices and values. Ziman suggests that academic research is best described as an institution based around a culture of shared traditions and citizenship. Whilst there are significant differences between academic disciplines in the type and form of knowledge produced, there is a common principle of sharing, discussing, and collectively evaluating codified contributions to knowledge from the academic community.
Ziman describes traditional academia as a pre-modern institution without clear written rules, operating according to shared values and ethos of Communalism, Universalism, Disinterestedness, Originality, and Scepticism (CUDOS) that differentiate academic research from other kinds of knowledge generation in society. These values \u2013 linked to Merton's norms - are closely tied to the non-instrumental nature of much traditional academic research: being driven by intellectual curiosity alone is what grants academics the autonomy and freedom to share knowledge freely, universally accept contributions of merit, remain disinterested to biasing incentives, pursue originality, and express scepticism. This autonomy applies to both individual academics (to pursue careers in their topics of interest), but also to the collective to exercise 'academic freedom' and critically evaluate society. Academics are traditionally seen not to follow any ideology other than the search for truth, and so can generally be trusted to advise society from a position as neutral experts, even if those truths may be uncomfortable (e.g., the need to divest from fossil fuels to prevent climate change or reduce interpersonal contact to control the spread of a pandemic).
The autonomous nature of academic research is reflected in the way that research is typically organized and rewarded in a largely individualistic way. Careers are mainly built on the qualitative prestige of individual contributions to the scientific record, meaning that any individual could aspire to pursue their own research program and achieve tenure and field-wide recognition. Researchers have always been aware that academia is competitive but traditionally there was no upper cap on the number of 'boffins' allowed: If your contributions were valuable, there was a place for you. But Ziman argues that this model was never going to be sustainable: In 1961, Derek de la Solla Price published a graph showing that the number of publications (and therefore researchers) was growing exponentially, as new discoveries produced more new questions and professors developed more prot\u00e9g\u00e9s to answer them. If this continued, we would eventually end up with more scientists than people on the planet! The limiting factor became money: eventually, academic research became too expensive to society to be allowed to keep growing. But in an age of perpetual 'progress', knowledge generation wasn't allowed to slow down. The answer is that the funders of academic research (primarily governments) needed the process to become more efficient.
As a consequence, Ziman's naturalistic description observes a change in the institution of research. He argues that we have now entered a stage of \"post-academic\" research, in which the culture and structure are morphing to a new model designed to achieve more efficiency in the process of turning funding into knowledge that is useful to society. A key part of this change is the collectivisation of research. No longer do individual researchers primarily pursue their own creativity and curiosity-driven academic interests, but instead work together on an industrial scale to tackle 'problems' or 'challenges' commissioned by research funders relevant to society's needs. Academics are strategically employed as a workforce of expert problem solvers, rather than appointed to a university as autonomous intellectuals driven by a thirst for knowledge.
Researchers are required to hype up and justify the potential impact and applications of their work to these problems (Hyland & Jiang, 2021). Pursing a deliberately 'useless' line of research like G.H. Hardy seems unfathomable in the hypercompetitive world of grant applications and would be seen as antithetical to producing knowledge 'efficiently'. The collectivisation of research also leads to specialisation and selects for different academic abilities. Researchers who have the organizational and social skills to act as effective managers of grant-winning and publication-generating teams of specialist post-docs are more likely to succeed than awkward 'eggheads' who don't know how to 'play the game'. Ziman summarises the post-academic ethos with the acronym 'PLACE': a research culture that increasingly produces Proprietary knowledge, is focused on Local problems rather than contributing to general understanding, is managed by a hierarchical Authority, works on Commissioned projects rather than long-term issues of curiosity, and is staffed with technical Experts rather than independent thinkers.
From his naturalistic perspective, Ziman refrains from speculating on the causes of these changes. The application of neoliberal government policy to the funding and organization of research and universities is surely a large contributor and has been the subject of significant study in the theory of \"academic capitalism\" (Jessop, 2018). However, the development of technology, the internet, and the growth of neoliberalism, legislation, and quantification in society more generally (which research exists as a part of) are also important facets. Ultimately there is no one single cause, but the accumulated changes are undeniable. But how does the scientific reform movement relate to this?
The science reform movement will no doubt be familiar to readers of this essay, and here I use the term broadly to refer to advocates for the widespread promotion and adoption of open research practices. For many seasoned open research advocates, entry into the reform movement began with an interest in methodological innovations (such as preregistration, placing datasets online and so on) to discourage questionable research practices (QRPs), which had become common tricks employed to survive the cycle of publish or perish and competition in academia (Bakker et al., 2012). Explanation and promotion of open research practices subsequently became the topic of dozens of papers, workshops, conferences, and editorials across nearly all academic disciplines.
But open research advocates realized that widespread adoption of these practices would require more than grassroots methods workshops: it would require changes to the structure and systems of academia itself (Nosek, 2019). There was increasing debate about the incentives of academia, and how they shaped researcher behaviour. If advocates really wanted researchers to conduct open research, then rewards would need to be introduced higher up the system to funders, journals, and institutions (Nosek, Spies, & Motyl, 2012). To integrate open research into the system, appropriate infrastructure would need to be built to accommodate data, materials, code, and preregistrations. Policies and procedures would need to be developed to record and codify the use of open practices to be able to reward them appropriately. Smashing the pi\u00f1ata of 'closed' research meant academia would have to catch all the valuable open research sweets that tumbled out and this meant developing the appropriate baskets and buckets to do so.
These sweets have also turned out to be chewier than anticipated. Formatting data, writing analysis code, using version control, and writing detailed pre-registrations are often complex skills that require knowledge and training beyond the scope of many researchers, particularly those already stretched to breaking point with untenable workloads (Hostler, 2023). As the science reform movement germinates into new disciplines, standard practices developed in the context of combatting QRPs in quantitative experimental social psychology studies require reworking for new epistemologies and methodologies (E. A. Bennett, 2021; Leonelli, 2023). Sophisticated open research, such as making research materials interoperable across disciplines and questioning the theoretical basis of preregistration decisions are difficult challenges for many academics \u2013 especially those who during the decades of 'replication crises' have been advised or trained to generate 'easy' publications rather than be critically reflective methodologists.
An increasingly common solution to this issue is to propose further specialization and collectivisation: if researchers cannot be expected to accommodate the significant demands of open research themselves, then they should be supported by colleagues who can help them with this. Accordingly, there is an increased focus on the 'team infrastructure roles' that support research (Carter et al., 2019) and increased prominence for the role of open research specialists such as data stewards, methodologists, and software developers (A. J. Stewart et al., 2021). Support for this shift comes from the very top: a recent high-level report by the UK Government science and technology committee on reproducible research recommended increased funding for these roles (UK Science, Innovation, and Technology Committee, 2023). Relatedly, many science reform commentators have argued that the way that research is assessed should also focus more on team contributions, and that funding should reward configured teams, rather than individuals, to reflect the added work that goes into conducting open research (A. Bennett et al., 2022; Tiokhin et al., 2021). Positions such as 'open research officer' are increasingly common in universities to assist academics with the demands of open research, and researchers are encouraged to build in support for open research from specialists into grant applications (e.g., University of Bristol, n.d.).
There are clear parallels between the scientific reform movement's discourse on policy and infrastructure, and the existing observable changes in the shift to post-academic research: Both advocate for a focus on teams and collectives rather than individuals; for specialization in methods, especially with technical skills; and for an increased role for 'research professionals' as team members to support the logistics of open practices for specific research projects. A quote from Stewart et al. (2021) clearly captures this sentiment: \"The days of the 'lone genius' as the model of a researcher are fading fast, if not gone already. Therefore, there should be wider support, recognition and reward of team-based research\" (p.3). Open research and team research are increasingly linked and promoted as complementary: open research facilitates team research and collaboration by enabling participation through transparency; team research facilitates open research by dividing the potentially significant extra workload and technical demands of openness between many individuals.
Of course, there is not perfect overlap between the ideas of scientific reform and the post-academic research ethos. Open research ideals are antithetical to another aspect post-academic research identified by Ziman, which is the increased collaboration with industry and greater generation of proprietary knowledge. Ziman noted that increasing collaboration with industry led to greater control over knowledge generated through academic research and a rise in 'spin off' academic companies designed to monetize findings. This mindset is certainly incompatible with open research ideals of free and open access to research data and materials (which may lead to tension, e.g. Fern\u00e1ndez Pinto, 2020). Ziman's description of post-academic research as focussing on solving interdisciplinary 'local' problems is also tangential to advocating for greater collaboration and team-based research \u2013 logically, teams of specialists could be directed towards solving fundamental, curiosity driven questions rather than practical problems. The large particle accelerator research at CERN is an example of a team-based project that performs curiosity driven research, such as seeking to test theoretical predictions of the existence of subatomic particles2.
However, given a move away from autonomous individualized research is already a key element of post-academic research, then further collectivization and specialization (accelerated by science reformers) may contribute to a continued shift towards post-academic norms more broadly. Science reform advocates have called for adaptations to infrastructure and policy to accommodate open research practices and increased collectivization. But much of the groundwork for these changes has been laid already in the broader shift to post-academic research, with greater prominence for research professionals in the research process, as well as increases in research administration, and tracking and monitoring of research bids, outputs, and research data (Shepherd, 2008). Science reformers should be aware that utilizing these systems may lead to greater alignment with post-academic research norms in a broader sense, as well as facilitating open methodological practices.
Team-based, method-focused research is fundamentally different in many ways to traditional academic work, particularly when it comes to autonomy. There is the basic matter of compromise when working in larger teams that means it is more difficult for an individual to pursue their own ideas, and there is a difficulty in finding collaborators for ideas that may be questionable or controversial. Collaborating with non-academics also changes the dynamic of research. Existing research professionals (such as grant application managers) often lack autonomy in their jobs (A. Bennett et al., 2022) as their role is often tied more closely to supporting universities' 'research strategy', which typically supports specific types and topics of research and has a large focus on generating research income (Rees, 2015). As such, research professionals are often managed to prioritize supporting researchers whose work aligns with post-academic values, rather than quirky side projects. Expansion of these roles into open research may follow suit: employing core-funded technical research specialists to support academics to conduct research is a progressive move compared to the current precarious 'gig economy' of overworked temporary research positions (Nelson et al., 2020). However, such positions mean that universities may reserve greater control over how such specialists are deployed to support particular strategy-aligned projects, potentially limiting autonomy of researchers to pursue projects alone outside of this remit.
More broadly, open research reforms foreground issues centred around the research process. Several scholars have warned that a focus on methodological problems may neglect deeper philosophical issues in disciplines such as psychology (e.g., Haeffel, 2022; Muthukrishna & Henrich, 2019). However, a focus on methodology and producing transferable, reproducible knowledge is more amenable to a post-academic ethos focused on providing specialist technical solutions to specific local 'problems': It is not surprising that governments and funders are supportive of openness reforms to help improve the 'efficiency' of research and maximise value for money from research funding via open materials and data. Funding lone academics to indulge the whims of their curiosity and pursue careers musing on abstruse theoretical issues with questionable practical significance is less desirable or defensible in a neoliberal political climate.
As a naturalistic descriptor, 'post-academic research' is a valueless term and does not prescribe negative connotations: many aspects of post-academic research may be considered progressive. For example, there are arguments that the 'collectivisation' of research does not constitute a rethinking of the research process, but rather acknowledges the roles and effort that have always existed: It just makes their contributions visible and rewarded. There is certainly a substantial amount of truth in this. To a certain extent the idea of the 'lone genius' was always a myth: behind every Nobel Prize winner was a team of researcher assistants, lab managers, technicians, as well as individuals providing the 'soft' skills which smooth the process and contribute to the success of any piece of research. Recognising these roles is a progressive move to a more equitable research culture (Brand et al., 2015). Yet there is evidence that research teams have for a long time been getting larger, and single-author publications are increasingly rare (Thelwall & Maflahi, 2022). There has also been an undeniable growth in \"third space\" professionals in universities who academics are required to engage with to conduct research (Whitchurch, 2008). These research professional roles largely exist to support academics to navigate and conform to existing policy, legislation, technology, and administration controlled by non-academic entities, rather than to challenge them.
A second defence of post-academic and instrumental research is that it is problematic to use research funding (especially government funding indirectly paid for by tax-payers) to pay for research which does not have any clear benefit. Research is expected to have some articulable impact outside of academia to provide value for money and justify its cost (Doyle, 2018). Similar arguments have been used to justify open data, open materials, and open access (S. L. K. Stewart et al., 2022; UKRI, 2023): If tax-payers are funding our research, then we should aim to maximise the benefit from it and make the products of it available for re-use and scrutiny. Again, this argument has merit. Certainly, public funding should not be used to fund potentially fraudulent research, and open data and materials are often a necessary step for detecting this. However, the 'value for money' argument also relies on a utilitarian logic that prioritizes utility (or potential utility) above all else. The distinction between making data and publications open for scrutiny for bias and errors, and making them open on the basis of 'potential benefit' can become blurred in this discourse. The latter can subtly transform from an advantage of open research to a justifiable imperative, infused with the definitions of 'benefit' from non-academic sources such as funders and policymakers. In other words, it encourages and normalizes assessing research projects primarily on the benefit and impact generated from the researchers' use of resources.
A utilitarian discourse encourages applying neoliberal cost-benefit analysis to research which can lead to decisions to slash funding from non-instrumental research streams and disciplines, such as the Arts and Humanities, which in the UK has recently had its budget reduced by 50% (Weale, 2021). Conducting research for curiosity's sake alone, with no clear practical benefit, cannot be justified in such terms. Alternative human values, such as creativity, enlightenment, heritage, and tradition need to be invoked. But in a dangerous feedback loop, the disciplines and projects most likely to research and advocate for these values are the ones that are finding it harder to justify and find funding.
Reforms to methodological practices have challenged the way that researchers design studies and collect and analyse data, and these challenges have sometimes resulted in pushback from researchers. However, reforms that support the collectivisation of research and greater alignment with post academic norms impact not only the technical skills that researchers employ on a specific project, but their broader way of working and professional practice. More fundamentally, changes in academics' roles challenge academic identities.
There is a significant amount of scholarship of academic identity, particularly within an educational context. The industrialization and collectivization of academic research mirrors the industrialization and collectivization of academic teaching: with the 'massification' of higher education in many countries, the identity of the academic and the relationship between academics and students has changed considerably (Macfarlane, 2011). Many of the tasks that academics used to be responsible for, such as supporting students' career development and 'study skills', have been outsourced to professional services roles3. Macfarlane (2011) has argued that this \"unbundling\" of academic practice has led to a \"hollowing out\" of academic life and further propagates an instrumental and utilitarian mindset towards academic practice (e.g., students should be supported by professionals because the university needs to provide support as a contractual obligation, rather than as a humanistic value of academic citizenship). This hollowing out impacts the traditional identity of the academic as an 'all-round' intellectual and their role in a collegial culture. It raises questions about what it means to be an academic, the answers of which are strongly tied to academics' values and beliefs about the role of a university education (and universities more generally) in society (Henkel, 2005).
The impact of collectivization on research may be similar. In the shift to post-academic research, the definition of 'research' itself has changed (Macfarlane, 2021). Current discourse commonly describes \"research\" as a specific performative activity with a definable output. But traditionally, \"research\" referred to a culture of \"broader intellectual engagement\" (MacFarlane, 2021), encompassing everything from \"scholarly investigation, appreciation, creative and textual criticism, re-interpretation and a critical treatment of contemporary thought\" (Truscot, 1943). This broader definition supported the autonomy of researchers and an academic identity tied to the role of holding society account, birthed in the 'de-Nazification' of science after the second world war (Macfarlane, 2021). In a post-academic research culture, where teams of specialists compete for funding for research on specific pre-defined 'strategic priorities', academic identities are often in conflict. Researchers still espouse the importance of intellectual curiosity, collegiality, autonomy, and criticality, but often find these in conflict with a performative research practice centred around managing small contributions to multiple short-term entrepreneurial projects (T\u00fcl\u00fcba\u015f & G\u00f6kt\u00fcrk, 2023).
For Henkel (2005), autonomy and academic identity are \"integrally related\", but the meaning of autonomy has changed in a post-academic environment. The autonomy derived from sitting in a protected 'ivory tower' is harder to sustain as the boundaries of academia with industry, society, and funders have dissolved. Instead, in an increasingly collectivist research culture, autonomy is generated by managing relationships with those outside of academia to maintain a collective reputation which grants academics the authority to be critical. This model is likely to become more relevant as post-academic research progresses further.
Ziman believed that the shift to post-academic research had significant implications for the legitimacy of academia's authority on knowledge production and its critical role in society. He predicted certain aspects of the stereotypical 'legend' of science \u2013 that it somehow created knowledge that was more 'factual' than other means \u2013 would continue to come under increasing indefensible scrutiny \u2013 a prescient observation considering the growing \"decolonization of knowledge\" agenda, and criticisms of Western hegemony of knowledge production (Santos, 2008). Indeed, open research reforms are likely to accelerate this scrutiny further as the 'black box' of what researchers actually do to generate knowledge is revealed in preregistrations, open data, and open materials. High profile cases of errors and fraud are likely to become more common with increasing transparency. But Ziman discouraged promoting the idea that \"science\" had authority of knowledge on epistemic grounds. Rather, the culture and norms of academia supported an institution which generated knowledge in a way which was defensible on the grounds of being reasonable \u2013 researchers could articulate how their practice and culture reduced or acknowledged bias and errors, and therefore could argue that they produced a uniquely reliable kind of knowledge with value to society.
This view is what made Ziman concerned about the impact of post-academic norms on the cultural fabric of science. He was worried that the transformation of curiosity-driven independent academics into teams of expert problem-solvers working on commissioned projects threatened their autonomy. It was autonomy and the freedom to express intellectual curiosity that was the true basis of academia's authority on knowledge and for holding society to account. The autonomy was embedded into the culture and environment of academia that supported academics to uphold academic values. In post-academic research, the traditions and culture which support these values are continuing to change in numerous ways, accelerated by the science reform movement. It is clear that academic traditions of 'peer review', 'authorship', and the currency of research 'papers' are increasingly not fit for purpose in an open, digital, collectivist research culture. For example, open research reforms significantly increase the amount of 'research material' for scholars to consume - already so excessive that few can keep up \u2013 challenging traditional peer review mechanisms of scepticism and quality control (Hosseini et al., 2022). Larger 'team science' projects and CREDIT taxonomies (Holcombe, 2019) challenge traditional norms of who 'deserves' authorship and how credit is apportioned, disrupting old metrics and reward systems (Hosseini et al., 2022). A traditional focus on 'outputs' as indicators of prestige is increasingly replaced by a focus on the quality of the research process, reconfiguring how and when researchers should be assessed (Chambers & Tzavella, 2021). The values that these new traditions and cultures support \u2013 and whether they will enable the autonomy that is crucial to both academic success and identity \u2013 are yet to be established.
Ziman's naturalistic description of 'post-academic research' is a useful perspective from which to consider the scientific reform movement. The concept transcends the traditional level of analysis of meta-science to describe the inevitable and irreversible changes of an entire culture, and the identity and values that accompany it. The structural changes advocated for by the scientific reform movement are but a small tide in a larger sea change. However, they nevertheless play an important role, especially in accelerating the collectivisation of research through the increase in workload for open research, and in directly advocating for team-based science. Practical changes to the traditions of academic work impact on the identity, culture, and values of the institution. This in turn impacts how the institution is perceived by the society it exists in, raising existential questions about the role and purpose of academia itself. Post-academic research, as the 'post-' moniker implies, is a description of the present, irreversible reality. There is no 'going back' to the traditional model of academia. Ziman's argument was that those of us who go further into the new reality would have to reckon with how to sustain the role of intellectual curiosity, to defend academia's autonomy and role in society. \"The maintenance of the non-instrumental roles of science should be a primary consideration in debating every aspect of its future.\" (Ziman, 2003). In proposing changes to the structure of academia, this is something that science reformers should bear in mind.
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7(6), 543\u2013554. https://doi.org/10.1177/1745691612459060
Bennett, A., Garside, D., Gould van Pragg, C., Hostler, T. J., Kherroubi Garcia, I., Plomp, E., Schettino, A., Teplitzky, S., & Ye, H. (2022). A manifesto for rewarding and recognising Team Infrastructure Roles. Research Equals. https://doi.org/10.53962/knm3-bnvx
Bennett, E. A. (2021). Open Science From a Qualitative, Feminist Perspective: Epistemological Dogmas and a Call for Critical Examination. Psychology of Women Quarterly, 45(4), 448\u2013456. https://doi.org/10.1177/03616843211036460
Brand, A., Allen, L., Altman, M., Hlava, M., & Scott, J. (2015). Beyond authorship: Attribution, contribution, collaboration, and credit. Learned Publishing, 28, 151\u2013155. https://doi.org/doi:10.1087/20150211
Brink, C. (2018). The soul of a university: Why excellence is not enough. Bristol University Press.
Carter, S., Carlson, S., Crockett, J., Falk-Krzesinski, H. J., Lewis, K., & Walker, B. E. (2019). The Role of Research Development Professionals in Supporting Team Science. In K. L. Hall, A. L. Vogel, & R. T. Croyle (Eds.), Strategies for Team Science Success (pp. 375\u2013388). Springer International Publishing. https://doi.org/10.1007/978-3-030-20992-6_28
CERN. (n.d.). The CERN KT Forum: Definition and Mandate. CERN. https://kt.cern/sites/default/files/resource/other/ktforummandate/ktforummandate_2.pdf
Chambers, C. D., & Tzavella, L. (2021). The Past, Present, & Future of Registered Reports. Nature Human Behaviour, 6, 29-42. https://doi.org/10.1038/s41562-021-01193-7
de la Solla Price, D. (1961). Science since Babylon. Yale University Press.
Doyle, J. (2018). Reconceptualising research impact: Reflections on the real-world impact of research in an Australian context. Higher Education Research & Development, 37(7), 1366\u20131379. https://doi.org/10.1080/07294360.2018.1504005
Fern\u00e1ndez Pinto, M. (2020). Open Science for private Interests? How the Logic of Open Science Contributes to the Commercialization of Research. Frontiers in Research Metrics and Analytics, 5, 588331. https://doi.org/10.3389/frma.2020.588331
Haeffel, G. J. (2022). Psychology needs to get tired of winning. Royal Society Open Science, 9(6), 220099. https://doi.org/10.1098/rsos.220099
Hardy, G. H. (1908). Mendelian Proportions in a Mixed Population. Science, XXVIII, 49\u201350.
Hardy, G. H. (1945, 2005). A Mathematician's Apology. http://www.arvindguptatoys.com/arvindgupta/mathsapology-hardy.pdf
Henkel, M. (2005). Academic identity and autonomy in a changing policy environment. Higher Education, 49(1\u20132), 155\u2013176. https://doi.org/10.1007/s10734-004-2919-1
Hogan, J. (2011). Is higher education spending more on administration and, if so, why? Perspectives: Policy and Practice in Higher Education, 15(1), 7\u201313. https://doi.org/10.1080/13603108.2010.532316
Holcombe, A. O. (2019). Contributorship, Not Authorship: Use CRediT to Indicate Who Did What. Publications, 7, 48. https://doi.org/10.3390/publications7030048
Hostler, T. J. (2023). The Invisible Workload of Open Research. Journal of Trial & Error. http://doi.org/10.36850/mr5
Hosseini, M., Senabre Hidalgo, E., Horbach, S. P. J. M., G\u00fcttinger, S., & Penders, B. (2022). Messing with Merton: The intersection between open science practices and Mertonian values. Accountability in Research, 1\u201328. https://doi.org/10.1080/08989621.2022.2141625
Hyland, K., & Jiang, F. (Kevin). (2021). 'Our striking results demonstrate \u2026': Persuasion and the growth of academic hype. Journal of Pragmatics, 182, 189\u2013202. https://doi.org/10.1016/j.pragma.2021.06.018
Institute of Physics. (2023). Bin the Boffin. Institute of Physics. https://www.iop.org/sites/default/files/2023-05/Bin-the-Boffin-pamphlet-v2.pdf
Jessop, B. (2018). On academic capitalism. Critical Policy Studies, 12(1), 104\u2013109. https://doi.org/10.1080/19460171.2017.1403342
Leonelli, S. (2023). Philosophy of Open Science (1st ed.). Cambridge University Press. https://doi.org/10.1017/9781009416368
Macfarlane, B. (2011). The Morphing of Academic Practice: Unbundling and the Rise of the Para-academic: The Morphing of Academic Practice. Higher Education Quarterly, 65(1), 59\u201373. https://doi.org/10.1111/j.1468-2273.2010.00467.x
Macfarlane, B. (2021). The spirit of research. Oxford Review of Education, 47(6), 737\u2013751. https://doi.org/10.1080/03054985.2021.1884058
Muthukrishna, M., & Henrich, J. (2019). A problem in theory. Nature Human Behaviour, 3(3), 221\u2013229. https://doi.org/10.1038/s41562-018-0522-1
Nelson, G., Monson, M. J., & Adibifar, K. (2020). The gig economy comes to academia: Job satisfaction among adjunct faculty. Cogent Education, 7(1), 1786338. https://doi.org/10.1080/2331186X.2020.1786338
Rees, T. (2015). Developing a Research Strategy at a Research Intensive University: A Pro Vice Chancellor's Perspective. In R. Dingwall & M. McDonnell (Eds.), The SAGE Handbook of Research Management (pp. 565\u2013580). SAGE Publications Ltd.
Santos, B. de S. (Ed.). (2008). Another knowledge is possible: Beyond northern epistemologies. Verso.
Shepherd, S. (2008). Managerialism: An ideal type. Studies in Higher Education, 43(9), 1668\u20131678. https://doi.org/10.1080/03075079.2017.1281239
Spellman, B. A. (2015). A Short (Personal) Future History of Revolution 2.0. Perspectives on Psychological Science, 10(6), 886\u2013899. https://doi.org/10.1177/1745691615609918
Stewart, A. J., Farran, E. K., Grange, J. A., Macleod, M., Munaf\u00f2, M., Newton, P., Shanks, D. R., & the UKRN Institutional Leads. (2021). Improving research quality: The view from the UK Reproducibility Network institutional leads for research improvement. BMC Research Notes, 14(1), 458. https://doi.org/10.1186/s13104-021-05883-3
Stewart, S. L. K., Pennington, C. R., da Silva, G. R., Ballou, N., Butler, J., Dienes, Z., Jay, C., Rossit, S., Samara, A., & U. K. Reproducibility Network (UKRN) Local Network Leads. (2022). Reforms to improve reproducibility and quality must be coordinated across the research ecosystem: The view from the UKRN Local Network Leads. BMC Research Notes, 15(1), 58. https://doi.org/10.1186/s13104-022-05949-w
Thelwall, M., & Maflahi, N. (2022). Research coauthorship 1900\u20132020: Continuous, universal, and ongoing expansion. Quantitative Science Studies, 3(2), 331\u2013344. https://doi.org/10.1162/qss_a_00188
Tiokhin, L., Panchanathan, K., Smaldino, P. E., & Lakens, D. (2021). Shifting the level of selection in science [Preprint]. MetaArXiv. https://doi.org/10.31222/osf.io/juwck
Truscot. (1943). Red Brick University. Faber & Faber.
T\u00fcl\u00fcba\u015f, T., & G\u00f6kt\u00fcrk, \u015e. (2023). A meta-synthesis on academic identity in the neoliberal context of academy. Higher Education, 86(1), 139\u2013161. https://doi.org/10.1007/s10734-022-00906-9
UK Science, Innovation, and Technology Committee. (2023). Reproducibility and Research Integrity Report (Sixth report of the session 2022-2023). https://committees.parliament.uk/publications/39343/documents/194466/default/
UKRI. (2023). Open Research. https://www.ukri.org/what-we-do/good-research-resource-hub/open-research/
University of Bristol. (n.d.). How to Fund Research Software or Cost It into Grants. https://www.bristol.ac.uk/acrc/research-software-engineering/software-howtos/how-to-fund-software/
Weale, S. (2021, July 20). Funding cuts to go ahead for university arts courses in England despite opposition. The Guardian. https://www.theguardian.com/education/2021/jul/20/funding-cuts-to-go-ahead-for-university-arts-courses-in-england-despite-opposition
Whitchurch, C. (2008). Shifting Identities and Blurring Boundaries: The Emergence of Third Space Professionals in UK Higher Education. Higher Education Quarterly, 62(4), 377\u2013396. https://doi.org/10.1111/j.1468-2273.2008.00387.x
Ziman, J. (2003). Non-instrumental roles of science. Science and Engineering Ethics, 9(1), 17\u201327. https://doi.org/10.1007/s11948-003-0016-y
Ziman, J. M. (2000). Real Science: What It Is and What It Means. Cambridge University Press.
Hostler, T. (2023, August 8). Scientific reform, post-academic research, and academic identity. Critical Metascience. https://markrubin.substack.com/p/scientific-reform-post-academic-research
Tom Hostler is a lecturer at Manchester Metropolitan Univeristy. For more information about his work, please visit: https://scholar.google.com/citations?user=ZKMS8m0AAAAJ&hl=en To contact Tom, please email: t.hostler@mmu.ac.uk
Not a universally admired term, as highlighted by the Institute of Physics recent \"Bin the Boffin\" campaign (Institute of Physics, 2023).
Although it should be noted that project has a dedicated \"knowledge transfer forum\", with the mandate to \"[ensure] that new technology disclosures presented by CERN\u2026 are brought to the attention of potentially interested industrial and academic partners in their country\", showing that funding is partly based on potential applied utility. (CERN, n.d.)
Yet paradoxically, this has not reduced workload: academics are still required to engage with these professionals, which takes significant time (Hogan, 2011).