The Unjournal
Journal-independent evaluation of impactful research
We are a group of researchers and open science advocates building a better system for evaluating rigorous, impactful research. Organizer and contact point: David Reinstein
We are not a journal! The Unjournal will not 'publish any papers in a journal'; we will organize and fund public journal-independent feedback, rating, and evaluation of hosted papers and dynamically-presented research projects. Our initial focus: quantitative work that informs global priorities, especially in economics, policy, and social science. We will encourage better research by making it easier for researchers to get feedback and credible ratings on their work.
Why? Peer review is great, but academic publication processes are wasteful, slow, and rent-extracting. They discourage innovation and encourage a great deal of effort spent 'gaming the system'. We will both provide an immediate alternative and also offer a bridge to a better system.
Identifying, soliciting, and choosing relevant research work, to be hosted on any open platform in any format that can gain a time-stamped DOI. Authors can present their work in the ways they find most convenient, complete, and legible. We will encourage and enable dynamic documents and other formats that promote replicability and open science. (See: the benefits of dynamic docs)
We will also directly evaluate high-impact work in the high-profile NBER working papers series (see our direct evaluation track).
Eliciting quantifiable and comparable metrics of research quality as credible measures of value. (Why quantitative metrics?)
Public evaluation: Reviews will typically be made public, along with authors' possible replies. This will facilitate dialogue and a greater understanding of research.
Linking work but not 'publishing it'. This process will not be 'exclusive': authors can 'submit their work to a journal' at any point. This will also help benchmark our evaluations against 'traditional publication outcomes'.
Awarding financial prizes (combined with public presentations) for work judged strongest
Transparency: Aiming to be as transparent as possible in our processes and judgments
For example, this proposal is closely related to Elife's "Publish, Review, Curate" model; see their updated (Oct 2022) model here. (However, we cover a different research focus, and make some different choices, discussed below.)
Below, we discuss other Parallel/partner initiatives and resources, many of whom we hope to work with. However, we think we are the only group funded to do this in this particular research area/focus. We are also taking a different approach to previous efforts, including funding evaluation (see 'Why pay evaluators (reviewers)'?) and asking for quantified ratings and predictions (see Guidelines for Evaluators).
Our current funding comes from ACX/LTFF grant proposal (as submitted, successful) grant (ACX passed it to the Long Term Future Fund, who awarded it). This funding will run out in February 2023
We have submitted some other grant applications. E.g., see our unsuccessful: FTX application HERE, other grant applications are linked below. We are sharing these in the spirit of transparency.
We are still seeking more funding, several applications are pending.
Academics and funders have complained about this stuff for years, and continue to do so every day on social media... and we suspect readers will agree with most of our critiques of the traditional review/publication process.
So why hasn't academia and research community been able to move to something new? There is a difficult collective action problem. Individual researchers and universities find it risky to move unilaterally. But we think we now have a good chance of finally changing this and moving to a better equilibrium.
- We can take risks: Many people involved with The Unjournal management ourselves are not traditional academics; we can stick our necks out. We are also bringing on board established senior academics who are less professionally vulnerable.
- Bringing in new interests, external funding, and incentives: There are a range of well-funded and powerful organizations – such as the Sloan Foundation and Open Philanthropy – with a strong inherent interest in high-impact research being reliable, robust, and reasoning transparent. This support can change the fundamental incentive structure.
- Less risky 'bridging steps': In particular, as noted above, The Unjournal allows researchers to also submit their work to traditional journals. In fact, this will provide a benchmark to help our quantitative ratings build and demonstrate their value.
- Communicating with researchers and stakeholders to make our processes easy, clear, and useful to them
- Making our output useful: It may take years for university departments and grant-funders to incorporate journal-independent evaluations as part of their metrics and reward systems. The Unjournal can be somewhat patient: our evaluation, rating, feedback, and communication will provide a valuable service to authors, policymakers, and other researchers in the meantime.
- Technology is improving: A new set of open-access tools (such as those funded by Sloan Scholarly Communications) makes what we are trying to do easier, and makes 'formats other than frozen pdfs' more and more useful every day.
- Rewarding early adopters with prizes and recognition: We can replace 'fear of standing out' with 'fear of missing out'. In particular, authors and research institutions that commit to publicly engaging with evaluations and critiques of their work should be commended and rewarded. And we intend to do this.
Organizes ideas and resources, and tracks our progress towards The Unjournal's twin goals of:
- 1.Making ‘peer evaluation and rating’ of open projects (instead of conventional 0/1 publishing of frozen pdf's) become a standard 'high status' outcome in academia/research, especially in economics and social sciences
- 2.Creating and coalescing around an efficient system for 'publishing', gaining credibility, and getting feedback for effective altruism and global-priorities-aligned research. (Making rigorous research more impactful, and impactful research more rigorous.)
Discussion space: I've set up a post-grant "'Unjournal'": Action plan discussion space" HERE. 25 Jun 2022 update: I have not kept the above discussion space fully updated.
Please let me know if you want edit/comment access to the present Gitbook.
Please do weigh in, all suggestions and comments will be credited... See also Unjournal: public-facing FAQ in progress.
Policies: projects, evaluation, workflow: How we choose papers/projects to evaluate, how we assign evaluators, etc.
Specific "Action steps" taken, in progress, and under discussion: Built founding committee, Action: Overall scoping
Last modified 21d ago