PrototypeCoverage and scoring depth are still improving. Internal prioritization uses The Unjournal's Coda workflow; this public page supports discovery, comments, suggestions, and feedback.
AI-assisted prioritization for The Unjournal evaluation
About this prioritization tool What the scores mean, sources, methodology, and feedback
What is The Unjournal?
We commission and publish independent, public evaluations of research that can inform high-stakes global decisions. We focus on economics, quantitative social science, forecasting, and policy-relevant research—including development economics, global health, animal welfare, AI governance, climate policy, and catastrophic risks.
Learn more →
Early prototype (March 2026). Coverage and scoring depth will improve as we expand sources and
incorporate human feedback. Scores are AI-generated suggestions to help identify candidates for evaluation.
How it works: Papers are automatically discovered from
multiple academic sourcesCurrently scanning: NBER (economics working papers), arXiv (econ, quantitative finance, and cs.CY for AI governance/social impact), CEPR (European economics), EA Forum (effective altruism research links), Semantic Scholar (AI-powered search by cause area), OpenAlex/SSRN (social science preprints), RePEC (economics working papers), Anthropic Economic Research & Societal Impacts team pages, DeepMind and AI governance org papers (GovAI, CSET, GPI). New papers are fetched periodically and scored automatically.,
then scored by AI models against Unjournal's prioritization criteria.
Scores reflect
evaluation priorityHow strongly we recommend commissioning an independent Unjournal evaluation of this paper. This considers: (1) Is this research relevant to important global welfare decisions? (2) Would independent evaluation add value beyond existing peer review? (3) Is the paper at a stage where feedback can improve it? (4) Are the authors likely to engage? A high priority score does NOT mean the research is good or bad—it means evaluation would be particularly valuable.—the
expected value of commissioning an independent evaluation—not an assessment of research quality.
We welcome both team and public feedback; human ratings and written prioritization notes are used in the direct prioritization workflow and as calibration examples for improving the AI scorer.
Core principle
Prioritization = expected value of commissioning an evaluation, not quality endorsement.
A prominent but flawed paper may score HIGHER than a rigorous but obscure one, because independent evaluation adds more value there.
Two display lenses (toggle above the list)
The list can be scored and sorted through either of two transparent weightings of the six sub-scores — toggle
Evaluation-relevance weighting in the controls, and unfold What is this score? to see the live weights:
Criterion
Research relevance (readers)
Evaluation priority (UJ, default)
Decision relevance
40%
30%
Methodological potential
25%
10%
Real-world influence
20%
10%
Prominence / attention
15%
0%
Timing value
0%
20%
Neglectedness / value of evaluation
0%
30%
Research relevance ranks work by importance, rigor, and relevance for readers looking for useful research. Evaluation priority (the default) adds timing and value-of-evaluation (neglectedness) — the expected value of commissioning an independent Unjournal evaluation. If a sub-score is missing for a paper, its weight is dropped and the rest renormalized; out-of-scope papers are floored.
Model two-track scoring
Underneath the lenses, the AI model treats prominent and less-prominent work differently. Prominence is not a quality bonus for obscure work; it mainly tells us whether independent evaluation could matter because the work is already influential, or because a neglected but solid piece might otherwise be missed.
Criterion
Prominent work
Less-prominent work
Decision relevance
40%
30%
Timing value
25%
15%
Real-world influence
20%
20%
Methodological potential
10%
25%
Prominence / existing attention
5%
Context only
For prominent work (NBER, CEPR, World Bank, top journals), decision-relevance dominates; methodological weakness can make public evaluation more valuable rather than less. For less-prominent work, the data, identification strategy, clarity, and evaluability matter more, because serious flaws make evaluation less likely to change decisions.
Scoring rubrics (0–10 each)
1. Global Decision-Relevance (most important)
9–10: Directly informs active decisions by major funders or policymakers (GiveWell cost-effectiveness, WHO policy, climate treaty design). Specific organizations can be named.
7–8: Addresses a recognized global priority with clear policy implications, but the link to specific decisions is less direct.
5–6: Relevant to global welfare in a general sense. Interesting for the field but specific decision-relevance is moderate.
3–4: Tangentially related to global priorities. Primarily academic interest.
1–2: No clear connection to decisions affecting global welfare.
Field-specific: Development economics & LMIC health are our strongest areas. AI governance papers must be genuinely quantitative, not conceptual think-pieces. Animal welfare intervention evidence is highly valued.
2. Prominence
9–10: NBER working paper, top-5 journal, Nobel/Clark laureate, >500 citations, major media coverage.
1–2: Unknown author, self-published, no institutional backing.
Note: NBER/CEPR/World Bank/IMF sources are treated as high-attention signals, not proof of quality or publication stage. Some NBER papers are later published in journals, and this should be checked when timing matters. Prominent flawed work can be valuable to evaluate because people may already be using it.
3. Real-World Influence
9–10: Already cited in policy documents, GiveWell/Open Phil analyses, government reports. Named organizations are using this.
7–8: Likely to influence decisions soon. In an active policy debate. Authors have policy connections.
5–6: Could influence decisions if findings hold up. Relevant to active debates but not yet cited.
3–4: Academic contribution with indirect policy relevance.
1–2: Purely academic exercise with no clear path to influence.
4. Timing Value
9–10: Working paper/preprint released in last 6 months. No peer review yet. Authors actively seeking feedback.
7–8: Working paper 6–18 months old. Under review but not yet published.
5–6: Recently published (1–2 years) in a venue where more review would add value. R&R at journal.
3–4: Published 2+ years ago but still influential. Adds transparency but less urgency.
1–2: Old published work with established peer review. Feedback largely moot.
By methodology: RCTs & field experiments benefit most from early feedback (pre-registration, pre-analysis). Policy reports have narrow windows. Theoretical work is less time-sensitive.
5. Methodological Potential
For prominent work: This is a secondary consideration. If it’s prominent and decision-relevant, score 7+ and move on. Quality assessment is for the evaluation stage.
For less-prominent work (the tie-breaker):
9–10: Innovative methodology, strong identification strategy, credible real-world outcome data where possible, reproducible analysis with shared code/data.
7–8: Solid methods appropriate for the research question.
3–4: Methodological concerns that would make evaluation difficult.
1–2: Not really quantitative. Literature review, opinion piece, or purely conceptual.
Field-appropriate standards are illustrative, not mechanical. We should not penalize fields where RCTs are impossible, but observed choices and measured outcomes usually deserve more weight than recall or hypothetical stated-preference evidence when both are available.
Development/health: RCTs, DiD, regression discontinuity, IV
Environmental/climate: Integrated assessment models, panel data, natural experiments
AI governance: Mixed methods, surveys, formal models
Animal welfare: Revealed-preference or behavioral evidence where available; stated preference, DCEs, and welfare calculations where direct evidence is limited
Political science: Quasi-experimental, panel data, surveys
Macro/trade: DSGE, gravity equations, synthetic control
Score interpretation
Score range
Recommended action
What it means
75–100
Prioritize now
Strong candidate. Matches papers that were actually sent for evaluation by the UJ team.
50–74
Monitor
Borderline. In the range where the human team often disagreed.
25–49
Deprioritize
Below threshold. Matches papers human assessors scored low.
<25
Out of scope
Not quantitative social science, or fundamentally outside UJ coverage.
Suggesting — A paper is suggested (by AI or human) with a 0–100 rating and discussion of relevance
Assessing — A second team member gives an independent rating (without seeing the first)
Voting — If avg rating ≥ 65%, the field group votes (Strong Yes to Strong No)
Evaluation — An evaluation manager commissions 2+ public evaluations via PubPub
Comment directly on this page using the
Hypothes.is sidebar
(look for the < tab on the right edge of the page). Highlight any text and add your annotation —
visible to all Hypothes.is users. You can also use the feedback buttons on each paper card.
Show rating buttons on every card. Use this when you want to give quick prioritization ratings across many papers; written comments can still be added from each card's Details / Rate panel.
-
Shown
-
High eval. priority
What is this score? (weights for the active lens)Custom weighting & sorting
Adjust these sliders to create your own priority score, then sort by Custom weights. This changes only your browser view; it does not change The Unjournal scorer.
Quick-rate mode: for each paper, how strongly should The Unjournal commission an evaluation?
(-- Strong No … ++ Strong Yes).
Each click is saved to The Unjournal's server — you'll see ✓ recorded appear next to the buttons. Add your email below if you'd like follow-up.0 rated this session
Suggest research for this list
Know of high-impact research we should consider for evaluation — or other work this tool should be doing?
Leave an email if you're open to follow-up discussion; if we later introduce compensation for useful
contributions, earlier contributors will be grandfathered in.
Is this your own research? This form is for suggesting others’ work.
To submit your own work (or your organisation’s) for evaluation, please
use the author submission form ↓.
On The Unjournal team? Use the
team suggestion form ↓
instead — it adds the internal fields (your team identity, COI considerations, second-opinion routing, fast-tracking) and is the reward-eligible route.
Add more detail (optional) — helps us prioritise faster
Suggestion received — thank you!
Submit your own research
Are you an author — or a research organisation — who would like The Unjournal to commission a public
evaluation of your work? Submit it here. Submissions feed the same prioritization database.
(To suggest someone else’s research, use the suggestion form above.)
Questions: contact@unjournal.org.
Open the author submission form
Submission received — thank you!
Fields marked * are needed for us to process your submission.
Suggest research — Unjournal team
For Unjournal team members and field specialists. Feeds the same database as the public
suggestion form, but adds the internal fields (your team identity, COI considerations, second-opinion
routing, fast-tracking) and is the reward-eligible route.
Not on the team? Use the
public suggestion form above.
Open the team suggestion form
Prioritisation detail (optional)
Team-only
If the paper fits any of these, tick all that apply:
Suggestion received — thank you!
Fields marked * help us process and credit your suggestion.
Vision: How this tool will work
We are building an efficient, AI-augmented prioritization pipeline:
AI discovery & preliminary rating — The tool finds, vets, and suggests research from multiple sources (NBER, arXiv, SSRN, EA Forum, Anthropic/DeepMind research pages, GovAI/CSET/GPI, etc.), giving a preliminary score and adding it to the prioritization database.
Human suggestions — Team members and the public can also add research directly as a "suggester" or "submitter," in which case the AI provides an additional analysis report.
Notifications — Sign up for alerts when new high-potential research in your area is added.
Team assessment & human labeling — Team members review suggestions, find those of most interest, and give independent ratings. Together with the public quick-rate votes and suggestions collected on this page, these human judgments form a growing set of labeled examples — each one a paper paired with a human verdict on its evaluation priority.
Voting & decisions — The team votes (as in our current process), moving papers forward for commissioned evaluation. The voting outcomes themselves become further labels.
Learning loop — human labels improve the AI — The AI's preliminary scores are systematically compared against these accumulated human labels (team assessments, voting outcomes, and public quick-rate votes), and the model is recalibrated with field-specific corrections — extending the calibration already done against
353 past prioritization decisions.
As more humans rate more papers, the AI's discovery and preliminary scoring (step 1) get steadily more accurate — closing the loop.
The AI uses Unjournal's core principles and previous prioritization decisions as context, and every human rating you contribute — including the quick-rate votes above — feeds back to make its suggestions better.
We welcome your thoughts on this workflow — use the Hypothes.is sidebar or email
contact@unjournal.org.