articLLMate

Active

An LLM-based auditing tool that reads a whole corpus of academic papers and audits how each one handles response-validity threats in self-report data — tested at a scale of nearly 50,000 articles.

View code

LLM Meta-science R Reproducibility Text mining Batch API

articLLMate points a large language model at a folder of academic articles and asks a focused, repeatable question of each one: how well does this paper actually handle threats to the validity of its self-report data — things like careless responding, social desirability, or acquiescence bias? Instead of a human reading thousands of methods sections by hand, the tool turns that audit into a structured, reproducible pipeline that can run across an entire literature.

It’s a working example of the lab’s broader interest in using LLMs as a meta-science instrument — auditing the research literature itself, not just individual participants’ data.

What it can do

Audit a whole corpus, not one paper at a time. Point it at a folder of articles, choose a prompt, and it produces one structured result per paper — from a handful of test files up to tens of thousands.
Run wherever your data-governance requires. Three execution modes share the same pipeline: API (OpenAI, Anthropic, or Gemini, one paper at a time), Local (private, on-device inference through Ollama, so sensitive text never leaves the machine), and Batch API (roughly 50% cheaper and rate-limit-free for large runs).
Ask your own question. The audit is defined by a swappable prompt plus a YAML schema, so the same machinery can score any construct — the current prompt audits response-validity handling; an earlier one classified papers by dimensional-vs-categorical framing.
Survive long runs. Every paper’s result is written to its own JSON file the moment the model responds, so a crash, timeout, or cancelled run resumes exactly where it stopped instead of starting over.
Hand you analysis-ready data. Results are flattened from nested JSON into three tidy CSVs (main / metadata / detail) plus a human-readable summary, ready to drop straight into R.

Big-number highlights: about 50,000 articles audited in the flagship study, three LLM providers supported, 50% cheaper in batch mode, and one JSON checkpoint saved per paper for crash-safe runs.

How it works

articLLMate is two pipelines in sequence, each driven by a single master script. The first turns a raw PDF corpus into clean, analysis-ready text; the second runs the audit and aggregates the results.

End-to-end flow: (1) preprocess a PDF corpus with GROBID into structured XML and strip it to the article body; (2) check the environment, discover articles, and select a prompt, then run the audit through one of three execution modes — API, local Ollama, or the batch API — saving one crash-safe JSON per paper; (3) aggregate everything into a summary report and three CSV tables.

1 · Preprocess the corpus. _MASTER_PREPROCESS_PIPELINE.R converts a folder of PDFs into machine-readable XML (via GROBID), matches each article to its Crossref metadata, and strips the markup down to the article body. It keeps a single audit ledger tracking every article from the original corpus through each cleaning stage, so nothing silently drops out of the sample.

2 · Run the audit. _MASTER_RUN_PIPELINE.R sources a series of modular steps: it validates the environment and API key, discovers the XML articles, loads the chosen prompt, and then runs the analysis in whichever execution mode you selected. Each paper comes back as structured JSON — with a relevance check (so off-topic papers are marked SKIPPED_IRRELEVANT rather than forced into an answer), the model’s chain-of-thought, and the audit fields defined by the prompt — and is saved immediately for crash safety.

3 · Aggregate. The final steps write a plain-language summary of the run and flatten the per-paper JSON into three CSV tables using a YAML mapping: main (one row per article, scalar fields), metadata (long-text fields like the reasoning trace and summary), and detail (long-format — one row per array element, e.g. each individual validity method a paper reported).

The whole run is configured from one block at the top of the master script — project name, execution mode, provider, model, prompt, input folder, and how often to checkpoint — so a new audit is a matter of editing a dozen lines and sourcing the file.

Tested at the scale of a literature

articLLMate isn’t a proof-of-concept that works on five papers. Honours student William Harvey ran the tool’s first full-scale study, auditing nearly 50,000 articles for how the published literature treats and operationalises dimensional versus categorical frameworks of psychopathology and personality — a question that would be impractical to answer by hand at that scale.

That study is what shaped the tool’s production features: batch mode (to make tens of thousands of model calls affordable), the crash-safe per-paper checkpointing (so a multi-day run survives interruptions), the relevance gate (to keep irrelevant hits out of the results), and the audit ledger (so the final sample is fully accountable back to the original corpus). A monitoring dashboard tracks progress live while a large run is under way.

The code

articLLMate is open source (MIT) and documented for reuse — including a step-by-step batch guide and preprocessing walkthrough:

github.com/ConalMonaghan/articLLMate

Because the audit is defined by a prompt and a schema rather than hard-coded, the same pipeline can be pointed at any research question that can be asked of a paper’s text — making it a general-purpose engine for auditing the literature, not just a single study.