An end-to-end research pipeline that converts Large Language Model (LLM) reasoning traces into a Reasoning Language Model (RLM) dataset for verifier model training. The project extracts reasoning steps from public datasets, generates corrupted reasoning examples, constructs a balanced verification dataset, and trains a sequence classification model to distinguish between correct and incorrect reasoning steps. The processed dataset can then be exported for downstream reasoning and verification research.
Training reasoning verification models requires datasets containing both correct and incorrect reasoning steps. Since public reasoning datasets primarily contain valid solutions, the challenge was to automatically generate realistic negative examples while preserving the reasoning context. The generated examples needed to remain sufficiently similar to valid reasoning so that the verifier model could learn to identify subtle logical errors rather than relying on simple patterns.
The project addresses this by systematically corrupting reasoning steps and constructing balanced datasets suitable for verifier model training. The process also needed to maintain consistent data formatting, accurate labeling, and sufficient diversity across reasoning examples to support reliable model training and evaluation. This approach enables the creation of scalable verification datasets while reducing the need for manually labeled incorrect reasoning traces.
The pipeline extracts individual reasoning steps from public reasoning datasets and automatically constructs labeled training samples containing the original question, previous reasoning context, current reasoning step, and verification label. This creates a structured dataset suitable for reasoning verification tasks.
To generate negative training examples, the system introduces both numerical calculation errors and logical reasoning mistakes into otherwise correct solutions. Every original reasoning step produces one positive sample and multiple negative samples, resulting in a balanced dataset without requiring manual annotation.
The project fine-tunes a DistilBERT sequence classification model to distinguish between correct and corrupted reasoning steps. After tokenization and dataset splitting, the model is trained and evaluated using the Hugging Face Trainer, producing a verifier checkpoint that can be reused for downstream reasoning research.
The Reasoning LLM Pipeline provides a complete research workflow for transforming LLM-generated reasoning into structured datasets suitable for reasoning verification. By combining automated reasoning extraction, synthetic error generation, balanced dataset construction, and verifier model training, the pipeline enables a systematic approach to studying the quality and reliability of model-generated reasoning. The project supports experimentation with Reasoning Language Models and model-based verification of chain-of-thought reasoning by providing flexible and reusable data processing stages. Its modular design also allows researchers to experiment with different datasets, corruption techniques, and verification models, making it a useful foundation for further research into reliable and interpretable AI reasoning systems.