PoetX, Transformer & LoRA Comparison
PoetX is a research project that compares three character-level language model architectures trained on Shakespeare's complete works: a baseline Transformer, a LoRA-augmented Transformer, and the PoetX model incorporating orthogonal rotation layers. The project demonstrates loading pre-trained checkpoints, generating text from each model, and comparing different parameter-efficient approaches for language modeling within a single notebook-based workflow.
Challenge
The project explores different approaches for improving or modifying a character-level Transformer language model while keeping the implementation lightweight. The primary challenge was implementing and comparing multiple model variants—including LoRA-based adaptation and orthogonal rotation layers—using the same dataset and generation pipeline for consistent comparison. The models needed to maintain a common training and inference setup so that differences in generated text could be attributed to the architectural or parameter-efficient modifications rather than variations in the experimental process. Another challenge was integrating pre-trained checkpoints and ensuring that each model variant could be loaded, executed, and evaluated within a unified notebook workflow. The project also required a simple and reproducible setup for generating outputs from each architecture, making it easier to analyze their behavior and compare the effectiveness of different approaches to character-level language modeling.
Complexity & Innovation
- The repository implements three distinct language model architectures sharing a common Transformer backbone while introducing different parameter adaptation techniques. Alongside a standard Transformer and a LoRA-based variant, the PoetX architecture introduces orthogonal block-rotation layers before and after linear projections.
- The project also provides utilities for loading pretrained checkpoints and generating text from each architecture within a unified comparison pipeline.
Process
- Load and tokenize the Shakespeare dataset.
- Initialize the three language model variants.
- Load pretrained model checkpoints.
- Generate text from the same prompt using each model.
- Compare the generated outputs.
Feature Inventory
- Multiple Language Model Variants: The project implements and compares three language model architectures: a baseline Transformer, a LoRA-enhanced Transformer, and the PoetX model with orthogonal rotation layers. Each model shares the same Transformer foundation, enabling direct architectural comparison.
- Custom Parameter-Efficient Layers: The repository includes implementations of LoRALinear, BlockOrthogonal, and POETLinear, demonstrating different approaches for modifying linear layers. These components form the basis of the LoRA and PoetX model variants and allow experimentation with alternative parameterization strategies.
- Checkpoint Loading & Text Generation: Pretrained checkpoints for each model can be loaded directly within the notebook, after which each model generates text continuations from the same prompt using top-k sampling and temperature scaling. This provides a simple workflow for comparing qualitative generation behavior across architectures.
Conclusion
The PoetX, Transformer & LoRA Comparison case study demonstrates a practical approach to evaluating different character-level language model architectures under a consistent experimental setup. By comparing a baseline Transformer, a LoRA-augmented Transformer, and the PoetX model with orthogonal rotation layers, the study highlights how architectural and parameter-efficient modifications can influence language modeling and text generation.