PoetX, Transformer & LoRA Comparison
PoetX is a research project that compares three character-level language model architectures trained on Shakespeare's complete works: a baseline Transformer, a LoRA-augmented Transformer, and the PoetX model incorporating orthogonal rotation layers. The project demonstrates loading pre-trained checkpoints, generating text from each model, and comparing different parameter-efficient approaches for language modeling within a single notebook-based workflow.
Challenge
The project explores different approaches for improving or modifying a character-level Transformer language model while keeping the implementation lightweight. The primary challenge was implementing and comparing multiple model variants—including LoRA-based adaptation and orthogonal rotation layers—using the same dataset and generation pipeline for consistent comparison.
Complexity & Innovation
- The repository implements three distinct language model architectures sharing a common Transformer backbone while introducing different parameter adaptation techniques. Alongside a standard Transformer and a LoRA-based variant, the PoetX architecture introduces orthogonal block-rotation layers before and after linear projections.
- The project also provides utilities for loading pretrained checkpoints and generating text from each architecture within a unified comparison pipeline.
Process
- Load and tokenize the Shakespeare dataset.
- Initialize the three language model variants.
- Load pretrained model checkpoints.
- Generate text from the same prompt using each model.
- Compare the generated outputs.
Feature Inventory
- Multiple Language Model Variants: The project implements and compares three language model architectures: a baseline Transformer, a LoRA-enhanced Transformer, and the PoetX model with orthogonal rotation layers. Each model shares the same Transformer foundation, enabling direct architectural comparison.
- Custom Parameter-Efficient Layers: The repository includes implementations of LoRALinear, BlockOrthogonal, and POETLinear, demonstrating different approaches for modifying linear layers. These components form the basis of the LoRA and PoetX model variants and allow experimentation with alternative parameterization strategies.
- Checkpoint Loading & Text Generation: Pretrained checkpoints for each model can be loaded directly within the notebook, after which each model generates text continuations from the same prompt using top-k sampling and temperature scaling. This provides a simple workflow for comparing qualitative generation behavior across architectures.
Conclusion
Serves as an experimental framework for comparing parameter-efficient Transformer architectures on character-level language modeling.