PoetX is a research project that compares three character-level language model architectures trained on Shakespeare's complete works: a baseline Transformer, a LoRA-augmented Transformer, and the PoetX model incorporating orthogonal rotation layers. The project demonstrates loading pre-trained checkpoints, generating text from each model, and comparing different parameter-efficient approaches for language modeling within a single notebook-based workflow.
The project explores different approaches for improving or modifying a character-level Transformer language model while keeping the implementation lightweight. The primary challenge was implementing and comparing multiple model variants—including LoRA-based adaptation and orthogonal rotation layers—using the same dataset and generation pipeline for consistent comparison. The models needed to maintain a common training and inference setup so that differences in generated text could be attributed to the architectural or parameter-efficient modifications rather than variations in the experimental process. Another challenge was integrating pre-trained checkpoints and ensuring that each model variant could be loaded, executed, and evaluated within a unified notebook workflow. The project also required a simple and reproducible setup for generating outputs from each architecture, making it easier to analyze their behavior and compare the effectiveness of different approaches to character-level language modeling.
The project implements and compares three language model architectures: a baseline Transformer, a LoRA-enhanced Transformer, and the PoetX model with orthogonal rotation layers. Each model shares the same Transformer foundation, enabling direct architectural comparison.
The repository includes implementations of LoRALinear, BlockOrthogonal, and POETLinear, demonstrating different approaches for modifying linear layers. These components form the basis of the LoRA and PoetX model variants and allow experimentation with alternative parameterization strategies.
Pretrained checkpoints for each model can be loaded directly within the notebook, after which each model generates text continuations from the same prompt using top-k sampling and temperature scaling. This provides a simple workflow for comparing qualitative generation behavior across architectures.
The PoetX, Transformer & LoRA Comparison case study demonstrates a practical approach to evaluating different character-level language model architectures under a consistent experimental setup. By comparing a baseline Transformer, a LoRA-augmented Transformer, and the PoetX model with orthogonal rotation layers, the study highlights how architectural and parameter-efficient modifications can influence language modeling and text generation.
Overall, the case study provides a lightweight and reproducible framework for exploring efficient Transformer adaptations and serves as a foundation for further experimentation with alternative architectures and parameter-efficient learning techniques.