Overview
Unaligned is a scratchpad repo for running small‑scale Direct Preference Optimization (DPO) experiments with TinyLlama variants: training, evaluation, and simple inference/chat CLIs.
What’s inside
- DPO training script with LoRA adapters (TinyLlama base)
- Logging of preference metrics (accuracy, reward margins) and diagnostics
- CLI tools for single‑prompt generation and interactive chat
- Checkpoint management and merged‑weights export
Notes
- CPU‑friendly tips (shorter max lengths, smaller evals)
- Safety disclaimer: datasets may include toxic content; use for research only