TL;DR: we can transplant >80% instruction-following performance from small models to large models, without actually tuning them.
Table of Contents Emulated finetuning (EFT) A model-as-rm perspective EFT for scale decoupling Experiments Observation 1: pretraining vs finetuning ⇒ factuality vs helpfulness Observation 2: EFT enables dynamic test-time reward interpolation Observation 3: speculative decoding speeds up EFT up-scaling Observation 4: up-scaling can be further amplified Observation 5: …
Generated reasoning is faithful to the model’s true reasoning, if it can “accurately represents the reasoning process behind the model’s prediction”. This is particularly important 1) in high-stakes settings, such as medical decision-making, and 2) for gaining a better understanding of how reasoning works in LLMs. This work provides a timely investigation into the faithfulness of CoT reasoning for LLMs, adding to previous research that suggests LLM-generated reasoning may not be faithful.