Constrained Decoding

Constrained decoding is getting more and more attention in the field of large language models (LLMs). It aims to generate sequences of tokens that satisfy certain constraints. A typical example is to force the generation from LLM to satisfy a given JSON schema so that the generated JSON data can be used directly in a downstream application, such as tool use.

If you are familiar with constrained decoding, you might have occasionally heard people mentioning the word “posterior inference” in the context of constrained decoding. In this post, I will explain why constrained decoding can(should) be seen as a form of posterior inference.