A metric used to evaluate the effectiveness of a language model in predicting a given sample. It quantifies the uncertainty associated with the model’s predictions for the next word in a sequence. A higher perplexity value indicates poorer predictive performance. The perplexity formula is represented as:
PP(W) = P(w1, w2, …, wN)-1/N
Here, PP denotes perplexity, W is the test set, P signifies the probability of the test set, and N represents the number of words in the test set.
