The three terms are not a rabbinic invention. They stand together, in one sentence, in the instruction for the scapegoat: Aaron is to lay both hands on the head of the goat and confess over it the iniquities of the people of Israel, and all their transgressions, in all their sins. They stand together again in the thirteen attributes proclaimed at Sinai, where God is described as bearing iniquity, transgression and sin. Whatever the writers intended, the three were already a set.
What they do not do in the Hebrew Bible is define themselves. The graded reading has to be recovered from how each root behaves elsewhere. Chet is built on a verb meaning to miss, and the book of Judges uses it of Benjaminite slingers who could aim at a hair and not miss; the noun therefore reads naturally as an act that went wide of its mark. Avon carries the sense of a bending or twisting, of something that has been made crooked. Pesha is the harshest of the three, because outside the cultic vocabulary it is a political word: it is what a vassal kingdom does when it breaks with its overlord, and the books of Kings use it exactly that way of Moab against Israel.
The rabbinic literature then fixes the scale. The Babylonian Talmud, in the tractate that treats the Day of Atonement, glosses the trio as inadvertent offences, deliberate ones, and rebellions, and it preserves an argument about the order in which the high priest was to name them. The argument is not pedantry. Confessing upward from the least, or downward from the worst, encodes two different accounts of what a confession is for, and the liturgy that came out of it has had to make a choice on every Day of Atonement since.