Because it is followed by what appears to be the genitive marker 叱 (*-s), 第 is probably a bound noun. Thus 音 may have been the original ordinal-deriving suffix, with 第 added at some later point. See a possible Modern Korean reflex of *-m in 이듬 (ideum, “second plowing of the fields”), 이듬해 (ideumhae, “next year”).
音第叱 (*-m CAHI-s)
音 and 叱 are well-accepted Old Korean phonograms for coda consonants *-m and *-s respectively. The Old Korean orthography gives no clues as to how 第 is supposed to be read, as it is being used solely as a logogram for its semantic value here (see Chinese 第 (dì)), but it is conventionally reconstructed as *CAHI, because it appears to be a bound noun and Middle Korean features the ordinal-deriving bound noun 자히 (Yale: cahi), which became Modern Korean 째 (-jjae) with the same function. Note that there is no actual evidence for this reconstruction of convenience.
In Middle Korean, *-m and *-s have both disappeared. 자히 (Yale: cahi) combines to the number via an intervening genitive particle ㅅ (Yale: -s), and is directly followed by the noun.