It's not really a glitch. But it was largely unintentional in Smash 64. Whatever algorithm decided how high characters flew into the air after a shield break went apeshit when it got to Jiggs, and sent her flying off the top.
Naturally, everyone found this hilarious, and it's been a part of the character ever since.
Short story, don't get shield-broken as Jigglypuff.
There is no algorithm to shield break heights. All characters have an attribute that is the shield break height. It's the same way both in melee and brawl. It is most definitely the same way in smash 64 as well.