I just tried this and actually found it more natural to pull-off downward rather than push-off. Maybe I’m just weird, but I typically do that when pulling-off on the D string.
I ran into the same problem you did when pushing-off, though I’d guess that there is a possible remedy if you feel like you must push-off: practice extremely slowly and use extra pressure with the middle finger, maybe even pulling the string back slightly, so it doesn’t roll off. This is a really small movement and a tough habit to build, honestly, and it might just be easier to pull-off downward for that note.
I’d be interested in Ben’s opinion, as I notice he’s playing his Phantom banjo which has a tunneled 5th string. It’s got quite a bit of extra fretboard real estate on that side, meaning pushing off would never be a problem.