Learning by ear is something that took a bit of time before I could call myself good at it. I would say it came on a timeline of weaning off tablature, listening to recordings a lot more, listening to other players more, etc. I remember the first time I actually used rolling backup. My old in-person banjo teacher went through Ben’s utility rolls with me and then told me to jump in as he started playing guitar and singing Man of Constant Sorrow. I discovered the dynamics of backup with those rolls, and my subconscious realized that I had more freedom than I thought I did playing backup.
I worked on smoothing out my chord changes, etc., and a while later my family and I attended a Doyle Lawson & Quicksilver concert. Joe Dean, the banjo player at the time, was playing rolls that sounded somewhat like the utility rolls. And since a lot of DLQ’s songs have a similar structure, I got to hear that roll over and over and over again, and I didn’t let it out of me. I didn’t have to learn it… it ingrained itself in me.
That style of roll, which has been used by like every single Scruggs-Crowe etc. style players, is my immediate foundation for backup.
So I kept on subconsciously learning to play by ear. I listened a LOT to DLQ, RSKT, etc. and could hear things I didn’t before.
This is why it’s sort of difficult for me to give rock solid advice, because I didn’t really make that much of a conscious effort to learn by ear. It just happened.
My ability to pick out things different from what I’ve been playing I think is kind of a side effect of seeing in my mind’s eye the things I regularly play over and over again. I also have synesthesia, so I think that helps me to visualize things too.
In the song you posted, the banjo is mirroring the whistle in the beginning. This is the basic melody of the song. The melody notes are something like D-F-G-G-G-g-G-A-G-F-D, but I’m thinking more along the lines of low5-flat7-1-1-1, etc., but I’m also not really thinking along those lines as much as I am subconsciously thinking “red-pink-silver-silver-dark blue…” You may not see the colors, but I think you can train yourself to see the intervals.
Hope this crazy ramble helped you somehow!