I tried playing banjo for a bit and didn’t feel comfy with picks, so I kind of ran into the same question. The problem is that you won’t be able to get the traditional BG sound or volume without picks. There are styles of playing banjo that don’t use picks, but it won’t have the Scruggs sound. As a result, I’d say it’s not sacrilege to go without, but it won’t sound like typical “BlueGrass” banjo and it will limit your volume when playing with others.
I think part of problem before was that I didn’t tweak the finger picks to correctly fit my fingers. If I try to get into banjo again, I plan to get some picks, fit them and just do rolls until my hand gets used to it.
Just my two cents,