It may be your rhythm isn't so bad until you turn on the metronome. I know I had this problem when I first started working with a metronome. I'd focus so much on listening for the clicks, I'd lose the feel of my strumming. The metronome seemed like it just confused things. It does get easier with regular use, though.
I'd suggest going as slowly as necessary to stay with the metronome at first - even if it's a lot slower than you can actually play. I've been working with a metronome for 30 minutes to an hour every day for a year or so, and it seems way less obtrusive now. I hardly even notice it's on unless I'm having rhythmic problems.
I think regular use of a metronome is really valuable. I know it has bred a lot of confidence for me at jams. Before the metronome, my first instinct would be to blame myself if I got off rhythm at a jam; after the metronome, I realized it usually wasn't me, and I'm a lot more assertive about my playing in those situations, now.