Hi Steve, and welcome to the forum!
I’m not Ben, but the answer you’ll most likely get is “yes, it’s a bad habit”. When you come back to picking you’ll be doing down-strokes on the “up” beat. This can/will effect what comes out of the instrument, in a number of ways, and may leave you hanging in the middle of a tune trying to figure out where to switch back.
To remedy the problem, you can try isolating just those few notes, or that measure. Slowing WAY down, concentrate on continuing your alternating pick-stroke while playing the hammer-on…in other words pick through it…then once you have that down, work at continuing that same hand motion while not actually picking the note of the hammer-on. Do this all slowly until it feels comfortable.
Hope this helps and is understandable,