Hey Jesse,
I’m very late to this conversation (Covid and all) but I have some thoughts to share.
First,
Free improvisation probably negates the idea that you are following theoretical “rules”. However, if you apply some music theory rules to your improvisations, then you might find that other folks will understand your improvisations better and in turn enjoy listening to the more.
This is why I always ask great improvisers (like David Grier) “what are you thinking when you play over (such and such)”? I am trying to understand their reasons for their improvised choices.
Many years ago, I was talking to a good friend, who was a very good improviser (bebop jazz and free jazz styles), about teaching improvisation. He stated blankly, “you can’t teach improvising”. I disagree. It can be taught as long as you understand what works and why.
First off, there are 2 distinct styles of improvising: following a melody and free improvising. Bluegrass “tends” to follow the first thought. When following a melody to create your improvisation, the listener should be able to “hear” the tunes melody weaving in and out of the improvisation. This creates a foundation for the improvisation and tends to require very little from the listener as far as trying to understand what is going on at any given moment (it is easy to listen to). Free improvisation tends to require more creativity from the player and closer listening from the listener. Free Improvisation generally just follows the basic harmony of the song and allows the player to create and completely new melodic line. Still, there are rules to such creativity that have to do with helping the listener to understand your improvisation.
At this point, I generally teach the six steps to creating an improvisation. Keep in mind that these are often taught to absolute beginners to improvisation and are meant to create a “map” for getting through a solo. These steps are for those who might like to try their hand at “Free Improvisation”.
Playing a solo is a lot like telling a story. Unfortunately, notes are not full words by themselves and most folks find listening to someone improvise to be difficult to follow. So your job as the improviser is to give the solo to the listener in a way that will feel organic, following a story line. Here are the 6 steps and why you should try to follow them:
-
Start with a simple melody. If you are free improvising, then this would be a simple 4 to 7 note phrase. This should be simple enough that a blind, deaf person could hum it the first time hearing it. SIMPLE.
-
Repeat the melody. You want to do this to create familiarity. If you are playing to a room of people, most will not hear what you first played. If you repeat it, half the people in the room may look up and think, “I heard that somewhere before”, not realizing that you just played it seconds earlier. You are just trying to get their attention and make them comfortable. If instead you choose to play something completely new, you will lose your audience immediately.
-
Elaborate on the melody. This means that you may take that original simple melody and change the rhythm slightly. Or you may add a note (or possibly 2) to your original melody. DO NOT play something totally new yet. You want to pull the audience into your story. If you change things up too soon, you will lose them.
-
Create tension. Now that you have your audiences’ attention, you want to make it more fun for them. Creating tension in your solo will make your “story” come alive. There are many ways to create tension:
• A. Speed: playing faster will create tension. There are whole styles of music that rely near solely on speed to create tension (energy) in a solo, like hard rock/metal. You do not need to be Steve Vai to create decent tension with the addition of some speed to your solo.
• B. Pitch: higher notes create tension. Play the same lick on the lowest notes of your guitar and then play it up and octave and then finally up another octave. You can see immediately that higher notes bring energy or tension to a solo.
• C. Tricks: This is actually a large category and includes things like double stops (two notes at a time), excessive bending (bending or sliding in and out of blue notes repeatedly, finger tapping (not a lot of this in bluegrass (why not?)), octaves (playing 2 notes at the same time that are octaves from each other, cross-picking (a common feature of bluegrass) Fast cross-picking can be quite dramatic….etc.
• D. Melodic tension: Playing in and out of a specific key. This can also be quite dramatic if used sparingly and properly. For example, you might play and short riff (3 or 4 notes) that is within the key and move that riff chromatically (by half step or 1 fret) up or down the fingerboard till you are in a position that has the notes back in the key again. If you do this right, you sound amazing, do it wrong (by ending outside the key) and you sound like you have lost your mind. 
• E. Rhythmic tension: This could either encompass the concept that you are rushing to the end of phrases early, ending the phrase late, or superimposing a rhythm on top of the song that is outside the rhythm of the tune (playing 3 beats on top of 2 or 5 notes where there would normally be 4 etc.)
-
Climax the solo/improvisation: As with a great meal, or a good book, there needs to be a climax to the solo. This generally happens near the end of the solo. In 32 bars, this might happen around bar 28 – 30). This is the point where you have built up the most tension in your solo.
-
Release tension/Repeat the original melody: This is self explanatory but think about going back to the melody and bringing the tension back down by staying in thekey and slowing down and playing lower notes…etc
To understand this better, think of this as a short story a teen might tell their mother:
“Hey Mom, I’m going down to the store to pick up a loaf of bread”
obviously, the mother doesn’t really hear the son and she responds, “WHAT?!?”
so he repeats
“Hey Mom, I’m going down to the store to pick up a loaf of bread”
and elaborates
“Mom, I’m gonna pick up some rye bread at the store”
Mother is sort of getting what he just said. But now he wants to have some fun with her so he creates some tension.
“While I’m at the store, I’m gonna pick up some beer and rolling papers and stop by my friend’s house”
The mother eyes open wide and she runs to the room where her son is.
The boy climaxes the story, “When I get there, he has some hooker friends and we are going to have the party of our lives. Don’t worry though cause I’ll drive slowly on the way home cause I’ll be totally blasted by that time”
Stunned, the mother screams.
The boy laughs and says, “I’m kidding mom, I’m just going to the store to pick up some bread”.
He drew her in, he elaborated, created tension, climaxed the story and let her off easy.
That is the first lesson to learning what to do with all those notes under your fingers (you need to know the entire fingerboard pretty well to feel comfortable do this). However, you can apply these thoughts/tools even if you only know a small section of the fingerboard.
This first clip of the tune Summertime is sort of a combination of following the melody (you can hear it rear it’s ugly head throughout the solo) and free improvising. You will notice that in the last chorus, I play the highest notes in the solo and the I move away from the melody as I get further into the solo.
To be clear, each style of music has it’s own rules for improvisation. For example, bluegrass needs you to start and end each phrase on a chord tone (very different from free jazz). And the way you incorporate blue notes into a solo and how they are set up is very different between bluegrass, straight blues, swing jazz, free jazz and folk/pop music.
If you are interested, I can elaborate another time.
Take care,
Mike
(thanx for pointing it out!!) Truth in advertising - I did say it was all way over my head though 
! Just plz excuse me & carry on 
!