I have medium-long hair and it seems to take me forever to dry it. I've found that if I pull it up and do it in layers it goes a lot faster. I do the same thing for curling/straightening and just do it in layers.
It probably takes me about 15 minutes to either curl or straighten my hair after it's dry by doing it this way.
My hair is about 3-4 inches below my shoulders and it takes a couple hours to air dry my hair. Blow drying only 10min. Straightening or curling takes me about 20 min. I straighten and curl my hair the same as you do, in layers.
Maybe 10 minutes to dry, it is about an inch or so above my shoulders. Then if I leave it straight (natural) I use a round brush while drying and that adds in an extra 10 minutes. If I curl it, the curling takes me about 10-15 minutes.