I do not think this "busy thing" is unique to pastors. I think this is a problem we all face in culture today, especially with the use of all the wonderful technology, which is a blessing AND a curse. 24 hour connectivity is great for those who need to get a hold of you, but isn't good for the soul.
Take (1) day of rest, a sabbath - Exodus 20:8–11
It's important to have (1) day you cease from your work. Do what you love to do. On that day do only what refreshes you, builds you up, encourages you and makes you smile. Make sure to surround yourself with people who make you laugh, smile and feel good about life.
I must warn you, guard this day with your life! And invite close friends and family to hold you accountable to it. People will try to steal it from you, not intentionally of course, but it will happen. You are often the only one who can stand guard over your life, do it with all diligence! Your Soul will be glad!
I read an amazing book that touches on this topic even more, called The Top Five Regrets of the Dying, and I did a review here!