Trust the Lord and council with your Bishop when it comes to callings. During my divorce, I was called to nursery. I had all kinds of reasons that I could have resented the calling. Common gossip would say that this is where divorced women are sent because they aren't "worthy" to hold any other calling. I had five kids at home; I needed the break. One of my own kids was in nursery! I needed support of other women. I'd already served my time in nursery. I don't like little kids. But, the Lord knew my situation, my needs, and my heart. He knew how fragile I was. he knew that I sobbed through every sacrament meeting. He knew that my heart couldn't handle two additional hours of the usual teaching. He knew that I needed the gentle, simple teachings in nursery: "God loves you," "God made the trees," "God gave you loving parents." He knew that I needed the one-on-one visits with the other sisters in nursery. He knew that too many people and too many questions was more than I could handle. He knew that I was worthy to enter His temple and that His children needed a worthy teacher. He knew that my baby in nursery needed the extra time with me and he grew more confident. I served there a short time. I wasn't excited for the calling. I never loved it. I went home exhausted each week. I celebrated the day I was released. But, I am deeply grateful for the calling and a perceptive Bishop who extended it to me despite the fact that I might have been offended by it. I'm grateful for a loving Father in Heaven who gave me what I needed even when I didn't know it was what I needed and didn't want it anyway. We are blessed when we accept His gifts.
All that said, Bishops are human and sometimes I also think they are prompted to extend callings as a way to get us in their office and share what's going on in our lives. It's ok to council with your Bishop and explain your concerns with a new or current calling. Your time spent in council may result in solutions that hadn't occurred to you before, the calling being withdrawn, revised or released, or you accepting the calling with more peace. Whatever the case, don't just unilaterally make the decision. Take advantage of the opportunity for a two way discussion with the Lord's representative.
Callings are a gift from God. They aren't all fun, but they can all be a blessing.