Getting outside your usual writing zone

We all have different genres that we like to write. Such as non-fiction, historical fiction, romance, and so on and so forth. But what if you wanted to take a break from writing your usual genre and try something new? For example, your usual genre is mysteries and you want to try your hand at writing picture books. How hard do you think it would be to make the transfer? I know my two genres are mystery and historical fiction. I'm currently branching out and trying something along the lines of a romantic comedy. It isn't easy, because I'm not weaving clues or historical accuracy into the storyline. But it's a challenge that can help broaden what I write. What was it like when you stepped out of your usual writing zone to try something new? Was it frustrating to the point of wanting to make you give up? Or did it push you to see if you could do it?