We went in March this year, twice! Warm during the afternoon but pretty chilly in the morning and evening. Depends what you want to do, it's a good time of year for getting out and about exploring, a bit too cold for sitting by a pool but probably not much chance of that with a 3 year old anyway!
On our second visit the motorbikes has been banned from the medina and the difference it made was incredible, so much more relaxing to walk through, not sure if it was a short term or ongoing thing though.
We loved it and can't wait to go again, I'm sure you'd all have a great time, and that time of year would certainly be better than July which was when we went to Fez and hardly left our riad, it was just too hot