How do I help my husband and my three-month-old get along?

I stay at home with my 3 m.o. baby. He's great with me: he very rarely cries, and we spend most of our days cooing at each other, feeding, and napping together. Some evenings, though, I like to have time to do personal things like shower, go to the gym, or catch up with my friends, and I leave him with his dad. My husband says that he screams and screams when they're alone together, even when I've left him with an ample supply of milk. I come home to a depressed husband (he thinks the baby hates him), and a stressed-out, sniffling baby. Other nights when I'm home all evening, everything is copacetic. Isn't it too early for seperation anxiety? Has any one else had this problem? And how can I help my husband not feel like it's personal? He wants so badly to be a fantastic dad, I dont want him to be discouraged.