A more general version of this statement was shown by Kimura in Acta Arith. (2003). His corollary gives $\gg \sqrt{X}/\log X$ such fields ${\Bbb Q}(\sqrt{-d})$ with $d\le X$, and also allows you to add further
splitting conditions. There is an extensive literature on divisibility and indivisibility of class numbers and Kimura's paper has many other relevant references.
Update There is also an erratum to Kimura's paper -- it seems that he needs the existence of one such field to get a lower bound for the number of such fields; for large $p$ this is guaranteed by work of Horie. In the meantime a paper of Wiles has addressed the general version of this question, and there is a quantification of Wiles's result due to Beckwith.