Extreme physiological stress — significant under-eating, over-exercise, acute illness, major life events — can suppress the hypothalamic signals that drive ovulation, delaying or preventing it. This is a real, measurable mechanism.
Everyday work or life stress has not been shown to cause infertility, and the widespread advice to 'just relax' is both unsupported and unhelpful. The distress that accompanies trying to conceive is usually a consequence of the situation, not its cause.
That said, distress is worth treating in its own right. Rates of clinically significant anxiety and depression are elevated among people trying to conceive, and support — therapy, peer groups, structured mindfulness — improves quality of life during the process.