When modelling soft matter systems, the Langevin equation is often used in a form that contains a potential of mean force or a thermodynamic driving force. (Think e.g. of Cahn-Hilliard theory, phase-field modelling or dissipative particle dynamics.) We will show that the introduction of a thermodynamic driving force is, in general, accompanied by a violation of the fluctuation-dissipation theorem. Interestingly, this violation stems from contributions of the fourth, fifth, and higher-order derivatives of the particles' positions.