If your goal is targeting a specific muscle and you stop feeling that muscle due to various compensation mechanisms, it stops being effective. I stop the set at that point. This is mostly applicable to more targeted movements like single-joint or machine exercises.
With more complex movements, I see technical breakdown as the end of the set. Keep in mind, I’m talking about significant breakdown like turning a squat into a good-morning squat or lifting the butt off the bench.
In both cases, this typically falls at a point where you have 1-3 reps in reserve were you to accept continuing the set with those compensations.
Very rarely do I recommend exercise failure (true failure) except, for example, on single-joint/machine exercises if it coincides with tension failure, which is the case with advanced lifters with a great mind-muscle connection.