The test protocol must also be normalized, such as SAMVIQ or ACR protocols. The vision of observers should also be checked in advance so as not to employ visually impaired observers (nearsighted and uncorrected observers, color blind). It is also necessary to use standardized statistical methods to detect and remove observers who have given inconsistent judgments. Finally, the confidence interval for each mean opinion score (MOS) should be calculated to know the accuracy of the test results and to decide when the tests have reached sufficient accuracy to be stopped.
Not only is the subjective assessment of quality difficult and time consuming. So in addition, setting up a normalized room, performing subjective tests and calculating the results usually takes 1 to 2 weeks for a set of 30 videos. Therefore, companies often neglect the correct testing conditions - this leads to consumer disappointment and loss of loyalty to the company.