[M4IF Technotes] Re: Codec testing methodology

[Rob Koenen]

> I asked whether there were any norms for testing the quality of the
> intermediary stages.

No, there are not in MPEG -- and actually note anywhere else either.
There is a great deal of research on automated quality evaluation. 
See http://www.vqeg.org/ , and some manufacturers may tell you
there is an answer, but measures are far from perfect. 
THe only way to really know the quality is to do double-blind 
subjective testing with quite a fews subjects. 
You will find codec comparison test results that are less than perfect, 
and very questionable from a scientific/statistical point of view. 
Always carefully see how tests were done!
It is very important to point out that MPEG doesn't standardize the
encoding process, on purpose. This allows competing products in the
market while maintaining interoperability. It also means that the 
"quality of MPEG-4" will improve over time -- and varies from 
one encoder manufacturer to the other. 
You'll have seen the same with MPEG-2.
Rob
> -----Original Message-----
> From: the_ether [mailto:the_ether   btinternet.com]
> Sent: Friday, January 18, 2024 17:52
> To: technotes   lists.m4if.org
> Subject: Re: [M4IF Technotes] Re: Codec testing methodology
> 
> 
> Thanks but this refers to conformance testing, no?
> 
> My post asked about _quality_ testing and of each stage in 
> the compression
> cycle. Three codecs might conform to the standard but the 
> quality of a) the
> final images and b) the performance of each individual stage (ME,
> quantisation etc.) could vary enormously.
> 
> I asked whether there were any norms for testing the quality of the
> intermediary stages.
> 
> 
> ----- Original Message -----
> From: "Rob Koenen" <rkoenen   intertrust.com>
> To: "the_ether" <the_ether   btinternet.com>; <technotes   lists.m4if.org>
> Sent: 19 January 2024 01:41
> Subject: RE: [M4IF Technotes] Re: Codec testing methodology
> 
> 
> > The same applies to Video. Using info obtained from the chair
> > of the Video Group:
> >
> >   Yes, there are conformance bitstreams for video too.
> >   ftp://ftp.tnt.uni-hannover.de/pub/MPEG/video/conformance
> >   has the MPEG-4 confrmance bitstreams.
> >
> >   The spirit for pass/fail criteria is essentially the same
> >   as for audio.
> >
> >   I think the only distinction is that IDCT allows some
> >   rounding error tolerance.  But that tolerance is well specified.
> >
> > Rob
> >
> > > -----Original Message-----
> > > From: Ralph Sperschneider [mailto:sps   iis.fhg.de]
> > > Sent: Thursday, January 17, 2024 11:26
> > > To: the_ether
> > > Cc: technotes   lists.m4if.org
> > > Subject: [M4IF Technotes] Re: Codec testing methodology
> > >
> > >
> > > Dear Garham, all,
> > >
> > > I cannot tell you the details for video, but for audio there
> > > are a bunch of sequences of compressed audio coming along
> > > with reference waveforms. For most of these sequences there
> > > is a test procedure described that compares the output of
> > > your decoder with the reference waveform. You may call your
> > > decoder conform with MPEG-4 only, if the decoder passes the
> > > appropriate tests. Details can be found in the conformance
> > > part of the standard (ISO/IEC 14496-4:2000, Amd1 seems not
> > > yet been published).
> > >
> > > Best regards,
> > >
> > > Ralph
> > >
> > >
> > > the_ether wrote:
> > > >
> > > > Hi
> > > >
> > > > Is there any standard testing regime for MPEG-4 codec? I am
> > > aware of the
> > > > video files used for conformance testing but I was
> > > wondering whether there
> > > > were any other guidelines such as for the intermediary
> > > steps such as ME or
> > > > comparisons of signal to noise ratios for a given input
> > > image and a given
> > > > target bit rate?
> > > >
> > > > As far as I know there are only conformance tests and
> > > subjective tests of
> > > > the quality of the end, decompressed images.
> > > >
> > > > Regards
> > > >
> > > > garham
> > > >
> > > > _______________________________________________
> > > > Technotes mailing list
> > > > Technotes   lists.m4if.org
> > > > http://lists.m4if.org/mailman/listinfo/technotes
> > >
> > > --
> > > Dipl.-Ing. Ralph Sperschneider  | Phone: +49 9131 776 344
> > > FhG IIS-A                       | Fax:   +49 9131 776 398
> > > Am Weichselgarten 3             | mailto:sps   iis.fhg.de
> > > D 91058 Erlangen                | http://www.iis.fhg.de/amm/
> > >
> > _______________________________________________
> > Technotes mailing list
> > Technotes   lists.m4if.org
> > http://lists.m4if.org/mailman/listinfo/technotes
> 
> 
> _______________________________________________
> Technotes mailing list
> Technotes   lists.m4if.org
> http://lists.m4if.org/mailman/listinfo/technotes
>