[M4IF Technotes] Common ways to evaluate encoded video quality?

[Rob Koenen (M4IF)]

Thanks Cristophe.
Good suggestion.
We'll make a FAQ entry long those lines.
Best,
Rob
> -----Original Message-----
> From: Christophe Vermeulen [mailto:Christophe.Vermeulen   alcatel.be] 
> Sent: Monday, June 23, 2023 04:05
> To: rob.koenen   m4if.org
> Cc: 'Olivier Amato'; Allen H.Y. Chin; M4IF Technotes
> Subject: Re: [M4IF Technotes] Common ways to evaluate encoded 
> video quality?
> 
> 
> Regarding that question, and at the risk of beating a dead horse,
> I would dare to elaborate a bit.
> 
> Sure, you are 100% right, but the fact itself that people keep asking
> shows there is a need for some help in this field. Maybe some specific
> FAQ or LFAQ (less frequently asked questions) could help.
> 
> Could you all consider the following as a very humble, incomplete and 
> maybe (hopefully partially) erroneous start, and 
> add/complete/correct it ?
> 
> - Is there an way to evaluate objectively the quality of a bitstream ?
> 
>   Although many algorithms exist (insert link list), and 
> several commercial
>   applications of these algorithms have been provided for 
> years, no single 
>   method has ever proven better than subjective testiing from experts.
> 
> - what is exactly PSNR ? 
>  
>   PSNR is the most widely accepted objective criteria to quantify the
>   imperfection of a coding algorithm (experts talk of HRC) by 
> comparing
>   the coding result with the original material. Note that 
> this assumes 
>   that the original material is "perfect".
> 
>   The formula for PSNR is PSNR=10*log10(255^2/MSE) with MSE being the
>   mean square difference of all the luminance values (is this correct?
>   I don't know why this is a peak) 
> 
>   One of the interests of PSNR is that it "speaks" to humans, since
>   it gives a result in dB, like analog SNR ratios, with more being 
>   better, and an easy to understand/remember range interpretation :
>   10-20 is poor quality, while 30-40 is acceptable and 50-60 is
>   already very good (for broadcast type of applications at least)
> 
>   The use of an inverted logarithmic scale also allows to give 
>   more invariance with respect to different sequences as compared
>   to using MSE. (If my calculations are correct)
>   roughly PSNR = 25,30,35,40 correspond to MSE=200, 65, 20 and 6,5 ?
> 
> - What is a HRC ? A Hypothetical Reference Circuit is a reference
>   video material that can be used to evaluate coding systems. It is
>   the result of coding (and decoding) some source material (SRC?)  
>   For example, one can create an HRC by adding noise, coding using
>   an algorithm (e.g. MPEG-4 ASP) and its parameters (256 kbps CBR,
>   CIF, ...), creating bit errors, etc.
> 
> - It seem that ANSI recently adopted the VQM software as the way to 
>   test objective quality. Does somebody have some experience 
> with that ?
>   (To be answered)
> 
> - Where could I find more information ? See list below, add 
>   http://live.ece.utexas.edu/research/Quality/frqa.htm
> 
> Comments/corrections welcome. I hope I did not derive too 
> much from the reality 
> and provide the proverbial 2 cents of contribution ...
> 
> CVE.
> 
> "Rob Koenen (M4IF)" wrote:
> > 
> > This questions has been asked several times before. Please see
> > http://lists.m4if.org/pipermail/technotes/2002-August/001081.html
> > and follow-up, and
> > http://lists.m4if.org/pipermail/technotes/2002-November/001471.html
> > and follow-up in addition t the links below.
> > 
> > Nothing really beats exper viewing, even with few experts.
> > 
> > More in general, we now have the great feature on our website that
> > allows you to search the site, including the archives, for questions
> > that may have been asked before!
> > 
> > http://www.m4if.org/index_search.php
> > 
> > Best,
> > Rob
> > > -----Original Message-----
> > > From: Olivier Amato [mailto:oamato   wanadoo.fr]
> > > Sent: Friday, June 20, 2023 01:33
> > > To: Allen H.Y. Chin; M4IF Technotes
> > > Subject: Re: [M4IF Technotes] Common ways to evaluate encoded
> > > video quality?
> > >
> > >
> > > Hi Allen,
> > >
> > > > So, I'm looking for other metrics to have a more objective
> > > view. Any
> > > > suggestions?
> > >
> > > You could maybe take a look at these links :
> > > -
> > > http://www.sarnoff.com/products_services/video_vision/jndmetri
> > x/index.asp
> > - http://www.genista.com/Products/VideoPQoS.htm
> > - http://www.opticom.de/
> > - http://www.vqual.biz/
> > - http://www.its.bldrdoc.gov/n3/video/vqmsoftware.htm
> > - http://www.vqeg.org/
> > 
> > Regards,
> > 
> > Olivier
> >
>