FW: [Mp4-tech] Regarding padding in MPEG-4 part 2

Fri Nov 24 15:48:54 ESTEDT 2006

Another bounced mail from anon-subsribed address (different than
yesterday's). 
Yi Shin - please subscribe your sending address. See
http://www.m4if.org/public/publiclistreg.php
Thanks.
-----Original Message-----
From: Yi-Shin Tung [mailto:tung setabox.com.tw] 
Sent: Friday, 24 November 2023 03:19
To: Herbert Thoma; Gary Sullivan
Cc: "Rickard Sjöberg (KI/EAB)"; mp4-tech lists.mpegif.org; Tung yi-Shin;
ohm ient.rwth-aachen.de
Subject: Re: [Mp4-tech] Regarding padding in MPEG-4 part 2
Hi Rickard, Gary, Herbert and Jens,
I think both versions of RefSW had been changed to consistent with what 
specified in the 3rd version.
I do not remember there exists some bitstreams available for this test in 
the current conformance set. If you can provide some, I could put them in 
the extended list in the future.
Regards,
Yi-Shin
----- Original Message ----- 
From: "Herbert Thoma" <tma iis.fhg.de>
To: "Gary Sullivan" <garysull windows.microsoft.com>
Cc: ""Rickard Sjöberg (KI/EAB)"" <rickard.sjoberg ericsson.com>; 
<mp4-tech lists.mpegif.org>; "Tung yi-Shin" <tung cmlab.csie.ntu.edu.tw>; 
"Yi-Shin Tung" <tung setabox.com.tw>; <ohm ient.rwth-aachen.de>
Sent: Thursday, November 23, 2023 7:59 PM
Subject: Re: [Mp4-tech] Regarding padding in MPEG-4 part 2
> Gary, Yi-Shin, Jens,
>
> Please look at the 3rd (2004) edition:
>
> The sentence
> "Note that for rectangular VOP, a reference VOP is defined by
> video_object_layer_width and video_object_layer_height."
> in the 2nd edition was changed to
> "Note that for a rectangular VOP, a reference VOP is defined by
> video_object_layer_width and video_object_layer_height, extended to
> a multiple of 16"
> in the 3rd edition.
>
> I guess this schould make you happy, Gary :-)
>
> Regards,
>  Herbert.
>
> Gary Sullivan wrote:
>> Copying Yi-Shin and Jens, who perhaps remember this topic,
>>
>> Actually the action I discussed may have happened longer ago than that. 
>> I looked in N6362
>> and although it touched on some relevant sections (I think including 
>> 7.6.3) I did not quite
>> find an answer to the question there.  I also looked in N3664 and didn't 
>> find it there either.
>> I then looked in the text of the 2nd (2001) edition and although I got a 
>> bit confused by what it
>> says in various plances (although I haven't really studied the topic 
>> fully yet), I think I found it.
>>
>> What I think I see supports what I said on this thread.  At the end of 
>> 7.6.4 in the 2nd edition
>> it says the following:
>>
>>  xref = MIN ( MAX (xcurr+dx, vhmcsr), xdim+vhmcsr-1 ) yref = MIN ( MAX 
>> (ycurr+dy, vvmcsr), ydim+vvmcsr-1 )
>>
>> and
>>
>> "(ydim, xdim) are the dimensions of the bounding rectangle of the 
>> reference VOP"
>>
>> and
>>
>> "Note that for rectangular VOP, a reference VOP is defined by 
>> video_object_layer_width and video_object_layer_height."
>>
>>
>> Now we know that video_object_layer_width and video_object_layer_height 
>> might not be multiples of 16.
>>
>> Based on those equations and quoted sentences, it sounds to me like 
>> padding is applied for any location beyond the rectangle having (width, 
>> height) = (video_object_layer_width, video_object_layer_height).
>>
>> This is not something that I am happy about, but I think it is what is 
>> currently written and I think there was some historical corrigendum 
>> action that changed it to say that.
>>
>> There has been a long history of confusion over this issue.  I am not 
>> entirely sure that I have read all relevant parts of the spec, but I 
>> don't see how the above quoted statements can be interpreted any other 
>> way.
>>
>> Best Regards,
>>
>> Gary Sullivan
>>
>>
>> +> -----Original Message-----
>> +> From: Gary Sullivan +> Sent: Thursday, November 23, 2023 2:20 AM
>> +> To: 'Herbert Thoma'
>> +> Cc: "Rickard Sjöberg (KI/EAB)"; mp4-tech lists.mpegif.org
>> +> Subject: RE: [Mp4-tech] Regarding padding in MPEG-4 part 2
>> +> +> Have you looked at N6362?
>> +> +> Best Regards,
>> +> +> -Gary Sullivan
>> +> +> +> -----Original Message-----
>> +> +> From: Herbert Thoma [mailto:herbert.thoma iis.fraunhofer.de] +> +> 
>> Sent: Thursday, November 23, 2023 2:11 AM
>> +> +> To: Gary Sullivan
>> +> +> Cc: "Rickard Sjöberg (KI/EAB)"; mp4-tech lists.mpegif.org
>> +> +> Subject: Re: [Mp4-tech] Regarding padding in MPEG-4 part 2
>> +> +> +> +> Gary, Rickard,
>> +> +> +> +> I am pretty sure that the padding from 128x176 is the +> +> 
>> correct interpretation
>> +> +> (meaning that the pixels outside of 120x170 but inside of +> +> 
>> 128x176 shall be left
>> +> +> as they were decoded).
>> +> +> +> +> I remember the discussion in MPEG very well, because I +> +> 
>> originally implemented
>> +> +> it the other way in my encoder and decoder and changed that +> +> 
>> after the corrigendum.
>> +> +> +> +> I don't konw if there are any comformance bitstreams +> +> 
>> available, but I attached
>> +> +> a few frames of forman cropped to 170x120 and encoded with +> +> my

>> encoder. (I can
>> +> +> not guarantee that there are actually motion vectors that +> +> 
>> test the problem
>> +> +> in there, though.)
>> +> +> +> +> Kind regards,
>> +> +>   Herbert.
>> +> +> +> +> Gary Sullivan wrote:
>> +> +> > Rickard et al,
>> +> +> > +> +> > For a long time, I was pretty sure that I knew what the 
>> +> +> answer was, and my interpretation agreed
>> +> +> > with yours.  Certainly that is the way the similar feature +> +> 
>> works in H.263 (although that fact is
>> +> +> > not directly relevant to the question at hand, since +> +> 
>> "motion vectors over picture boundaries" is
>> +> +> > a non-Baseline feature of H.263 Annex D, while MPEG-4 part +> +> 
>> 2 only tries to be compatible with the
>> +> +> > Baseline).  I vaguely recall that at one time one +> +> 
>> implementation of the reference software was
>> +> +> > doing it one way and the other was doing it the other way, +> +> 
>> and I believe MPEG eventually approved
>> +> +> > a corrigendum to Part 2 and a bug fix to the software to +> +> 
>> clarify it.  Unfortunately, I believe the
>> +> +> > clarification was according to the other interpretation.
>> +> +> > +> +> > There was a corrigendum finalized in 2004, which I think 
>> +> +> corresponded to MPEG output document
>> +> +> > N6362.  I believe this subject was addressed in that +> 
>> corrigendum.
>> +> +> > +> +> > If I was making an encoder, I might design it to avoid +>

>> +> motion vectors reaching beyond the bottom
>> +> +> > and right edges of the reference pictures to be sure that +> +> I

>> would work with decoders that used
>> +> +> > either interpretation.  +> +> > +> +> > I guess another decent 
>> encoder approach would be to use +> +> padding in the source for those 
>> areas before
>> +> +> > encoding too, so that the only difference between the two +> +> 
>> interpretations would be the quantization
>> +> +> > error.  With that approach, a little drift might not +> +> 
>> produce very bad artifacts.
>> +> +> > +> +> > Best Regards,
>> +> +> > +> +> > Gary Sullivan
>> +> +> > +> +> > +> -----Original Message-----
>> +> +> > +> From: mp4-tech-bounces lists.mpegif.org +> +> > +> 
>> [mailto:mp4-tech-bounces lists.mpegif.org] On Behalf Of +> +> > +> 
>> Rickard Sjöberg (KI/EAB)
>> +> +> > +> Sent: Wednesday, November 22, 2023 5:36 AM
>> +> +> > +> To: mp4-tech lists.mpegif.org
>> +> +> > +> Subject: [Mp4-tech] Regarding padding in MPEG-4 part 2
>> +> +> > +> +> +> > +> +> +> > +> Dear experts,
>> +> +> > +> +> +> > +> assume that a simple profile video stream with 
>> height and +> +> > +> width of 120 and 170 pixels respectively shall be 
>> decoded. +> +> > +> The bounding rectangle of the reference VOP is 
>> 128x176. Now +> +> > +> my question is whether you should pad outside of 
>> 120x170 or +> +> > +> 128x176 when referencing pixels for motion 
>> compensation?
>> +> +> > +> +> +> > +> The relevant part of the standard is section 7.6.4 
>> i believe +> +> > +> (I looking at the 3rd edition, that's ISO/IEC +> +> 
>> 14496-2:2004, N5515):
>> +> +> > +> +> +> > +> The coordinates of a reference sample in the 
>> reference VOP, +> +> > +> (yref, xref) is determined as follows :
>> +> +> > +> xref = MIN ( MAX (xcurr+dx, vhmcsr), xdim+vhmcsr-1 )
>> +> +> > +> yref = MIN ( MAX (ycurr+dy, vvmcsr), ydim+vvmcsr-1)
>> +> +> > +> +> +> > +> My interpretation of this is that padding should be

>> done +> +> > +> outside of 128x176 (this means that the pixels outside of

>> +> +> > +> 120x170 but inside of 128x176 shall be left as they were +> +>

>>  > +> decoded), is this correct?
>> +> +> > +> +> +> > +> Is there any conformance bitstream that tests this 
>> behaviour?
>> +> +> > +> +> +> > +> /
>> +> +> > +> Rickard Sjoberg
>> +> +> > +> Ericsson
>> +> +> > +> +> +> > +> _______________________________________________
>> +> +> > +> NOTE: Please use clear subject lines for your posts. Include 
>> +> +> > +> [audio, [video], [systems], [general] or another +> +> > +> 
>> apppropriate identifier to indicate the type of +> +> question you have.
>> +> +> > +> +> +> > +> Note: Conduct on the mailing list is subject to the

>> +> +> > +> Antitrust guidelines found at +> +> > +> 
>> http://www.mpegif.org/public/documents/vault/mp-out-30042-Ant
>> +> +> > +> itrust.php
>> +> +> > +> +> +> > +> +> > 
>> _______________________________________________
>> +> +> > NOTE: Please use clear subject lines for your posts. +> +> 
>> Include [audio, [video], [systems], [general] or another +> +> 
>> apppropriate identifier to indicate the type of question you have.
>> +> +> > +> +> > Note: Conduct on the mailing list is subject to the +> +>

>> Antitrust guidelines found at +> +> 
>> http://www.mpegif.org/public/documents/vault/mp-out-30042-Ant
>> +> +> itrust.php
>> +> +> > +> +> +> +> -- 
>> +> +> Herbert Thoma
>> +> +> Head of Video Group
>> +> +> Multimedia Realtime Systems Department
>> +> +> Fraunhofer IIS
>> +> +> Am Wolfsmantel 33, 91058 Erlangen, Germany
>> +> +> Phone: +49-9131-776-323
>> +> +> Fax:   +49-9131-776-399
>> +> +> email: tma iis.fhg.de
>> +> +> www: http://www.iis.fhg.de/
>> +> +>
>
> -- 
> Herbert Thoma
> Head of Video Group
> Multimedia Realtime Systems Department
> Fraunhofer IIS
> Am Wolfsmantel 33, 91058 Erlangen, Germany
> Phone: +49-9131-776-323
> Fax:   +49-9131-776-399
> email: tma iis.fhg.de
> www: http://www.iis.fhg.de/
>