[M4IF Technotes] GMC

[Kris Huber]

Sorin,
Thanks for your clarifications.  I think the rotation aspect of GMC
prediction is important.  In my opinion, commercial encoders are more likely
to restrict the searched parameter space to a region small enough to
accomodate a 25% per frame zoom than they are to restrict the parameters to
have zero tolerance for rotation and skew.  A zero rotation and skew
constraint doesn't occur so naturally in an encoder, since the estimation
involved may be about the same complexity without that constraint--I suppose
it depends on encoder estimation method.  Note that with zoom-out restricted
to 25%, with rotation it seems to me that you can still predict that a block
of reference memory no bigger than NxN where N=ceil(1+(16*1.25)*sqrt(2))=29
will be needed to compute the predicted macroblock, and you could at least
get that block into some faster memory ahead of time.  I'm not sure how
you'd limit skew of the 3 warping points model.
As for mirrored prediction, I can imagine it being useful for interesting
special effects.  And I suppose how frequently mirrored prediction is used
may depend upon how frequently users of encoder tools decide to use the
special effects built from it.  I wouldn't argue as much about the value of
mirrored prediction as I would about the value of some rotation and skew.
For example, rotation of 90 degrees from frame to frame seems more than
necessary for natural video and useful only for special effects like
mirrored prediction.
Of course, for encoder implementers it seems most convenient to have no
restrictions in order to allow for maximum innovation and product
distinction.
Best Regards,
Kris
-----Original Message-----
From: "Sorin C. Cismas" <sorin   mobilygen.com>
Cc: <technotes   lists.m4if.org>
Subject: RE: [M4IF Technotes] GMC
Date: Wed, 15 May 2024 00:01:25 -0700
Kris and all,
Indeed, limitations on the negative side are needed too.
The restriction should apply to the absolute value of du[i] and dv[i].
My interest is mainly in the ASP.  I'm implementing the ASP   L5 with
some hardware support and I'm looking at complexity issues and memory
bandwidth utilization.
If it will be up to me to decide what GMC to support in ASP, it will
be only with one warping point.  If that's not acceptable, the next
complexity level will be to support only rectangle based prediction
with some decent limitations on du[i] and dv[i].  This will support
independent stretch/shrink on both directions (zoom is a particular
case) but without any rotation.  In this case, the restrictions will be:
Abs(du[1]) < W//2;      dv[1]  = 0   ;  // +-25% limitation
    du[2]  = 0   ;  Abs(dv[2]) < H//2;  // +-25% limitation
The current GMC is beautiful in its generality, but lacks some
practicality.  GMC can support 10x zoom and mirrored prediction
(for example, 2 warping points with du[1] = -4*W and dv[1] = 0).
GMC is usefull to transmit only once a motion vector that applies or
can be extrapolated by simple scaling to several macroblocks.  Beyond
that, I'm not sure about its real life usefullness.  Without apriori
knowledge, it will be quite difficult for the encoder to figure out how
to use this feature.  And even if it does, I'm not sure how many bits
are saved.  But on the downside, an MPEG4 decoder which claims to be
ASP compliant, has to support these features even if all commercial
encoders will never use affine based prediction in all its generality.
Regards,
Sorin Cismas
-----Original Message-----
From: technotes-admin   lists.m4if.org
[mailto:technotes-admin   lists.m4if.org]On Behalf Of Kris Huber
Sent: Tuesday, May 14, 2024 6:28 PM
To: 'sorin   mobilygen.com'
Cc: 'technotes   lists.m4if.org'
Subject: RE: [M4IF Technotes] GMC
Soren,
This idea sounds interesting for limiting complexity of GMC.  Are any
limitations on the negative side of du[i] and dv[i] needed?  Once fully
fleshed out I think it possibly could be added as corrigendum.  It is one
more thing that encoder tools would have to worry about, though.  But the
constraints might be loose enough that nobody would be concerned.
Regards,
Kris
-----Original Message-----
From: "Sorin C. Cismas" <sorin   mobilygen.com>
To: <technotes   lists.m4if.org>
Subject: RE: [M4IF Technotes] GMC
Date: Sun, 12 May 2024 00:00:04 -0700
I don't think there are any restrictions on the motion vectors.  I have
the same concern about the need to fetch 64 non-contiguous 2x2 pels
to predict a macroblock.  To reduce complexity and bandwidth requirements,
it is highly desirable to put some limits on du[i] and dv[i] for i>0.
2 and 3 warping points are usefull for zoom-in and zoom-out, however, it is
not realistic to assume, for example, a 10x zoom-out on an S(GMC)
prediction.
A 50% or even 25% zoom-out restriction will be more than sufficient.
For 50%, this will translate to du[i]<W    and dv[i]<H    for i>0.
For 25%, this will translate to du[i]<W//2 and dv[i]<H//2 for i>0.
The 50% restriction will limit the macroblock luma prediction to 25x25.
The 25% restriction will limit the macroblock luma prediction to 21x21.
Can these or similar restrictions be considered for a future corrigendum?
Thanks,
Sorin Cismas
> -----Original Message-----
> From: technotes-admin   lists.m4if.org
> [mailto:technotes-admin   lists.m4if.org]On Behalf Of Kasturi Rangam
> Sent: Friday, May 10, 2024 2:46 PM
> To: technotes   lists.m4if.org
> Subject: [M4IF Technotes] GMC
> 
> 
> I am trying to under GMC in MPEG4.
> Looks like you can have 0, 1, 2 or 3 sprite_warping_points.
> 
> To decode a 0 and 1 sprite_warping_points macroblock, you can
> fetch the whole macroblock data from the reference to do pixel
> prediction.
> 
> However, for 2 and 3 warping_points, the motion vector for each
> pixel can be anywhere in the reference frame. Thus we might need
> to perform 64 data fetches to predict one macroblock.
> 
> Is there a range that can be calculated for each macroblock, so that
> we can fetch data only once?
> 
> Thanks,
> 
> Kasturi
> _______________________________________________
> Technotes mailing list
> Technotes   lists.m4if.org
> http://lists.m4if.org/mailman/listinfo/technotes