[M4IF Technotes] MPEG 4 Timing Questions

Mariano Padilla mpadilla sdi-media-usa.com
Tue Oct 23 18:54:29 EDT 2001


Gary,
    Thank you so much for your reply.  The 29.97 FPS, or commonly know as
"Drop Frame" can be compensated.  We currently have some calculations we
perform to compensate for the drop frame calculation.  What I would like, if
possible for you to elaborate on your comment: 
        "I believe the time_code values in MPEG-4
are meant to be taken literally as precise multiples of exact seconds,
whereas
the values in 60461 are not.  This aspect should probably be clarified, and
may
cause difficulties when synchronizing with devices that use conventional
time code if not interpreted correctly. "
    Even though we can compensate for the "drop frame" phenomenon on Mpeg-1,
AVI, Mpeg-2 and Mpeg-4, it seems to me that MPEG-4 time_code values are
meant to be taken as precise as multiples of exact seconds, therefore
leaving the frames to be randomly updated by the decoder to "compensate" to
match the SS parameter structure of the MPEG-4 time_code.  Therefore, we
cannot synchronize anything precisely to the frame within a second of the
MPEG-4 VOP since it only support HH:MM:SS time_code structure.  Can you
confirm this, or elaborate if I am at error?  Thank you again.
Mariano J. Padilla
SDI Media USA
Director of IT/R&D
Tel: 323-602-5406
Fax: 323-602-5450
mpadilla   sdi-media-usa.com
www.sdi-media-usa.com
-----Original Message-----
From: Gary Sullivan [mailto:garysull   microsoft.com]
Sent: Tue, October 23, 2023 5:17 PM
To: Mariano Padilla; technotes   lists.m4if.org
Subject: RE: [M4IF Technotes] MPEG 4 Timing Questions
To Mariano Padilla et al,
Using the fixed_vop_rate flag and the fixed_vop_time_increment parameter,
it is possible in MPEG-4 video to specify a constant and fixed frame rate
(as, for example, is typical in MPEG-2 use).  It is also possible for the
frame
rate to not be constant, and in fact it is possible for the timing between
frames
to vary in a highly flexible fashion from frame to frame in MPEG-4.  The
encoder
governs whether to send frames at fixed intervals or not.
I frankly don't understand the jerkiness comment.  Unless the encoder
wants the frame rate to vary, there is nothing inherently different in
handling
MPEG-4 video than in handling MPEG-2 or MPEG-1 video.
Displaying the reference frames in the manner you described and repeating
frame decodings sounds simply like a poor system design.  The proper thing
to do is to store the necessary anchor frames in the decoder and not repeat
their decoding during reverse playback.  Even if the decoding of them is
repeated,
their display need not be repeated.  This is really the same whether using
MPEG-4
or MPEG-2 or MPEG-1.
The Group_of_VideoObjectPlane structure in MPEG-4 includes a
HH:MM:SS time_code structure (although I don't think the encoder is required
to send
this structure if it doesn't want to).  The time of each individual VOP is
governed by the
use of that time_code in conjunction with the vop_time_increment_resolution,
modulo_time_base, and vop_time_increment parameters (along with the
previously-mentioned fixed_vop_rate flag and fixed_vop_time_increment
parameter).
You should be aware that conventional HH:MM:SS:FF timecode
drifts relative to true time in regard to the values of its HH:MM:SS
parameters
whenever the frame rate is not an integer number of frames per second (which
it is not in 29.97 Hz video).  This may be the cause of your difficulties in
synchronizing.
MPEG-4 mentions IEC 60461 in describing the time_code parameters.
However, I think perhaps this was not written properly.  I believe each
increment of the SS parameter in the 60461 timecode does not necessarily
represent a time lapse of exactly one second.  For example, in 29.97 Hz
video, each increment of the seconds parameter in 60461 normally represents
a time interval of 30*1001/30000 seconds, which is 30030/30000 seconds, not
exactly one second.  And I believe the 60461 timecode is not as general as
the MPEG-4 visual timing design.  I believe the time_code values in MPEG-4
are meant to be taken literally as precise multiples of exact seconds,
whereas
the values in 60461 are not.  This aspect should probably be clarified, and
may
cause difficulties when synchronizing with devices that use conventional
timecode if not interpreted correctly.  The drift between conventional
timecode
and true time can exceed 86 seconds per 24 hour period.
Best Regards,
Gary Sullivan
-----Original Message-----
From: Mariano Padilla [mailto:mpadilla   sdi-media-usa.com] 
Sent: Tuesday, October 23, 2023 3:48 PM
To: 'technotes   lists.m4if.org'
Subject: [M4IF Technotes] MPEG 4 Timing Questions
Ladies, Gentlemen, 
We are currently working along with SMPTE, Society of Motion Pictures and
Television Engineers, on the D-Cinema Subtitle Specs where MPEG-4 is being
considered for the video format. Our concerns listed below are crucial,
however now I have a couple more specific questions that I hope some of you
may know the answer to. 
1) Is the timing model of MPEG-4 specific to the normal timing model 
of other video formats? For example, if in an AVI we have 30 FPS 
we expect the timing model to be 30 frames per second and so on. 
I can accurately synchronize on frame 10 of the 25th second of the 
23rd minute of the movie. 
2) If I have an Mpeg-4 video file and a separate PNG or TIFF format 
graphic file with yet another separate text navigation file that gives 
me time cues down to the frame, i.e., '00:23:25.10', will I be 
able to synchronize the pictures to the video with frame accuracy? 
This is crucial for the display of subtitles at the right time. 
3) If the MPEG-4 time model is not accurate to the frame, is there 
any alternative to synchronize to the frame, or will I have to 
synchronize to the nearest second? 
Lastly, the jerky playback of MPEG-4 files is for example, I want to play a
movie at 1/4 the speed with the sound mimicking the sound of a tape at 1/4
the speed, where the sound actually sounds a couple of octaves lower, but it
is understandable. This method is commonly known as "scrubbing audio", which
works good on forward playback of the MPEG4 movie. But during the process of
reversing playback, we have problems of "jerkiness", where the frames need
to be "de-coded" or "re-render" to actually just display the previous frame.
For example, I stop at frame 10, and want to go back to frame 9, if frame 9
is not a reference frame, the video playback visibly has to go back to the
key reference frame and "re-render" all the other frames in between the key
reference frame and frame 9. This is my main concern for synchronization. On
a test we've performed, we've noticed that on the long run, subtitles that
we attempted to synchronize 23 minutes into the movie, seem to be off by a
second or so. 
Thanks you all for your feedback, and I apologize for the length of the
e-mail. 
Mariano J. Padilla
SDI Media USA
Director of IT/R&D
Tel: 323-602-5406
Fax: 323-602-5450
mpadilla   sdi-media-usa.com
www.sdi-media-usa.com 
	-----Original Message-----
From: Mariano Padilla
Sent: Fri, October 19, 2023 9:40 AM
To: 'technotes   lists.m4if.org'
Cc: Scott Rose
Subject: RE: MPEG 4
Again, I thank you all for your comments. Hence, if someone on the list can
answer my below questions. 
Thank you much. If you do have the time to elaborate on the strict
Timestamps. Our concern is that when we provide for an MPEG-4 video time
specific "Subtitle Stream" which is controlled by a navigation file
containing time stamps for cue in and cue out, the temporal difference from
an "outside" graphic, in this case the subtitle superimposed on the Mpeg-4
video, can it create the same effect as the DVD MPEG-2 standard? With DVD
there is a variance of approximately 4 to 10 frames or more in some
instances. In NTSC, being 30 fps, it's a bit difficult to notice, but on PAL
at 25 fps it is a bit more noticeable. We've been experimenting also with
MPEG-4, and since it is not as accurate as MPEG-1 I-Frame, we notice that a)
when going back it does not stop on the exact frame, usually re-renders the
frames and you can visually see the re-rendering of frames. b) slow
playback, with sound, is extremely jerky and looses track where it is, hence
it needs to re-render the frames. Are these behaviors part of the internal
time correction as well as just the "de-compression" of the frames? Would
these behaviors affect time cueing of non-internal MPEG-4 objects? For
example, I have a graphic that is to be superimposed on a video at exactly
00:23:20.10, would the 10th frame be a reference frame or would it have to
"de-compress" and generate from reference frames and how would this affect
the external temporal synch? Your e-mail below says the MPEG-4 timestamps
can be as accurate as I wish, in this case I would need it to be down to the
frame, is this possible? Would I need to create the subtitle stream of
external tiffs as MPEG-4 BIFS in order to achieve the frame accurate sync? 
	Mariano J. Padilla
Director of IT/R&D
SDI Media USA
T 323-602-5406
F 323-602-5450
www.sdi-media-usa.com 
	-----Original Message-----
From: Rob Koenen [ mailto:rkoenen   intertrust.com
<mailto:rkoenen   intertrust.com> ]
Sent: Fri, October 19, 2023 9:34 AM
To: 'Mariano Padilla'; 'Michelle Y Kim'
Cc: Rob Koenen; Scott Rose
Subject: RE: MPEG 4
I second Michelle's proposal to send to the list. 
However, oit looks to me as if your issue is with the 
implementation you are using, not with the standard. 
Jerky playback is not something that is built into MPEG-4. 
Rob 
	-----Original Message-----
From: Mariano Padilla [ mailto:mpadilla   sdi-media-usa.com
<mailto:mpadilla   sdi-media-usa.com> ]
Sent: Friday, October 19, 2023 9:25 AM
To: 'Michelle Y Kim'
Cc: Rob Koenen; Scott Rose
Subject: RE: MPEG 4
Thank you much. If you do have the time to elaborate on the strict
Timestamps. Our concern is that when we provide for an MPEG-4 video time
specific "Subtitle Stream" which is controlled by a navigation file
containing time stamps for cue in and cue out, the temporal difference from
an "outside" graphic, in this case the subtitle superimposed on the Mpeg-4
video, can it create the same effect as the DVD MPEG-2 standard? With DVD
there is a variance of approximately 4 to 10 frames or more in some
instances. In NTSC, being 30 fps, it's a bit difficult to notice, but on PAL
at 25 fps it is a bit more noticeable. We've been experimenting also with
MPEG-4, and since it is not as accurate as MPEG-1 I-Frame, we notice that a)
when going back it does not stop on the exact frame, usually re-renders the
frames and you can visually see the re-rendering of frames. b) slow
playback, with sound, is extremely jerky and looses track where it is, hence
it needs to re-render the frames. Are these behaviors part of the internal
time correction as well as just the "de-compression" of the frames? Would
these behaviors affect time cueing of non-internal MPEG-4 objects? For
example, I have a graphic that is to be superimposed on a video at exactly
00:23:20.10, would the 10th frame be a reference frame or would it have to
"de-compress" and generate from reference frames and how would this affect
the external temporal synch? Your e-mail below says the MPEG-4 timestamps
can be as accurate as I wish, in this case I would need it to be down to the
frame, is this possible? Would I need to create the subtitle stream of
external tiffs as MPEG-4 BIFS in order to achieve the frame accurate sync?
	I thank you all for your comments. 
Mariano J. Padilla
Director of IT/R&D
SDI Media USA
T 323-602-5406
F 323-602-5450
www.sdi-media-usa.com 
-----Original Message-----
From: Michelle Y Kim [< mailto:mykim   us.ibm.com <mailto:mykim   us.ibm.com> >]
Sent: Fri, October 19, 2023 9:21 AM
To: 'Mariano Padilla'
Cc: Rob Koenen; Scott Rose
Subject: RE: MPEG 4 
	Mariano, 
Since I don't know what kind of problems you're trying to solve, I can only
offer you the following very general response, and will have to echo Rob
and ask you to bring this to the reflector. 
If you want to achieve frame-level synchronization of subtitles and video,
and if the video delivery is reliable, then you can achieve synchronization
without FlexTime. You can simply align video timestamps to those of BIFS
that contain subtitles. Note that presently MPEG-4 does not support text
streams. Text in MPEG-4 is delivered in BIFS as timed insertions and
deletions, where the timestamps can be as accurate as you wish. The level
of synchornization you can from MPEG-4 is at a finer granularity than SMIL
if you're coming from the SMIL experience. 
But if the video delivery is non-deterministic, then FlexTime is a technic
you can use, which I can help you with, as I understand the nature of your
question better. 
Regards,
Michelle 
Michelle Y. Kim, Ph. D.
Manager, Composite Media Technologies
IBM T. J. Watson Research Center
30 Saw Mill River Road, Hawthorne NY, 10532 
mykim   us.ibm.com
phone: 914-784-7709, fax: 914-784-7455 (tie: 863-7709) 
	Rob Koenen <rkoenen   intertrust.com> on 10/18/2001 12:43:01 PM 
To: "'Mariano Padilla'" <mpadilla   sdi-media-usa.com>, Rob Koenen
<rkoenen   intertrust.com>
cc: Scott Rose <srose   sdi-media-usa.com>, Michelle Y Kim/Watson/IBM   IBMUS
Subject: RE: MPEG 4 
	Again I suggest that you ask these questions on the list.
But I copy Michelle Kim of IBM - there is no better expert
on flextime to be found. 
Rob
-----Original Message-----
From: Mariano Padilla [< mailto:mpadilla   sdi-media-usa.com
<mailto:mpadilla   sdi-media-usa.com> >]
Sent: Thursday, October 18, 2023 9:38 AM
To: 'Rob Koenen'
Cc: Scott Rose
Subject: RE: MPEG 4 
Rob, 
Thank you. I will subscribe. Specifically we are wondering the
synchronization of subtitles on MPEG 4 stream. In the overview, written
by you, in section 8.3.2, 'The FlexTime model allows the content author to
express synchronization among MPEG-4 objects with streams or stream
segments, by assigning temporal relationships among them.' then the
following questions: 
1) Are subtitle streams to be synchronized considered MPEG objects?
2) If the subtitle streams to be synchronized are MPEG objects, do
they have to behave according to the SMIL text guidelines in section 4.2.2
of your overview?
3) If 2 above is true, then does SMIL used in MPEG4 allow milisecond
or frame-accurate synchronization with the video?
4) If there is no text/SMIL synchronization of multinational
subtitles, how can synchronization of external images be achieved? Is it
similar to the MPEG2/DVD model? 
Thanks again, 
Mariano 
-----Original Message-----
From: Rob Koenen [< mailto:rkoenen   intertrust.com
<mailto:rkoenen   intertrust.com> >]
Sent: Wed, October 17, 2023 5:17 PM
To: 'Mariano Padilla'; Rob Koenen
Cc: Scott Rose
Subject: RE: MPEG 4 
Mariano, 
you can ask me but I am not the best source for technical
details. I could forward your questions to experts. I would suggest,
however, that you subscribe yourself to the technotes mailing list
of the MPEG-4 Industry Forum (this is a public list) and ask your
question there. Then I will monitor that you indeed do get an answer. 
See < http://www.m4if.org <http://www.m4if.org> > and in particular
< http://www.m4if.org/public/publiclistreg.html
<http://www.m4if.org/public/publiclistreg.html> > 
Hope this helps, 
Rob 
-----Original Message-----
From: Mariano Padilla [< mailto:mpadilla   sdi-media-usa.com
<mailto:mpadilla   sdi-media-usa.com> >]
Sent: Wednesday, October 17, 2023 5:11 PM
To: 'rob   intertrust.com'
Cc: Scott Rose
Subject: MPEG 4 
	Rob, 
I got your e-mail address form the MPEG-4 standards article and I
have a couple of questions, specifically regarding section 8.3 'FlexTime'
and synchronization of subtitles. Can I address these questions to you,
or if you have someone else that can answer them for me? Please advise. 
We at SDI Media USA specialize in Theatrical, DVD, Home Video and
broadcast subtitles and captions for over 40 languages. Naturally we are
looking at how subtitles are synchronized in MPEG-4. 
I thank you in advanced for the time spent. 
Mariano J. Padilla
Director of IT/R&D
SDI Media USA
T 323-602-5406
F 323-602-5450
www.sdi-media-usa.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: /pipermail/mp4-tech/attachments/20011023/c2382a1c/attachment.html


More information about the Mp4-tech mailing list