[Mp4-tech] [H.264] purpose of non-existing pictures

Thu Jun 1 11:11:23 EDT 2006

Hi Gary / Wesley et al,
thanks for your comments on this topic.
As far as I understood it from the standard a "non-existing" reference
picture may never be referred to by a reference index, which makes sense
in the context of sub-sequences, i.e., pictures from a sub-sequence
refer to pictures from the same sub-sequence, but never to
"non-existing" pictures from other sub-sequences.
However, in case of data loss over the transmission channel the encoder
would have to get feedback about which picture has been lost in order to
know which reference pictures one of the following encoded pictures may
refer to and which they may not refer to when he wants to use this
feature for purposes of error robustness. Of course, this is only the
case if gaps_in_frame_num_value_allowed_flag is 1 since otherwise the
standard leaves it open how to handle lossed pictures, as Wesley also
commented. But I understood that Gary pointed out setting
gaps_in_frame_num_value_allowed_flag to 1 and then using this for error
robustness. If the encoder gets no feedback then it will not be able to
create a stream which still conforms to the standard in the sense that
no references to non-existing pictures are allowed.
Best Regards,
Martin Lange
>-----Original Message-----
>From: Gary Sullivan [mailto:garysull windows.microsoft.com] 
>Sent: Wednesday, May 31, 2024 9:15 PM
>To: Lange Martin (SCI); mp4-tech lists.mpegif.org
>Subject: RE: [Mp4-tech] [H.264] purpose of non-existing pictures
>
>
>Martin et al,
>
>One type of use of this feature is to enable a form of 
>temporal bitstream scalability in which some reference 
>pictures can be removed from the bitstream without harming the 
>ability to decode the remaining pictures.  For example, I 
>suggest reading the information in the standard relating to 
>"sub-sequence" concepts (esp. D.2.11 through D.2.13).  I 
>believe this is the primary purpose.
>
>Also, it can provide a form of robustness to 
>transmission-channel losses of some reference pictures.  If a 
>reference picture becomes "non-existing" due to packet losses, 
>the remaining pictures in the bitstream may still be 
>decodable.  This definition provides a well-defined decoder 
>behavior for environments in which in the system operation may 
>result in the loss of some reference pictures.  For such a 
>case, it might not really be necessary to specify how the 
>decoder would respond to a picture loss -- however, this 
>feature allows the encoder to decide whether the decoder 
>should flag the loss of a picture as a real problem or not (by 
>setting the value of the gaps_in_frame_num_value_allowed_flag 
>to 0 or 1) and allows the encoder to understand how the 
>decoder will respond to such events, which can be beneficial 
>to know for various purposes.
>
>In fact, the way the decoder responds to a missing reference 
>picture when gaps_in_frame_num_value_allowed_flag is equal to 
>1 may often also be a good way for the decoder to respond to a 
>missing reference picture when 
>gaps_in_frame_num_value_allowed_flag is equal to 0 as well.  
>But when when gaps_in_frame_num_value_allowed_flag is equal to 
>0, it is up to the individual product designer to decide how 
>the decoder should respond to missing reference pictures.
>
>Best Regards,
>
>Gary Sullivan
>

>Dear Martin,
>
>I think you rather have to look at "non-existing" frames from 
>a decoder point of view. Then, temporal scalability is a 
>possible scenario in which gaps might be introduced in 
>frame_num. Frame loss due to packet dropping in a congested 
>network is another one.
>
>When no gaps in frame_num are allowed (the value of 
>gaps_in_frame_num_value_allowed_flag is equal to zero), 
>decoders infer disposed reference frames from gaps in the 
>frame_num syntax element and insert a "non-existing'' frame to 
>the reference picture buffer as if the frame was received and 
>decoded. This procedure guarantees that the contents of and 
>the picture order in the reference picture buffer remain 
>unchanged when it comes to the decoding of the remaining 
>pictures. This process is not executed when gaps in frame_num 
>are allowed (the value of gaps_in_frame_num_value_allowed_flag 
>is equal to one). The latter is for instance useful when 
>temporal scalability is implemented by making use of 
>sub-sequences. The sub-sequence feature of H.264/AVC enables 
>hierarchical temporal scalability, which allows disposal of 
>reference pictures from a coded bitstream without affecting 
>the decoding of the remaining pictures in the bitstream 
>(hereby intentionally introducing gaps in frame_num when 
>reference pictures are dropped).
>
>The latter is also discussed in the following papers:
>
>[1] Dong Tian et al.: Sub-Sequence Video Coding for Improved 
>Temporal Scalability; [2] Ville-Pekka Limnell et al.: Quality 
>Scalability in H.264-AVC Video Coding.
>
>From an encoder perspective, "non-existing" frames will 
>typically be created by using skipped slices in order to 
>achieve bit rate savings (or to keep the amount of pictures in 
>the coded bitstream the same after offline exploitation of 
>temporal scalability, something that significantly eases the 
>synchronization with an audio stream when varying coding 
>patterns are in use).
>
>Best regards,
>Wesley De Neve

>+> -----Original Message-----
>+> From: mp4-tech-bounces lists.mpegif.org 
>+> [mailto:mp4-tech-bounces lists.mpegif.org] On Behalf Of 
>+> Martin.Lange sci-worx.com
>+> Sent: Wednesday, May 31, 2024 7:14 AM
>+> To: mp4-tech lists.mpegif.org
>+> Subject: [Mp4-tech] [H.264] purpose of non-existing pictures
>+> 
>+> Hello everyone,
>+> 
>+> a thing which I was always wondering is the purpose of non-existing 
>+> pictures. What is a scenario in which an encoder would decide to 
>+> intentionally introduce gaps in frame_num, thus producing 
>+> non-existing pictures? The only effect I see is that they 
>make other 
>+> pictures fall out of the decoded picture buffer and that they also 
>+> get incorporated into reference lists, thus changing the reference 
>+> indices other pictures would have without their presence. Can this 
>+> lead to any gain in coding efficiency? Can anybody comment on what 
>+> was the intention of this tool?
>+> 
>+> If this is a question which has already been posted in the past, I 
>+> would be happy if anyone could point me to some kind of FAQ of this 
>+> forum, if it exists (does it?).
>+> 
>+> Best regards,
>+> Martin Lange
>+> 
>+> _______________________________________________
>+> NOTE: Please use clear subject lines for your posts. 
>Include [audio, 
>+> [video], [systems], [general] or another apppropriate identifier to 
>+> indicate the type of question you have.
>+> 
>+> Note: Conduct on the mailing list is subject to the Antitrust 
>+> guidelines found at 
>+> http://www.mpegif.org/public/documents/vault/mp-out-30042-Ant
>+> itrust.php
>+> 
>