MPEG Industry Forum

Interoperability is the key characteristic of standards-based system implementations. This applies to MPEG-4 too. This is why M4IF undertakes extensive interoperability tests of different aspects of MPEG-4. The tests include Systems layer, Video coders, and Audio coders. We do both offline exchange using the MP4 file format and real-time streaming according to the ISMA 1.0 specification. Usage of the ISMA spec shows our commitment to having only one MPEG-4 out there, and not multiple versions defined by different forums.

The goals of the work are as follows:

Testing products, debugging them, and correcting interoperability problems. Sometimes this reveals details of the MPEG-4 specification that may be unclear or ambiguous;
Showing and proving to potential users of MPEG-4 technology that MPEG-4 products are indeed interoperable;
Working towards a self-certification program to certify that tools are MPEG-4 conformant.

How are the tests done?

Interoperability is based on the conformance points as they are defined in the MPEG-4 standard: a Profile at a Level. A profile can be most easily understood as selecting some features (or tools) from the specification; a level complexity establishes limits on those tools, such as bitrates and frame rates.

All interop tests follow a basic pattern of some 8 steps.

The selection of the Profiles to test is determined by gauging the interest among the membership. MPEG-4 has many profiles, which target many different uses and will come to market at different times; industry interest in interoperability testing is a good indicator for which Profiles will be deployed next.
Testing conditions are defined for each interoperability point.
Bitstreams are generated by each encoder-participant (most of the participants have both encoders and decoders).
Bitstream exchange takes place or streams are sent over live server-client connections - each company decodes all the bitstreams from the other participants.
Each company reports results for all the bitstreams. The result can either be pass, or a problem report.
For each of the problems, the cause is determined, with the standard as the ultimate reference
As long as problems are present, participants modify their code to correct them, and then return to step 3.

In a graphic, this looks as follows:

(Step 3) Generating Bitstreams

(Step 4) Decoding and Playing Bitstreams

The following are issues are checked:

Does the input data play without problems?
Is the proper timing observed?
Is the display speed OK?
Is the synchronization of Audio and Video OK?
Are non-AV object played correctly??
Are BIFS (scene description) commands executed correctly?
Is the Object Descriptor data correct?

Summary of previous three rounds of testing

M4IF is now at its 4^th round of interoperability tests. The first three rounds of tests consisted of

MPEG-4 Simple Visual profile
MPEG-4 High Quality Audio Profile, consisting of Advanced Audio Coding (AAC) and CELP speech coding
... in an MP4 file or in raw bitstream formats.

We defined four categories as follows:

Visual Simple Profile + AAC . mp4
Visual Simple Profile + CELP . mp4
Visual Simple Profile /AAC/CELP . m4v/m4a(Raw streams)
Category 1 or Category 2 with BIFS (Binary Format for Scenes) information.

Successive rounds of tests went from simple raw bitstreams to complete MP4 files. The number of reported incompatibilities decreased over these rounds, while the number of participants increased. Testing category 1 through 3 has virtually finished now.

The goals were expanded for 4^th round tests, see below.

Table 1. The historical view of the interoperability test

Rnd	Schedule	Profiles	# Bitstreams	Participants
1	Mar. 2001 ~ June 2001	Visual Simple@L0	21	Avipix, Cirrus Logic, Comverse, Envivio, Hantro, Luxxon, MPEGSolutions, Serome Technology, Videospheres, WebCast (11)
		Audio AAC-LC
2	June 2001 ~ Oct. 2001	MP4 file format	80	Apple, Avipix, Cirrus Logic, Comverse, Diamondback Vision, ENST, Envivio, FhG, Hantro, Interra, iVast, Luxxon, MPEGSolutions, OntimeTek, Philips, Samsung, Serome Technology, Sorenson, Videospheres, Webcast (23)
		Visual Simple@L0
		Audio AAC-LC
3	Oct. 2001 ~ Jan. 2002	MP4 file format	129	Apple, Avipix, Cirrus Logic, Comverse, ENST, Envivio, FhG, Hantro Products Oy, Interra, iVast, Luxxon, MPEGSolutions, ObjectVideo, OntimeTek, Philips, PacketVideo, Samsung, Serome Technology, Sorenson, VideoSpheres, Webcast Technologies, (24)
		Visual Simple@L0
		Audio AAC-LC
4	Jan 2002 ~ Oct 2002	MP4 file format	TBD	Ahead Software, Amphion Semiconductor, Apple Computer, Avipix, Cirrus Logic Inc, comverse, dicas, Emblaze Systems, Envivio, Inc., Fraunhofer IIS (FhG) , Hantro, Huchison Mediator US, IBM, iVast, Motorola, NeoMagic Corporation, onTimetek Inc., Pace Soft Silicon, PacketVideo, Philips, Pixtree Technologies, Samsung Electronics, Sorenson Media, Telecom Italia Lab, Thomson multimedia, Tvia, WebCast Technologies (28 members as of 2 August and growing)
		System simple2D, Advanced2D, Core2D
		Visual Simple@L0, Simple Scalable@L0, Advanced Simple@L3
		Audio High Quality Audio@L4

Brief description of 4^th round of Interoperability tests

The 4^th round of tests is quite a bit more extensive than the first three rounds. It will add Systems Profiles to the testing, and it will include streaming according to the Internet Streaming Media Alliance's specification.

The table below show the 9 test categories:

Cat.	Profile			Remarks
Cat.	System	Visual	Audio	Remarks
1	Simple2D	Simple@L3	HQA@L4¹	Multiple AV objects
2		Simple Scalable@L0	HQA@L4
3		Advanced Simple@L3	HQA@L4
4	Advanced2D	TBD	TBD
5	Core2D	TBD	TBD
6	With BIFS information for the category 1 through 5
7	Raw bitstream for visual Simple scalable, Advanced Simple or High Quality Audio
8	Streaming interoperability			With ISMA spec. 1.0 for the transport protocol
9	Additional test with same conditions as in 3^rd round			For new companies that want to test again. Bitstreams from round 1 to 3 still available.