3GP and 3G2

http://www.cnwrecovery.com/html/3gp_recovery.html
http://www.mplayerhq.hu/DOCS/HTML/en/encoding-guide.html

Recovery of deleted and fragmented 3GP and MP4 files

CnW Recovery software can recover many fragmented video files - a feature that very few data recovery packages even try.  A deleted file is often fragmented, and some files are created fragmented.  Many packages just use data carving which finds sequential files, but can not recreate a fragmented file.
Cell phones and mobile phones often produce video files that are based on the Quick time format, with extensions such as .3GP, .MP4, .MOV.  As with any file, it is often possible for these file to be fragmented, and so if recovered with data carving, the file will not be valid.  CnW defragmenting routine is an option within data carving - at the end of carving the files may be optionally defragmented.  This option is part of even the most basic CnW package - because it is very important for successful recovery procedures.
The details below give a technical description but for most users it is only necessary to run the program.  New variations are being found and implemented all of the time - CnW are happy to accept new examples which will be implemented at no cost to the user. 

Typical data structure

The file structure is a series of 4 byte lengths followed by 4 byte tags

00000000   00 00 00 18 66 74 79 70 - 33 67 70 35 00 00 03 00        ftyp3gp5
This type of file can be recognised by it’s header which started with a 4 byte length (normally 0x14, 0x18, 0x1c 0x20 or 0x24) followed by the string ‘ftyp’.  The next 4 bytes gives the file type, such as 3gp, or mp42.  There are about 30 recognised file types - CnW software will convert most to be MP4.  See here for more details www.ftyps.com
The next tag is often a ‘moov’ tag which contains all data pointers etc.  The structure is moderately complex, and can allow for multiple tracks, typically one for video, and the other for audio

00000010   6D 70 34 32 6D 70 34 31 - 00 00 0B E6 6D 6F 6F 76    mp42mp41   æmoov
The final important tag is ‘mdat’ that contains all the video and sound data.
To reconstruct the files, the process is to analyze the moov segment and determine where each video frame starts.  (This is based on information in the stco and stsz tags).  On may files, the start of the video frame is fixed data, and so data cluster can be searched for and tested to make sure they contain a frame start (or starts) at a known offset within the cluster.
One significant problem with the above description is that many files are structured  ftyp-moov-mdat, but other ones are ftyp-mdat-moov. In the later case, it is necessary to search for the matching moov fragment before reconstructing the mdat segment.  With file fragmentation, the mdat is the one most likely to be fragmented as the moov segment is often less than 64K in length.  V3.29 will process both  ftyp-moov-mdat  and ftyp-mdat-moov files up to a maximum file size of 16MB.  Later versions will support any file length..  Current success rate is between 0 and 100%, it all depends how fragmented the files are, and if the fragments do actually exist.  We have also seen files where there is an incomplete mdat segment and no moov segment at all.  Also, the length of the ftyp header is zero  Current development plan working on reconstructing the a missing moov segment based on a sample from the same camera.  Due to variations this development is on a per codec basis.

How defragmenting works

The basic structure of the file is one of a four byte length, then a four byte tag, followed by data.  This means that the location of the next tag is always known, although the type of tag can only be guessed.  For the mdat segment, unfortunately there is only a single header and tag, but the moov segment may have a 100 or so tags.  Typically the moov segment is short, but a complete one is essential to view a possibly incomplete mdat.
With defragmenting, it is assumed that data is always in original clusters, although the location of a cluster is not known.  This mens that if a tag is expected at location 0x579421 in a file, and the cluster size is 0x4000, then clusters are tested to see if they have a suitable tag in location 0x1421  (ie 0x579421 modulus 0x4000).  For many moov segments this will produce a good reliable result.  Problems arise on long files, and sometimes the tag ‘stts’ and ‘stsz’ can be very long. If this sits over a fragment location, then different functions are required to analyse data and try and add the relevant pieces.
To reconstruct mdat segments is it largely essential to have a valid moov segment.  From this, key elements can be found, eg frame starts, and matched with possible data clusters.

No moov segment

Video has been come across where the camera did not complete the file.  ie it has a dummy ftyp header, a mdat section but no moov.  The indication that this is an incomplete file, rather than just fragmented is that the ftyp header length is set as zero.  CnW is currently working on creating a moov section to enable the mdat to be displayed.

Non sequential recording

An issue with several cameras is that the data is not recorded onto the memory chip sequentially.  This is because the moov segment can only be created when the mdat segment is complete.  The approach is to record the segements as ftyp-mdat-moov with each segment on a cluster boundary.  The FAT on the disk can then be changed to that logically the data is ftyp-moov-mdat.  However, when files are deleted, this modified FAT is lost and recovery difficult.  Fortunately, CnW Recovery defragmentation software detects this pattern and corrects the files, allowing for very successful recovery.

User support

CnW is very keen to help ever customer.  If a situation arises where recover has not been possible, CnW will assist.  If necessary, the media, or a copy, can be sent to CnW who will extract the files.  As this is both good customer support, and useful development, there is no extra charge for this work - except possible postage costs.
3GP is actually a simplified version of MPEG-4 (MP4).  Video frames can either be MPEG-4 or H.263

Incomplete files - or not finalised

Occasionally a camera will not finalise a file.  One example is when a camera is dropped and the battery drops out.  This can leave a file with a complete mdat fragment, but no meta data (ie the moov fragment).  This type of file cannot be viewed by normal packages.  CnW is developing tools to overcome this problem.  This involves recreating a moov fragment from scratch - often using a sample working file from the same camera.  The success rate does depend on whether or not CnW soltware recognised the video encoding system used, ie the codec.  CnW are always happy to undertake new examples as part of on going development - there is no charge for this.

Click here for a white paper on the subject.

 _____________________________________________________________________

3GP (3GPP file format) is a multimedia container format defined by the Third Generation Partnership Project (3GPP) for 3G UMTS multimedia services. It is used on 3G mobile phones but can also be played on some 2G and 4G phones.
3G2 (3GPP2 file format) is a multimedia container format defined by the 3GPP2 for 3G CDMA2000 multimedia services. It is very similar to the 3GP file format, but has some extensions and limitations in comparison to 3GP.

Contents 1 Specifications 2 Technical details 2.1 3GP 2.2 3G2 3 Device support 4 Software support 5 See also 6 References 7 External links

Specifications

3GP is defined in the ETSI 3GPP technical specification.^[1] 3GP is a required file format for video and associated speech/audio media types and timed text in ETSI 3GPP technical specifications for IP Multimedia Subsystem (IMS), Multimedia Messaging Service (MMS), Multimedia Broadcast/Multicast Service (MBMS) and Transparent end-to-end Packet-switched Streaming Service (PSS).^[2][3][4][5]
3G2 is defined in the 3GPP2 technical specification.^[6]

Technical details

Relations between ISO Base Media File Format, MP4 File Format, 3GPP file format and 3GPP2 file format. Based on the 3GPP2 technical specification published on 18 May 2007.^[7]

The 3GP and 3G2 file formats are both structurally based on the ISO base media file format defined in ISO/IEC 14496-12 - MPEG-4 Part 12,^[8][9][10] but older versions of the 3GP file format did not use some of its features.^[7] 3GP and 3G2 are container formats similar to MPEG-4 Part 14 (MP4), which is also based on MPEG-4 Part 12. The 3GP and 3G2 file format were designed to decrease storage and bandwidth requirements in order to accommodate mobile phones.
3GP and 3G2 are similar standards, but with some differences:

• 3GPP file format was designed for GSM-based Phones and may have the filename extension .3gp
• 3GPP2 file format was designed for CDMA-based Phones and may have the filename extension .3g2

Some cell phones use the .mp4 extension for 3GP video.

3GP

The 3GP file format stores video streams as MPEG-4 Part 2 or H.263 or MPEG-4 Part 10 (AVC/H.264), and audio streams as AMR-NB, AMR-WB, AMR-WB+, AAC-LC, HE-AAC v1 or Enhanced aacPlus (HE-AAC v2). 3GPP allowed use of AMR and H.263 codecs in the ISO base media file format (MPEG-4 Part 12), because 3GPP specified the usage of the Sample Entry and template fields in the ISO base media file format as well as defining new boxes to which codecs refer. These extensions were registered by the registration authority for code-points in ISO base media file format ("MP4 Family" files).^{[11] [12]} For the storage of MPEG-4 media specific information in 3GP files, the 3GP specification refers to MP4 and the AVC file format, which are also based on the ISO base media file format. The MP4 and the AVC file format specifications described usage of MPEG-4 content in the ISO base media file format.^[8]
A 3GP file is always big-endian, storing and transferring the most significant bytes first.

3G2

The 3G2 file format can store the same video streams and most of the audio streams used in the 3GP file format. In addition, 3G2 stores audio streams as EVRC, EVRC-B, EVRC-WB, 13K (QCELP), SMV or VMR-WB, which was specified by 3GPP2 for use in ISO base media file format.^[12] The 3G2 specification also defined some enhancements to 3GPP Timed Text. 3G2 file format does not store Enhanced aacPlus (HE-AAC v2) and AMR-WB+ audio streams.^[7] For the storage of MPEG-4 media (AAC audio, MPEG-4 Part 2 video, MPEG-4 Part 10 - H.264/AVC) in 3G2 files, the 3G2 specification refers to the MP4 file format and the AVC file format specification, which described usage of this content in the ISO base media file format. For the storage of H.263 and AMR content 3G2 specification refers to the 3GP file format specification.^[13]

Device support

• Most 3G capable mobile phones support the playback and recording of video in 3GP format (memory, maximum filesize for playback and recording, and resolution limits exist and vary).
• Some newer/higher-end phones without 3G capabilities may also playback and record in this format (again, with said limitations).
• Audio imported from CD onto a PlayStation 3 when it is set to encode to the MPEG-4 AAC codec will copy onto USB devices in the 3GP format.
• The Nintendo DSi supports .3gp on an SD card.

Software support

When transferred to a computer, 3GP movies can be viewed on Linux, Mac, and Windows platforms with MPlayer and VLC media player. Programs such as Media Player Classic, K-Multimedia Player, Totem, RealPlayer, QuickTime, and GOM Player can also be used.
3GP files can be encoded and decoded with open source software FFmpeg.^[14]

See also

• Comparison of (audio/video) container formats

References

1. ^ ETSI 3GPP 3GPP TS 26.244; Transparent end-to-end packet switched streaming service (PSS); 3GPP file format (3GP) Retrieved on 2009-06-02.
2. ^ ETSI (2009-04) ETSI TS 126 234 V8.2.0 (2009-04); 3GPP TS 26.234; Transparent end-to-end Packet-switched Streaming Service (PSS); Protocols and codecs Page 58. Retrieved on 2009-06-02.
3. ^ ETSI (2009-01) ETSI TS 126 140 V8.0.0 (2009-01); 3GPP TS 26.140; Multimedia Messaging Service (MMS); Media formats and codes Page 11. Retrieved on 2009-06-02.
4. ^ ETSI (2009-06). "ETSI TS 126 346 V8.3.0 (2009-06); 3GPP TS 26.346; Multimedia Broadcast/Multicast Service (MBMS); Protocols and codecs". ETSI. p. 85. Retrieved 2009-10-13.
5. ^ ETSI (2009-01) ETSI TS 126 141 V8.0.0 (2009-01); 3GPP TS 26.141; IP Multimedia System (IMS) Messaging and Presence; Media formats and codecs Page 10. Retrieved on 2009-06-02.
6. ^ 3GPP2 (2003). 3GPP2 C.S0050, 3GPP2 File Formats for Multimedia Services, File Format for Multimedia Services for cdma2000. 3GPP2. Retrieved 2009-06-12.
7. ^ ^{a b c} 3GPP2 (18 May 2007) (PDF). 3GPP2 C.S0050-B Version 1.0, 3GPP2 File Formats for Multimedia Services. 3GPP2. pp. 67, 68. Retrieved 2009-06-12.
8. ^ ^{a b} ETSI 3GPP (2008-12-11) (PDF). 3GPP TS 26.244; Transparent end-to-end packet switched streaming service (PSS); 3GPP file format (3GP). ETSI. p. 9. Retrieved 2009-05-30.
9. ^ ISO (2006-04). ISO Base Media File Format white paper - Proposal. archive.org. Archived from the original on 2008-07-14. Retrieved 2009-12-26.
10. ^ ISO (2009-10). ISO Base Media File Format white paper - Proposal. chiariglione.org. Retrieved 2009-12-26.
11. ^ International Organization for Standardization (2008) (PDF). ISO/IEC 14496-12:2008, Information technology -- Coding of audio-visual objects -- Part 12: ISO base media file format. International Organization for Standardization. p. 95. Retrieved 2009-05-30.
12. ^ ^{a b} Registered types - Codecs. Registration authority for code-points in "MP4 Family" files - mp4ra.org. 2008. Archived from the original on 19 April 2009. Retrieved 2009-05-31.
13. ^ 3GPP2 (18 May 2007) (PDF). 3GPP2 C.S0050-B Version 1.0, 3GPP2 File Formats for Multimedia Services. 3GPP2. p. 21. Retrieved 2009-06-12.
14. ^ "FFmpeg, General Documentation, Supported File Formats and Codecs". FFmpeg. Retrieved 2009-06-1
15.  
16.  
17.