High Efficiency Image File Format (HEIF, ISO/IEC 23008-12) specifies the storage of individual images, image sequences and their metadata into a container file conforming to the ISO Base Media File Format (ISO/IEC 14496-12). HEIF includes the storage specification of HEVC intra images and HEVC image sequences in which inter prediction is applied in a constrained manner. Use cases supported by HEIF include:
As HEVC provides support for various chroma formats and sample fidelities up to lossless coding, the format can serve the whole spectrum of use cases from today's consumer devices storing images typically at 8 bits per sample to high-end professional devices with sample fidelity and dynamic range requirements going all the way up to 16 bits per sample.
Computational photography forms a new category of use cases that can benefit from the HEIF file format. Now a set of related images can be stored in a single file with associated metadata indicating relationships between different pictures.
HEIF specifies a structural format, from which codec-specific image formats can be derived. HEIF also includes the specification for encapsulating images and image sequences conforming to the High Efficiency Video Coding (HEVC, ISO/IEC 23008-2 | ITU-T Rec. H.265), Versatile Video Coding (VVC, ISO/IEC 23090-3 | ITU-T Rec. H.266), Advanced Video Coding (AVC, ISO/IEC 14496-10 | ITU-T Rec. H.264), and JPEG (ISO/IEC 10918-1 | ITU-T Rec. T.81).
In ISOBMFF, a continuous or timed media or metadata stream forms a track, whereas static media or metadata is stored as items. Consequently, HEIF has the following basic design:
A file may contain both image items and image sequence tracks along with other media. For example, it is possible to create a file, which includes image items or image sequence tracks conforming to HEIF, and audio and timed text tracks conforming to any derivative format of the ISOBMFF.
Image I. ISOBMFF compliant file high level structure with meta box and items
Files conforming to ISOBMFF consist of a sequence of data structures called boxes, each containing a four-character code (4CC) indicating the type of the box, the size of the box in terms of bytes, and the payload of the box. Boxes may be nested, i.e. a box may contain other boxes. ISOBMFF and HEIF specify constraints on the allowed box order and hierarchy.
Files conforming to HEIF start with a FileTypeBox as defined in the ISOBMFF standard, which contains a list of brands the file complies with. As the FileTypeBox is located at the start of the file, it provides easily accessible indications of the file contents to media players. Each brand is identified by its unique four-character code. The specification of a brand can include requirements and constraints for files of the brand and for file players supporting the brand. A brand included in the FileTypeBox permits a player that supports the requirements of the brand to play the file.
The brands specified in the HEIF standard are presented in Table I. The HEIF standard specifies the 'mif1' and 'msf1' structural brands, as well as 'mif3' for the low-overhead (SlimHEIF) profile. Additionally, codec-specific brands are specified for HEVC, VVC, AVC, and JPEG as listed in Table I.
Table I. Brands, MIME subtypes, and file extensions for HEIF.
| Brand | Coding format | Image or sequence? | MIME Type | MIME subtype | File extension |
|---|---|---|---|---|---|
| mif1 | Any | image | image | heif | .heif |
| msf1 | Any | sequence | image | heif-sequence | .heif |
| heic | HEVC (Main or Main Still Picture profile) | image | image | heic | .heic |
| heix | HEVC (Main 10 or format range extensions profile) | image | image | heic | .heic |
| hevc | HEVC (Main or Main Still Picture profile) | sequence | image | heic-sequence | .heic |
| hevx | HEVC (Main 10 or format range extensions profile) | sequence | image | heic-sequence | .heic |
| vvic | VVC | image | image | heif | .heif |
| vvis | VVC | sequence | image | heif-sequence | .heif |
| avic | AVC | image | image | avif | .avif |
| avis | AVC | sequence | image | avif | .avif |
| jpeg | JPEG | image | image | heif | .heif |
| mif3 | Any (low-overhead profile) | image | image | heif | .heif |
Internet media types, also known as MIME (Multipurpose Internet Mail Extensions) types, are used by various applications to identify the type of a resource or a file. MIME types consist of a media type ('image' in the case of HEIF files), a subtype, and zero or more optional parameters. For multi-purpose files, the selection of the subtype can be made on the basis of the primary use of the file.
An optional codecs MIME parameter can be present to indicate the used coding formats of the tracks and items present in the file. The codecs MIME parameter also includes the profile-tier-level value to which an HEVC-coded image item or an image sequence track conforms. More information about the MIME type registration and optional parameter can be found in Annex D and Annex E of the HEIF standard.
Table IX provides a comparison of the features of HEIF to other selected image formats. It can be observed that HEIF is more extensible and comprehensive than the other compared file formats. Particularly the possibility to include other media types, the advanced multi-picture features, and the support for non-destructive editing make HEIF more advanced than the other formats. The rich set of features make HEIF suitable for a broad range of devices and applications, including for example burst photography.
The references used to conclude the information in Table II are included in Table X.
Table II. Comparison of the features of some image file formats.
| .heic | JPEG/Exif | PNG | GIF (89a) | WebP | JPEG-XR / TIFF | JPEG-XR / JPX | BPG | |
|---|---|---|---|---|---|---|---|---|
| Formats and extensibility | ||||||||
| Base container file format | ISOBMFF | TIFF | - | - | RIFF | TIFF | - 4 | - |
| Lossy compression | Yes (HEVC) | Yes (JPEG) | No | No | Yes (VP8) | Yes | Yes | Yes (HEVC10) |
| Lossless compression | Yes (HEVC) | Yes (TIFF Rev 6.0) | Yes (PNG)1 | Yes (GIF)1 | Yes (VP8L) | Yes | Yes | Yes (HEVC10) |
| Extensible to other coding formats | Yes | Yes8 | No | No | No | Yes8 | Yes5 | No |
| Metadata format (on top of internal) | Exif, XMP, MPEG-7 | Exif, XMP | - | - | Exif, XMP | Exif, XMP | JPX, (XMP)6 | Exif, XMP |
| Extensible to other metadata formats | Yes | No | No | No | No | No | Yes (XML-based) | Yes |
| Other media types (audio, text, etc.) | Yes | Audio2 | No | No | No | No | Yes7 | No |
| Multi-picture features | ||||||||
| Multiple images in the same file | Yes | No11 | No | Yes3 | Yes3 | No | Yes | Yes9 |
| Image sequences / animations | Yes | No | No | Yes | Yes | No | Yes | Yes |
| Image coding | Yes | No | No | No | No | No | No | Yes |
| Derived images | ||||||||
| Multiple-of-90-degree rotations | Yes | Yes | No | No | No | Yes | Yes | No |
| Cropping | Yes | No | No | No | No | No | Yes | No |
| Tiling/overlaying | Yes | No | No | No | Yes | No | Yes | No |
| Extensible to other editing operations | Yes | No | No | No | No | No | No | No |
| Auxiliary picture information | ||||||||
| Transparency (alpha plane) | Yes | No | Yes | No12 | Yes | Yes | Yes | Yes |
| Thumbnail image | Yes | Yes | No | No | No | Yes | Yes | Yes |
1 In GIF and indexed color PNG encoding, lossy color quantization is applied while the color-quantized image is losslessly compressed.
2 PCM, µ-Law PCM and ADPCM encapsulated in RIFF WAV
3 Only for animations and tiling/overlaying
4 JPX is a box-structured format compatible with ISOBMFF. However, only the File Type box is common in JPX and ISOBMFF.
5 Encapsulation of JPEG-2000 and JPEG-XR have been specified for JPX container. Mappings for other codecs could be similarly specified.
6 JPX (ITU-T T.800 and T.801) specifies an own metadata schema, but is capable of carrying an XML formatted metadata, such as XMP.
7 JPX can contain media complying with ISOBMFF (or derivatives thereof). No accurate synchronization between JPX animations and other media.
8 TIFF as a container format facilitates extensions to other coding formats.
9 Only for animations, thumbnails, and alpha planes. Non-timed image collections not supported.
10 HEVC Main 4:4:4 16 Still Picture profile, Level 8.5, with additional constraints
11 Can be enabled through the MP extension
12 A palette index for fully transparency can be specified
It is acknowledged that a summary such as that in Table II might be somewhat incomplete when it comes to features of different formats. For example, the table does not cover some of the extensions of JPEG. We welcome feedback and corrections to the table.
Table III. References for the compared image file formats
| Image format | Version or date | Reference and/or URL |
|---|---|---|
| HEIF (.heic) | 03/2015 |
ISO/IEC 23008-12 |
| JPEG |
ISO/IEC 10918-1 | ITU-T Rec. T.81 http://www.w3.org/Graphics/JPEG/itu-t81.pdf |
|
| Exif | http://www.cipa.jp/std/documents/e/DC-008-2012_E.pdf | |
| PNG | http://www.w3.org/TR/PNG/ | |
| GIF | 89a | http://www.w3.org/Graphics/GIF/spec-gif89a.txt |
| WebP |
https://developers.google.com/speed/webp/docs/riff_container VP8L: https://developers.google.com/speed/webp/docs/webp_lossless_bitstream_specification |
|
| JPEG-XR |
ISO/IEC 29199-2 | ITU-T Rec. T.832 ISO/IEC 15444-2 | ITU-T Rec. T.801 (for JPX) |
|
| BPG | 0.9.5 | http://bellard.org/bpg/bpg_spec.txt |
The file size of container files are directly affected by the compression performance of the image/video codec being utilized. Such a comparison can be found in Annex B of this document.
Table IV illustrates coding efficiency of HEVC intra coding with respect to well-known still picture codecs. The results indicate that JPEG would require on average 139 % higher bitrate than HEVC (i.e. 2.39 times the file size) in order to achieve the same objective picture quality. For JPEG-XR and JPEG-2000 the average increase in bitrates are 66 % and 44 %, respectively.
Table IV. HEVC intra coding performance with respect to legacy formats. Bitrate increase required to achieve the objective quality provided by HEVC intra coding is reported for each test category.| Class | Resolution | Charasteristics | JPEG | JPEG XR | JPEG 2000 |
|---|---|---|---|---|---|
| Class A | 2560x1600 | Cropped 4Kx2K sequences for Ultra HDTV services | 87 % | 44 % | 48 % |
| Class B | 1920x1080 | High resolution sequences for streaming and broadcast services | 124 % | 62 % | 15 % |
| Class C | 832x480 | Medium resolution sequences for Internet/mobile video services | 122 % | 53 % | 50 % |
| Class C | 832x480 | Medium resolution sequences for Internet/mobile video services | 122 % | 53 % | 50 % |
| Class D | 416x240 | Low resolution sequences for services to resource constrained devices | 110 % | 47 % | 43 % |
| Class E | 1280x720 | 720p sequences for video conferencing applications | 170 % | 73 % | 23 % |
| Class F | 1024x768, 1280x720 | Computer screen content and computer generated content | 223 % | 118 % | 87 % |
| Average | 139 % | 66 % | 44 % |
Table V indicate that one can expect that for natural content the restricted inter coding can typically provide two to three times better compression than intra picture coding. In special cases like animations where majority of the scene is static the compression efficiency can significantly exceed those levels and be tens of times more efficient than intra coding.
Table V. Coding efficiency improvements provided by low latency predictive coding of the HEVC Image File Format. Bitrate impact and coding gain are reported with respect to HEVC intra coding.| Content | Type | Frames | Bitrate change | Coding gain |
|---|---|---|---|---|
| Class A | Image burst | 8 | -46 % | 1.9 |
| Class B | Image burst | 8 | -51 % | 2.0 |
| Class C | Image burst | 8 | -60 % | 2.5 |
| Class D | Image burst | 8 | -63 % | 2.7 |
| Class E | Image burst | 8 | -79 % | 4.8 |
| Class F | Image burst | 8 | -55 % | 2.2 |
| Memorial | Exposure stack | 16 | -29 % | 1.4 |
| Mersu | Focal stack | 13 | -25 % | 1.3 |
| Car&Tractor | Cinemagraph | 48 | -97 % | 33.3 |
Multiple images can be stored in a HEIF file. It can be useful to differentiate between them by assigning them certain roles. The roles specified in HEIF are listed and described in Table VI. Note that a single image can be associated with more than one role.
Table VI. Roles of images.
| Role | Description |
|---|---|
| cover image |
A representative image of the image items and image sequence tracks of the file. The cover image should be displayed when no other information is available on the preference to display the image items of the file. The file can have only one cover image. |
| thumbnail image | A smaller-resolution representation of a master image. |
| auxiliary image |
An image that complements a master image. For example, an alpha plane or a depth map. Can assist in displaying the master image but is not typically displayed as such. |
| master image | An image that is not a thumbnail image or an auxiliary image. Typically represents a full-resolution displayable image. |
| hidden image | An image that should never be displayed. Can be present in the file for example as an input image for a derived image. |
| pre-derived coded image |
A coded image that has been derived from other images. For example, a high dynamic range image derived from an exposure-bracketed set of images. |
| coded image | A coded representation of an image. |
| derived image | An image that is represented in a file by an indicated operation to indicated input images and can be obtained by performing the indicated operation to the indicated input images. |
HEIF allows the storage of image properties which are shared among different image items in a compact way. These properties are stored in ItemPropertyContainerBox. There are mainly two types of properties: descriptive and transformative. Descriptive properties provide information about the image item without applying modifications on the image itself. Transformative properties provide information about the transformative modification that needs to be done on the image item. The order of application of these properties to the image items are defined in the standard. Table VII lists currently defined properties. In addition to descriptive image properties, image items can optionally be characterized with metadata items, the format of which follows Exif, XMP, or MPEG-7 metadata.
Table VII. Image Properties
| Name | Type | Description |
|---|---|---|
| Decoder configuration and initialization | Descriptive Property | The information needed to initialize the decoder. The structure of this information is usually defined in the related image coding format specification. |
| Image spatial Extents (‘ispe’) | Descriptive Property | indicates the width and height of the associated image item |
| Pixel Aspect Ratio (‘pasp’) | Descriptive Property | has the same syntax as the PixelAspectRatioBox as defined in ISO/IEC 14496-12. |
| Color Information (‘colr’) | Descriptive Property | has the same syntax as the ColourInformationBox as defined in ISO/IEC 14496-12. |
| Pixel Information (‘pixi’) | Descriptive Property | indicates the number and bit depth of colour components in the reconstructed image of the associated image item. |
| Relative Location (‘rloc’) | Descriptive Property | indicates the horizontal and vertical offset of the reconstructed image item relative to the associated image item. |
| Image Properties for Auxiliary Images (‘auxC’) | Descriptive Property | Auxiliary images must be associated with an AuxiliaryTypeProperty. |
| Image Rotation (‘irot’) | Transformative | Rotation by 90, 180, or 270 degrees. |
| Clean Aperture (‘clap’) | Transformative | Cropping according to a given cropping rectangle. |
Derived images enable non-destructive image editing, where the original coded images are kept in the file, while new images, called derived images, can be introduced by specifying a transformation operation that is applied to one or more input images. HEIF specifies the generic structures used for storing derived images as items as well as a few specific types of derived images. Derived images can also have descriptive or transformative image properties. Item references of type 'dimg' specify the input image(s) of the derived image. The input images can be coded images or derived images. The derived image types specified in the HEIF standard are listed in Table VIII. Other types may be specified in other documents or later versions of the HEIF standard.
Table VIII. Derived Images
| Name | Description |
|---|---|
| Identity transformation (‘iden’) | Cropping and/or rotation by 90, 180, or 270 degrees, imposed through the respective transformative properties. |
| Image Overlay (‘iovl’) | Overlaying any number of input images in indicated order and locations onto the canvas of the output image. |
| Image Grid (‘grid’) | Reconstructing a grid of input images of the same width and height. |
Here are some examples of Derived Images
HEIF files allow the storage of the metadata which can be related to images and image sequences. Such metadata can be information related to integrity checks, EXIF or XMP data or MPEG-7 related metadata. For image items, such metadata can be stored as a metadata item and reference the related image items with a ‘cdsc’ item reference. For image sequences, timed metadata tracks can refer to an image sequence track with a ‘cdsc’ track reference.
HEIF files can contain several image items representing the same image content (i.e. alternative representations). A media player may select one of the alternative representations of the same image content for displaying. A mechanism, known as entity grouping (defined in the GroupsListBox), is used for indicating alternate groups, which can contain both image items and media tracks (image sequences). The capability to define alternate groups between tracks and image items provides a unique mechanism for display initialization based on preferences of the media player and the content creator. Examples where alternate groups can be useful are:
Images that have a particular correlation (e.g. image bursts or animations like cinemagraphs) can be efficiently stored in HEIF files, thanks to the inherited media track features of ISOBMFF. Such images are called image sequences and they can reside in the same file with image items. Table IX lists the roles of the image sequence tracks which are currently defined.
Table IX. Roles of image sequence tracks.
| Role | Description |
|---|---|
| thumbnail image sequence | A smaller-resolution representation of a master image sequence. |
| auxiliary image sequence | An image sequence that complements a master image sequence. For example, a sequence of alpha plane or depth map images. Can assist in displaying the master image sequence but is not typically displayed as such. |
| master image sequence | An image sequence that is not a thumbnail image sequence or an auxiliary image sequence. Typically contains full-resolution displayable images. |
HEIF enables the usage of Inter-picture prediction for compact storage of image sequences. Moreover, image sequence use cases may require faster access to individual images and the ability to edit individual images without affecting any other images. HEIF therefore includes the following two features:
Figure 1 illustrates how a file player processes the coded images and the derived images included in a file. The file player decodes a coded image into a reconstructed image. Similarly, the file player applies the operation of the derived image to the indicated one or more input images to obtain the respective reconstructed image. The descriptive image properties generally describe the reconstructed image, with the exception of the decoder configuration and initialization information, which is associated with the coded image. The transformative image properties, if any, are applied to the reconstructed image to obtain an output image. The output image can be displayed, when the coded image or the derived image is not a hidden image. The output image can also act as an input image to derived images.
Figure 1. Operations performed by file players
The most important features that enable controlling the playback of an HEIF file are listed in Table X. Some of these features were introduced in the ISOBMFF or ISO/IEC 14496-15 and are explicitly inherited by HEIF, while other features were specifically designed for the HEIF standard.
Table X. Features controlling image sequence playback.
| Feature | First appeared in | Description |
|---|---|---|
| non-displayable sample | ISO/IEC 14496-15 | Is never displayed, but can be used as a reference for predicting other images in the track. |
| timed vs. non-timed playback | HEIF | In timed playback, the image sequence is played as video, whereas in non-timed playback the samples of the track are displayed by other means, such as an image gallery. Non-timed playback may be indicated e.g. when a track is used for achieving a better compression efficiency for an exposure stack. |
| edit list | ISOBMFF | A list of ranges of the image sequence track in their playback order. Enables modifying the playback order and pace of samples. |
| looping | HEIF | HEIF allows indicating edit list repetition e.g. for looping animations. The repetition can be indicated to last for a certain duration or be infinite. |
| cropping and rotation | ISOBMFF | Rectangular cropping and rotation by 90, 180, 270 degrees can be specified. |
The HEIF standard includes the specification for encapsulating HEVC-coded images and image sequences into HEIF-compliant files. The specification includes the following aspects:
The HEIF standard specifies the encapsulation of images and image sequences coded with Versatile Video Coding (VVC, ISO/IEC 23090-3 | ITU-T Rec. H.266) into HEIF-compliant files. VVC is the successor to HEVC, offering further compression efficiency improvements particularly for high-resolution and high-dynamic-range content.
VVC-coded image items use the 'vvc1' coding type. The decoder configuration
information is carried in the VVC configuration item property, whose format follows the
VVC decoder configuration record defined in ISO/IEC 14496-15. The item data format is
identical to the sample format of VVC video tracks, enabling reuse of the same coded
picture as both an image item and a sample in a sequence track.
VVC intra coding is expected to provide significant compression gains over HEVC intra coding, with reported bitrate savings in the range of 20–40% for equivalent subjective quality depending on content class and resolution.
The low-overhead amendment to the HEIF standard (ISO/IEC 23008-12) defines a compact
file-level structure identified by the 'mif3' brand. SlimHEIF reduces the
container overhead by eliminating redundant metadata boxes, making it particularly
effective for very small images such as icons, thumbnails, and emoji.
In a conventional HEIF file, the ISOBMFF box hierarchy introduces a fixed overhead that can be significant relative to the coded image data when the image itself is only a few hundred bytes. SlimHEIF addresses this by using a streamlined box layout that omits boxes whose information can be inferred from context.
Experimental results show that SlimHEIF can reduce file size by 30% to 80% compared to equivalent standard HEIF files for small single-image payloads. For larger images the relative savings are smaller but the absolute overhead reduction remains constant.
HEIF defines several item properties for signalling high dynamic range (HDR) and wide colour gamut (WCG) characteristics of images. These properties allow file players to correctly interpret and display HDR content, or to apply appropriate tone mapping for standard dynamic range displays.
The HDR-related item properties defined in the HEIF standard include:
Wide colour gamut is signalled through the Colour Information property (colr), which can carry an ICC colour profile or an NCLX colour description specifying the colour primaries, transfer characteristics, and matrix coefficients of the image.
HEIF supports tone-map derived images through the 'tmap' item type. A
tone-map item references a base image and a gain map image, enabling a single file
to serve both HDR-capable and SDR-only displays. The gain map encodes the per-pixel
ratio between the SDR and HDR representations, allowing the display device to
reconstruct the appropriate rendition.
This approach is conceptually similar to other gain-map solutions (such as those used in JPEG-R / ISO 21496-1), but is expressed natively within the HEIF derived-image framework using standard ISOBMFF referencing mechanisms.
The ISOBMFF entity grouping mechanism (GroupsListBox) provides a general facility for expressing relationships among image items and tracks. HEIF inherits and extends this mechanism for several standard-defined grouping types beyond the alternate groups described earlier.
Grouping types defined or referenced by the HEIF standard include:
The Multi-Image Application Format (MIAF, ISO/IEC 23000-22) is an application-level profile built on top of HEIF and ISOBMFF. MIAF defines a set of constraints and requirements intended to maximise interoperability between HEIF file producers and consumers. It specifies mandatory features, forbidden features, and recommended practices for a well-defined subset of the HEIF format.
MIAF introduces its own brand ('miaf') and defines application profiles
for specific codec and colour space combinations. A file branded as MIAF-compliant
signals to a player that the file conforms to a predictable, well-constrained subset
of HEIF, reducing the implementation burden on decoders.