High dynamic range, backwards

High dynamic range, backwards
USOO8891863B2
(12) United States Patent
Ninan et al.
(54)
(58)
HIGH DYNAMIC RANGE,
BACKWARDS-COMPATIBLE, DIGITAL
Field of Classi?cation Search
See application ?le for complete search history.
(75) Inventors: Ajit Ninan, San Jose, CA (US); Samir
(56)
References Cited
Hulyalkar, Los Gatos, CA (US)
U.S. PATENT DOCUMENTS
(73) Assignee: Dolby Laboratories Licensing
7,639,882 B2
12/2009 Itakura
Corporation, San Francisco, CA (US)
(Continued)
Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
FOREIGN PATENT DOCUMENTS
U.S.C. 154(b) by 49 days.
(21) App1.No.: 13/468,674
MPEG Video Compression” Jul. 2006.
Prior Publication Data
US 2012/0314944 A1
OTHER PUBLICATIONS
Hassan, F., et a1., “High Throughput JPEG2000 Compatible Encoder
for High Dynamic Range Images” ICIP 2008, pp. 1424-1427.
Mantiuk, R., et a1., “Backward Compatible High Dynamic Range
May 10, 2012
(65)
1/2011
12/2008
1/2010
2011-019126
2009002321
2010003692
This patent is subject to a terminal dis
claimer.
(22) Filed:
*Nov. 18, 2014
None
CINEMA
Notice:
US 8,891,863 B2
(10) Patent N0.:
(45) Date of Patent:
(Continued)
Dec. 13,2012
Primary Examiner * Sean Motsinger
(74) Attorney, Agent, or Firm * Fish & Richardson RC.
Related US. Application Data
(60)
Provisional application No. 61/496,445, ?led on Jun.
13, 2011.
(51)
(52)
Int. Cl.
G06K 9/00
(2006.01)
H04N19/85
H04N19/33
(2014.01)
(2014.01)
H04N 21/2343
H04N 19/64
(2011.01)
(2014.01)
H04N21/414
(2011.01)
(57)
ABSTRACT
HDR images are coded and distributed. An initial HDR image
is received. Processing the received HDR image creates a
JPEG-2000 DCI-compliant coded baseline image and an
HDR-enhancement image. The codedbaseline image has one
or more color components, each of which provide enhance
ment information that allows reconstruction of an instance of
the initial HDR image using the baseline image and the HDR
enhancement images. A data packet is computed, which has a
?rst and a second data set. The ?rst data set relates to the
US. Cl.
CPC . H04N 21/234327 (2013.01); H04N19/00903
(2013.01); H04N19/00436 (2013.01); H04N
19/00842 (2013.01); H04N21/41415 (2013.01)
baseline image color components, each of which has an appli
cation marker that relates to the HDR-enhancement images.
The second data set relates to the HDR-enhancement image.
The data packets are sent in a DCI-compliant bit stream.
19 Claims, 6 Drawing Sheets
USPC .......................... .. 382/166; 382/232; 382/248
1 05\
115\
120\
2K Files
2K master
2K
Projection
e?
System
A
110\
4K Files
2K Files
Q?
ele
’Q'5‘
\
4K
4K master
JPEG2000
X’Y’Z’
1 25
Projection
System
4K Files
US 8,891,863 B2
Page 2
(56)
References Cited
OTHER PUBLICATIONS
Ward, G., et al., “Subband Encoding of High Dynamic Range Imag
U.S. PATENT DOCUMENTS
ery» ACM 2004‘
Ward, G., et al., “JPEG-HDR: A Backwards-Compatible, High
Dynamic Range Extension to JPEG” ACM 2006.
*
,
a/emos
,
ard et al.
............... ..
348/223.1
8,334,911 B2 * 12/2012 Jia et 31‘ ““““““““““ H 348/2231
8,508,617 B2 * 8/2013 Jia et 31. ................... .. 348/2231
2007/0133889 A1* 6/2007 Horie etal. ................. .. 382/232
2007/0258641 A1
11/2007 Srinivasan
.
.
.
.
.
.
Dynamic Range Images Using JPEG2000” Proc. of SPIE, IS &T
2007/0291179 A1* 12/2007 Sterling et al._ .............. .. 348/642
Electronic Imaging, SPIE V01, 7257, 2009,
5383/3153??? £11: £333 855521;???182l; 5757345685
2009/0110073 A1,,
2010/0008418 A1 >1
.
Xu, Ruifeng, “Real-Time Realistic Rendering and High Dynam1c
Range Image Display and Compression” ACM 2005
Springer, D., et al., “Lossy Compression of Floating Point High
4/2009 Wu et al‘ n
1/2010 Wu et al‘ N
375/240‘15
375/240‘12
H» et a1» “1mg? 99mpressi°n Suitable for High DYnamic
ge Image Rendering 22nd International Conference on
Advanced Information Networking and ApplicationsiWorkshops,
2010/0027619 A1 *
2/2010 Doser et al. ............ .. 375/24002
2008 IEEE, pp 1029-1032
2010/0091840 A1*
2010/0111167 A1*
4/2010 Gao et al. ................. .. 375/2402
5/2010 Wu et al~ ~~
37504012
Xu, R., et al., “High Dynamic Range Still-Image Encoding in JPEG
2000” published by the IEEE Computer Society, Nov/Dec. 2005,
i
$30 egal'
in
aoe
l
a.
2010/0172411 A1 *
7/2010 Efregmov et a1.
2010/0208809 A1 *
8/2010 Yin et al‘
IEEE Computer Graphics and Applications, pp. 69-76.
..
.
-
“
-
-
-
375/24012
Mantluk R". et .al" H‘gh. Dynam1c Renge Image and Vlfleo com'
375/240‘12
pressioniFidelity Matching Human Visual Performance 2007.
2010/0208810 A1*
8/2010 Yin et al‘
375/240‘12
Richter, Thomas, “EvaluationofFloatingPointImageCompression”
2010/0220789 A1 *
9/2010 Yuwen et al.
375/240.16
2009 IEEE, pp 1909-1912
2010/0220795 A1 *
2010/0220796 A1 *
9/2010 Yin et al.
9/2010 Yin et al~
375/240.29
37504029
Okuda, M., et a1, “Effective Color Space Representation for Wavelet
Based Compression ofHDR Images” 14th International Conference
2011/0235720 A1*
2012/0057788 A1*
9/2011 Banterle et a1
3/2012 Fukuhara et a1
2012/0281009 A1 *
11/2012
.. 375/24025
382/173
Ward et al. .................. .. 345/589
on Image Analysis and Processing 2007‘
XDe th “Ab tXD th” dat d
dA
_ _P
_
O“
_ _eP_
“‘1
e aacfess‘?
Pr'
12 2011
’
'
_
2012/0314773 A1 * 12/2012 Gish et 31‘ “““““““ H 375/240‘16
2012/0314944 A1 * 12/2012 Ninan et al.
382/166
Digital Cinema Initiative, LLC, “Digital Cinema Systems Speci?ca
iion” Version 1~21Mai~2007~
2013/0064462 A1*
2013/0071022 A1*
* Cited by examiner
3/2013 Ninan et al.
. 382/233
3/2013 Jia et a1. ...................... .. 382/166
US. Patent
NOV. 18, 2014
Sheet 2 0f 6
FVIN
592
P
v?
V?
Va
N
F
O
m
P
0
m05NIg-6o8af//m.
.Nw_“_
8me|
I
\IEN
US 8,891,863 B2
US. Patent
NOV. 18, 2014
_mjvaw 580cm
m mg 680cm
9%/
Tml$2w5“
0%26m
mmm
A//9m @|5692
%
M
/ mom EDI Em o
EDI
o om
Lmnocm
Am3
Rm
//mmmm
US 8,891,863 B2
Sheet 3 0f 6
mmm
m mg
$25
1
mam
0mm/
EDI9. mam
(m5
O
O
00
US. Patent
Nov. 18, 2014
Sheet 6 0f6
US 8,891,863 B2
receive a high dynamic range (HDR) image @
receive a tone-mapped (TM) image generated
based on the HDR image @
compute luminance ratio values, on an individual
pixel basis, by dividing luminance values of the
HDR image with luminance values of the TM
image on the individual pixel basis @
apply the luminance ratio values to the HDR
image to create a re-mapped image @
determine residual values in color channels of a
color space based on the re-mapped image and
the TM image w
output the TM image with HDR reconstruction
data, the HDR reconstruction data being
derived from the luminance ratio values and the
residual values w
FIG. 6
US 8,891,863 B2
1
2
HIGH DYNAMIC RANGE,
BACKWARDS-COMPATIBLE, DIGITAL
typify a low dynamic range (LDR), also referred to as a
CINEMA
Advances in their underlying technology however allow
more modern display designs to render image and video
CROSS REFERENCE TO RELATED
APPLICATIONS
content with signi?cant improvements in various quality
standard dynamic range (SDR), in relation to VDR and HDR.
characteristics over the same content, as rendered on less
modern displays. For example, more modern display devices
may be capable of rendering high de?nition (HD) content
This Application claims the bene?t of priority to related,
and/or content that may be scaled according to various dis
play capabilities such as an image scaler. Moreover, some
more modern displays are capable of rendering content with
a DR that is higher than the SDR of conventional displays.
For example, some modern LCD displays have a backlight
Provisional US. Patent Application No. 61/496,445 ?led on
13 Jun. 2011 entitled “High Dynamic Range, Backwards
Compatible, Digital Cinema” by Ajit Ninan, et al. hereby
incorporated by reference in its entirety.
TECHNOLOGY
unit (BLU) that comprises a light emitting diode (LED) array.
The LEDs of the BLU array may be modulated separately
from modulation of the polarization states of the active LCD
The present invention relates generally to digital cinema.
More particularly, an embodiment of the present invention
relates to the coding and transmission of high dynamic range
(HDR) images in a format that is backwards compatible with
20
existing digital cinema speci?cations.
between the BLU array and the LCD screen elements. Their
LED array based BLUs and dual (or N-) modulation effec
tively increases the display referred DR of LCD monitors that
BACKGROUND
As used herein, the term ‘dynamic range’ (DR) may relate
to a capability of the human psychovisual system (HVS) to
perceive a range of intensity (e.g., luminance, luma) in an
image, e.g., from darkest darks to brightest brights. In this
25
have such features.
Such “HDR displays” as they are often called (although
actually, their capabilities may more closely approximate the
range of VDR) and the DR extension of which they are
capable, in relation to conventional SDR displays represent a
signi?cant advance in the ability to display images, video
sense, DR relates to a ‘ scene-referred’ intensity. DR may also
relate to the ability of a display device to adequately or
approximately render an intensity range of a particular
breadth. In this sense, DR relates to a ‘display-referred’ inten
sity. Unless a particular sense is explicitly speci?ed to have
elements. This dual modulation approach is extensible (e.g.,
to N-modulation layers wherein N comprises an integer
greater than two), such as with controllable intervening layers
30
content and other visual information. The color gamut that
such an HDR display may render may also signi?cantly
exceed the color gamut of more conventional displays, even
to the point of capably rendering a wide color gamut (WCG).
particular signi?cance at any point in the description herein,
Scene related HDR or VDR and WCG image content, such as
e.g. interchangeably.
As used herein, the term high dynamic range (HDR) relates
may be generated by “next generation” movie and TV cam
eras, may now be more faithfully and effectively displayed
with the “HDR” displays (hereinafter referred to as ‘HDR
to a DR breadth that spans the some 14-15 orders of magni
displays’).
it should be inferred that the term may be used in either sense,
35
tude of the human visual system (HVS). For example, well
adapted humans with essentially normal (e. g., in one or more
of a statistical, biometric or opthamological sense) have an
40
For example, scene referred HDR content that is captured
with a modern HDR capable camera may be used to generate
an SDR or a VDR version of the content, which may be
intensity range that spans about 15 orders of magnitude.
Adapted humans may perceive dim light sources of as few as
a mere handful of photons. Yet, these same humans may
perceive the near painfully brilliant intensity of the noonday
As with the scalable video coding and HDTV technologies,
extending image DR typically involves a bifurcate approach.
45
displayed on VDR displays or conventional SDR displays. In
one approach, as described in US. provisional application
sun in desert, sea or snow (or even glance into the sun, how
61/476,174 “Improved encoding, decoding, and representing
ever brie?y to prevent damage). This span though is available
to ‘adapted’ humans, e.g., those whose HVS has a time period
high dynamic range images,” by W. Jia et al., herein incorpo
rated by reference for all purposes, the SDR version is gen
erated from the captured VDR version by applying a tone
in which to reset and adjust.
In contrast, the DR over which a human may simulta
neously perceive an extensive breadth in intensity range may
be somewhat truncated, in relation to HDR. As used herein,
the terms ‘visual dynamic range’ or ‘variable dynamic range’
(VDR) may individually or interchangeably relate to the DR
that is simultaneously perceivable by a HVS. As used herein,
50
also describes how an HDR image can be represented by a
baseline SDR image, a ratio image, and a chroma residual.
55
The “Digital Cinema Systems Speci?cation,” Version 1.2,
Mar. 7, 2008, by Digital Cinema Initiatives, LLC, is referred
to herein as the DCI System Speci?cation, or simply as DCI,
VDR may relate to a DR that spans 5-6 orders of magnitude.
Thus while perhaps somewhat narrower in relation to true
scene referred HDR, VDR nonetheless represents a wide DR
breadth. As used herein, the term ‘simultaneous dynamic
range’ may relate to VDR.
mapping operator (TMO) to intensity (e.g., luminance, luma)
related pixel values in the HDR content. The Jia application
de?nes the technical speci?cations and requirements for the
mastering, coding, and distribution of digital cinema content.
DCI supports video streams at two resolutions: 2K (2048><
60
Until fairly recently, displays have had a signi?cantly nar
1080) and 4K (4096x2160). Furthermore, DCI images are
coded as 12-bit, X'Y'Z' color channels, using the J PEG 2000
rower DR than HDR or VDR. Television (TV) and computer
coding standard. This representation is inadequate to fully
monitor apparatus that use typical cathode ray tube (CRT),
represent HDR content. Embodiments of this invention
liquid crystal display (LCD) with constant ?uorescent white
back lighting or plasma screen technology may be con
strained in their DR rendering capability to approximately
three orders of magnitude. Such conventional displays thus
describe methods for the mastering, coding, and distribution
65
of HDR content in formats that are backwards compatible
with the existing DCI speci?cation. As used herein, “JPEG
2000” refers to the JPEG-2000 compressor/decompressor
US 8,891 ,863 B2
3
4
(codec) standard of the Joint Motion Picture Experts Group
(JPEG). As used herein, ‘TIFF’ refers to a Tagged Image File
that conforms to the DCI Systems Standard. Upon receipt of
the data packets, a DCI-compliant decoder may process the
Format.
packets to reconstruct an instance of the initial HDR image,
according to the information from the baseline image and the
The approaches described in this section are approaches
HDR-enhancement images.
that could be pursued, but not necessarily approaches that
have been previously conceived or pursued. Therefore, unless
Thus, an embodiment represents an original HDR image as
a JPEG-2000 baseline image with components that are added
otherwise indicated, it should not be assumed that any of the
approaches described in this section qualify as prior art
to enhance the baseline image. The additional enhancement
information components include a luminance ratio image and
one or more optional chroma residual images. DCI-compliant
decoders may all decode the baseline image. Decoders that
have the capability to decode HDR images may use the base
line image, along with the added enhancement components to
decode the full HDR image, e.g., an instance of the original
merely by virtue of their inclusion in this section. Similarly,
issues identi?ed with respect to one or more approaches
should not assume to have been recognized in any prior art on
the basis of this section, unless otherwise indicated.
BRIEF DESCRIPTION OF THE DRAWINGS
HDR image. An embodiment multiplexes the base layer and
the HDR-enhancement component data into a single, back
An embodiment of the present invention is illustrated by
way of example, and not in way by limitation, in the ?gures of
the accompanying drawings and in which like reference
wards compatible, DCI-compliant stream.
DCI System Speci?cation
numerals refer to similar elements and in which:
FIG. 1 depicts SDR digital cinema system work ?ow
according to the DCI speci?cation;
20
FIG. 1 depicts a simpli?ed diagram of a digital cinema
system work?ow according to the current DCI systems speci
?cation. DCI speci?es the structure of a Digital Cinema Dis
FIG. 2 depicts an SDR digital cinema codestream structure
tribution Master (DCDM) which is used to exchange image,
according to the DCI speci?cation;
FIG. 3 depicts an example layered HDR encoding system
audio, and metadata. Once a DCDM is compressed,
encrypted and packaged for distribution, then it is considered
according to an embodiment of the present invention;
FIG. 4 depicts an example HDR DCI codestream structure
according to one embodiment of the present invention;
FIG. 5 depicts an example, backwards compatible, HDR
DCI decoding system according to an embodiment of the
25
present invention.
30
a Digital Cinema Package (DCP) (115). DCDMs use a hier
archical image structure that supports both 2K (105) and 4K
(110) resolution master ?les. A studio can choose to deliver
either a 2K or a 4K master. Both 2K (120) and 4K (125)
projectors can decode both 2K and 4K distributions. A 4K
projector receiving a 2K ?le can optionally upscale it. A 2K
projector receiving a 4K master extracts and uses the data for
FIG. 6 depicts an example process ?ow to generate an HDR
decoding the 2K output from the 4K distribution. Up-sam
image according to an embodiment.
pling 2K content or down-sampling 4K content is not a
requirement for any of the DCI-compliant projectors.
DESCRIPTION OF EXAMPLE EMBODIMENTS
35
Uncompressed DCDM ?les use a TIFF-like ?le format
Embodiments describe the coding and transmission of high
dynamic range (HDR) images in a format that is backwards
where image frames are coded using 12-bit unsigned integers
compatible with existing digital cinema speci?cations, such
JPEG 2000 standard (ISO/IEC 15444-1).
as those promulgated by Digital Cinema Initiatives (DCI). In
the following description, for the purposes of explanation,
in the X'Y'Z' color space. For compression, DCI uses the
As depicted in FIG. 2, each compressed frame of a DCI
40
has exactly 3 tiles, each tile containing a Tile-part Header
(TH) 210 and data from one color component (206-0, 206-1,
numerous speci?c details are set forth in order to provide a
thorough understanding of the present invention. It will be
apparent, however, that the present invention may be prac
ticed without these speci?c details. In other instances, well
known structures and devices are not described in exhaustive
and 206-2). In a 4K distribution, each frame has 6 tile parts.
The ?rst 3 tile parts canbe used by a 2K decoder to reconstruct
45
detail, in order to avoid unnecessarily occluding, obscuring,
or obfuscating the present invention.
Overview
An embodiment of the present invention codes and distrib
utes high dynamic range (HDR) images. An initial HDR
image is received. Processing the received HDR image cre
HDR instance of content, such as a movie, are thus produced,
which essentially doubles the size of the ?les with which the
content is stored and/or transmitted, and concomitantly
increases the bandwidth consumed in such transmission. An
embodiment of the present invention provides an HDR-ca
Digital Cinema Systems Speci?cation of the Digital Cinema
image using the baseline image and the HDR-enhancement
images. A data packet is computed. The data packet has a ?rst
a 2K output. The remaining 3 tile parts (208-0, 208-1, and
208-2) contain additional data to decompress each of the 4K
color components.
To accommodate HDR images in a DCI compliant milieu,
an HDR DPCM master may be created. Both an SDR and an
50
ates a coded baseline image, which is compliant with the
Initiative (DCI Systems Standard), and one or more HDR
enhancement images. The coded baseline image has one or
more color components. The coded baseline image and the
one or more HDR-enhancement images each provide
enhancement information. The enhancement information
allows reconstruction of an instance of the received HDR
distribution consists of tiles. In a 2K distribution, each frame
55
pable DCI-compliant system that is backwards-compatible
with conventional DCI systems, which thus obviates dual
production, storage and transmission of both an SDR version
and an HDR version of the same content.
Example Enhancements
60
FIG. 3 depicts an example HDR DCI-like encoding system
300, according to an embodiment of the present invention. An
HDR image or video sequence is captured, e.g., using HDR
data set and a second data set, separated by an application
marker. The ?rst data set relates to the one or more baseline
camera 305. The HDR sequence may also be created or syn
image color components, each of which has an application
marker that relates to the HDR-enhancement images. The
thesized, such as with techniques that relate to computer
second data set is formed in relation to one of the HDR
enhancement images. The data packets are sent in a bit stream
65
generated images (CGI). Following capture, the captured
image or video is processed by a mastering process to create
a target HDR image 312. The mastering process may incor
US 8,891 ,863 B2
5
6
porate a variety of processing steps, such as: editing, primary
and secondary color correction, color transformation, and
noise ?ltering. The HDR output 312 of this process represents
another embodiment of this invention, if the values of the
residual chroma images are too small, then they can be
the director’s intent on how the captured image will be dis
played on a target HDR projector or display.
The input HDR image may be in any color space that
supports a high dynamic range color gamut. In an example
embodiment, the input HDR image is an RGB image. In an
images.
removed and not be included as part of the enhancement
FIG. 4 depicts how these three images are multiplexed and
packetized into a DCI-compliant bitstream (350) according to
an embodiment of the invention. JPEG 2000 coding syntax
allows an encoder to insert application speci?c markers into
the bitstream. A JPEG 2000 compliant decoder which does
example, each pixel in the input HDR image comprises ?oat
ing-point pixel values for all channels (e.g., red, green, and
not recognize the marker, will simply ignore all data follow
blue color channels in the RGB color space) de?ned in the
color space. In another example, each pixel in the input HDR
ing it. However, a J PEG 2000 decoder which recognizes the
marker can extract the additional information and process the
data as needed. The use of such markers allows embodiments
image comprises ?xed-point pixel values for all channels
(e. g., 16 bits or higher/lower numbers of bits ?xed-point pixel
values for red, green, and blue color channels in the RGB
color space) de?ned in the color space. Each pixel may
optionally and/or alternatively comprise down-sampled pixel
values for one or more of the channels in the color space.
The mastering process may also output a corresponding
SDR image 317, representing the director’s intent on how the
captured image will be displayed on a legacy SDR projector
or display. The SDR output 317 may be provided directly
from mastering circuit 310 or it may be generated by a sepa
20
rate HDR-to-SDR converter 315, such as tone mapping
operator (TMO). The input SDR image may also be in any
25
color space that supports a standard dynamic range color
gamut. In an example embodiment it may be in YCbCr or
YUV color space.
In an example embodiment, the HDR 312 and SDR 317
signals are input into Divider 320 which produces a ratio
30
image 322. Ratio image 322 represents the pixel by pixel
ratios of the HDR luminance component (say, Yh) over the
SDR luminance component (say, Ys). Depending on the color
space representation of both the HDR and SDR inputs,
Divider 320 may also perform inverse gamma operations,
color transformations, or other preprocessing operations (not
(405-2).
Under this scheme, a legacy DCI projector will ignore the
HDR-enhancement data and will simply decode an SDR 4K
35
replaced by subtraction.
or 2K bitstream. However, as shown in FIG. 5, an HDR DCI
decoder can extract all the HDR-enhancement information
and reconstruct and display an HDR bitstream.
It should be appreciated that the example packet structure
shown).
Because of the large dynamic range of the pixels in the
Ratio image 322, a Log unit 330 transforms image 322 into a
Log ratio image 332, which subsequently may also be pro
cessed by a Ratio image encoder 340, which outputs an
encoded ratio image 342. In an alternative implementation,
the Log operation may be performed directly on the lumi
nance (or luma) signaleh andYs, and division in 320 can be
of this invention to construct a single, fully compliant, DCI
stream, that supports both SDR and HDR playback.
Denote by APP SEQ (410) a JPEG 2000 marker that in a
DCI stream may identify HDR-speci?c data. Given a 4K DCI
bitstream, FIG. 4 depicts an example of how to embed HDR
enhancement information in to the 4K packets. Since each
packet 208 corresponds to bitstream information for a spe
ci?c color channel (X', Y', or Z'), one can apply a single
HDR-enhancement application marker to all of the HDR
enhancement information. In one implementation, the coded
ratio image data (415-0) may be embedded into the 4Ki0
packet (208-0), as part of the baseline X' data (405-0), the
coded residual for the ?rst color component (CC) (415-1), say
Cb, may be embedded into the 4Kil packet (208-1), as part
of the baseline Y' data (405-1), and the coded residual for the
second color component (415-2), say Cr, may be embedded
into the 4Ki2 packet (208-2), as part of the baseline Z' data
40
in FIG. 4 is independent on how the Ratio image and the
Chroma residual images are coded inside the packets. For
example, luma ratio and chroma residuals can be transformed
and coded in a device-independent color space, as the base
line images. Furthermore, instead of using a single applica
tion marker, two or more separate and distinct markers may
45
be used.
The packet structure shown in FIG. 4 is also applicable to
HDR signal 312 and the Ratio image 322 are also input into
other possible con?gurations. In an example implementation,
Divider 325. If the HDR to SDR transformation maintained
the HDR-enhancement data may be multiplexed with the
three 2K packets. In an example implementation, the HDR
enhancement data may be spread across both the 2K and 4K
the original color balance and there is no color clipping, then
a multiplication of the SDR image 317 with the Ratio image
322 should match the input HDR image color by color. How
ever, in practice, there are differences in the chrominance (or
chroma) channels between the HDR and SDR signals. Given
the ratio image 322, Divider 325 remaps the chroma channels
of HDR signal 312 into an approximation (327) of the chroma
channels of SDR input 317. Thus, subtractor 355, which
computes the chroma difference between signals 327 and the
50
FIG. 5 depicts an example implementation of a DCI com
pliant, backwards-compatible, decoding system according to
the methods of this invention. DCI packets are demultiplexed
and decoded according to the existing DCI speci?cation. A
55
legacy decoder will ignore application-speci?c HDR-en
hancement markers and extract a baseline SDR stream 512.
SDR signal 317, captures chroma information that might
However, an HDR DCI decoder will recognize the APP SEQ
markers (515) and extract the additional, HDR-enhancement
information, such as the coded Ratio image 518 and the coded
have been lost during the HDR to SDR process 315. This
chroma residual may be encoded and compressed further by
Residual encoder 335 to yield coded residual 337.
SDR signal 317 is also coded using a JPEG 2000 DCI
data packets.
60
chroma residuals 517. Following decoding (520 and 530)
which extracts a Log ratio image and converts it back to pixel
encoder (345) to output the baseline, DCI-compliant, image
data (532), the ratio image is multiplied by both the extracted
347.
In an embodiment of the present invention, each frame in
SDR image (512) and the chroma residuals (525) to derive a
?rst estimate of the HDR signal (542) and missing chroma
information (537). These two signals are added together in
the original HDR input 312 is coded using three images: a
baseline CDI-compliant image, a coded Ratio image, and two
coded Chroma residual images. Note that according to
65
adder 545 to create a reconstructed HDR signal 547. The
baseline SDR signal 512 and the HDR-enhancement infor
US 8,891,863 B2
7
8
mation (517 and 518) may be in different color spaces and
mapping operator (TMO) or a user. In an example embodi
ment, the HDR image encoder replaces zero, one, or more
chroma formats. Thus, an embodiment performs or accom
modates additional color correction, color transformation,
and post-processing operations. Following the extraction of
color-channel zero-values in the TM image with values
smaller than a threshold value. This threshold value may be 1,
the HDR 547 and SDR 512 signals, a DCI decoder may also
2, 3, . . . , 10, 11, etc. in various possible embodiments.
perform additional display-speci?c color management pro
In an example embodiment, any tone mapping operations
cessing.
with any TMO and/or any color alterations on any number of
FIG. 6 illustrates an example process ?ow according to a
possible embodiment of the present invention. In some pos
pixels in the TM image may be performed in the process of
generating the TM image.
sible embodiments, one or more computing devices or com
In an example embodiment, the HDR image encoder
ponents such as an HDR image encoder may perform this
applies a color space conversion to at least one of the HDR
process ?ow. The HDR image encoder may be implemented
image, the TM image, or the re-mapped image.
by adding one or more new processing blocks to and/or modi
fying one or more existing processing blocks in, a standard
based image encoder such as a JPEG image encoder. In block
In an example embodiment, luminance residual values
between the TM image and the re-mapped image are all zeros.
For example, in a color space (e.g., YUV) with a luminance
602, the HDR image encoder receives a high dynamic range
(HDR) image. In an example embodiment, the HDR image is
channel (e.g., Y) and two color channels (e.g., Cb and Cr),
one of a ?xed-point image or a ?oating-point image. In an
differences in luminance values between the TM image and
example embodiment, the HDR image is encoded in one of
the re-mapped image (e.g., already, or alternatively after a
JPEG, JPEG-2000, MPEG, AVI, TIFF, BMP, GIFF, or
another image format.
20
Example Computer System Implementation
Embodiments of the present invention may be imple
mented with a computer system, systems con?gured in elec
tronic circuitry and components, an integrated circuit (IC)
In block 604, the HDR image encoder also receives a
tone-mapped (TM) image that was generated based on the
HDR image. The TM image comprises one or more color
alterations that are not recoverable from the TM image with a
color space conversion, in the color space) may be all zeros.
25
luminance ratio image. In an example embodiment, at least
device such as a microcontroller, a ?eld programmable gate
array (FPGA), or another con?gurable or programmable
logic device (PLD), a discrete time or digital signal processor
one of the one or more color alterations in the TM image is
caused by one of clippings (e.g., in R, G, or B pixel values), or
(DSP), an application speci?c IC (ASIC), and/or apparatus
alterations of hues at one or more pixels.
that includes one or more of such systems, devices or com
In block 606, the HDR image encoder computes luminance
ratio values, on an individual pixel basis, by dividing lumi
nance values of the HDR image with luminance values of the
TM image on the individual pixel basis.
In block 608, the HDR image encoder applies the lumi
30
nance ratio values to the HDR image to create a re-mapped
35
ponents. The computer and/or IC may perform, control or
execute instructions relating to HDR image coding and
decoding, such as those described herein. The computer and/
or IC may compute any of a variety of parameters or values
image.
that relate to HDR coding. The image and video dynamic
range extension embodiments may be implemented in hard
ware, software, ?rmware and various combinations thereof.
Certain implementations of the invention comprise com
puter processors which execute software instructions which
In an example embodiment, the HDR image encoder con
verts at least one of the re-mapped image and the TM image
values in color channels of a color space based on the re
cause the processors to perform a method of the invention.
For example, one or more processors in a display, an encoder,
a set top box, a transcoder or the like may implement HDR
mapped image and the TM image. If the original color is
DCI coding and decoding methods as described above by
altered, at least one of the residual values is non-zero. In an
executing software instructions in a program memory acces
from one color space to a different color space.
In block 610, the HDR image encoder determines residual
example embodiment, the color space is an YCbCr color
space; the color channels of the color space comprise a Cb
40
45
color channel and a Cr color channel. The residual values in
the color channels of the color space are calculated as differ
able signals comprising instructions which, when executed
ences between ?rst pixel values, as derived from the
by a data processor, cause the data processor to execute a
re-mapped image, in the color channels and second pixel
values, as derived from the TM image, in the color channels.
In block 612, the HDR image encoder outputs a version of
the TM image with HDR reconstruction data. The HDR
reconstruction data is derived from the luminance ratio values
and the color-channel residual values.
In an example embodiment, the HDR reconstruction data
sible to the processors. The invention may also be provided in
the form of a program product. The program product may
comprise any medium which carries a set of computer-read
method of the invention. Program products according to the
50
invention may be in any of a wide variety of forms. The
program product may comprise, for example, physical media
such as magnetic data storage media including ?oppy dis
kettes, hard disk drives, optical data storage media including
CD ROMs, DVDs, electronic data storage media including
55
ROMs, ?ash RAM, or the like. The computer-readable sig
comprises a residual image with quantized values derived
nals on the program product may optionally be compressed or
from the luminance ratio values and the residual values in the
color channels of the color space. The HDR reconstruction
encrypted.
data may further comprise parameters specifying ranges of
the quantized values.
assembly, device, circuit, etc.) is referred to above, unless
In an example embodiment, the HDR reconstruction data is
stored in an application segment of an image ?le with the TM
image as a base image in the image ?le. In an example
embodiment, the image ?le is in a JPEG-HDR format.
In an example embodiment, the HDR image encoder may
Where a component (e.g. a software module, processor,
60
otherwise indicated, reference to that component (including a
reference to a “means”) should be interpreted as including as
perform one or more sanity checks on the HDR image, for
equivalents of that component any component which per
forms the function of the described component (e.g., that is
functionally equivalent), including components which are not
structurally equivalent to the disclosed structure which per
forms the function in the illustrated example embodiments of
example, before the HDR image is manipulated by a tone
the invention.
65
US 8,891,863 B2
10
determining the ?rst coded chroma residual image
and the second coded chroma residual image based
on the re-mapped image and the tone-mapped
EQUIVALENTS, EXTENSIONS, ALTERNATIVES
AND MISCELLANEOUS
Example embodiments that relate to applying HDR image
coding in digital cinema are thus described. In the foregoing
speci?cation, embodiments of the present invention have
image; and
5
been described With reference to numerous speci?c details
that may vary from implementation to implementation. Thus,
the sole and exclusive indicator of What is the invention, and
is intended by the applicants to be the invention, is the set of
claims that issue from this application, in the speci?c form in
Which such claims issue, including any subsequent correc
tion. Any de?nitions expressly set forth herein for terms con
tained in such claims shall govern the meaning of such terms
as used in the claims. Hence, no limitation, element, property,
feature, advantage or attribute that is not expressly recited in
3. The method as recited in claim 1 Wherein the coded
baseline image comprises a resolution of 4096x2160 (4K).
4. The method of claim 1 Wherein the at least three color
components comprise:
an X' color component;
a Y' color component; and
a Z' color component.
5. The method of claim 1, Wherein the third portion of the
data packet comprises an application marker that separates
a claim should limit the scope of such claim in any way. The
speci?cation and draWings are, accordingly, to be regarded in
the baseline data for the third color component and the coded
an illustrative rather than a restrictive sense.
20
the baseline data for the ?rst color component and the ?rst
25
dard), and (b) one or more HDR-enhancement images
are created from the received HDR image; Wherein the
coded baseline image comprises at least three color
components, Wherein the coded baseline image and the
ratio image.
6. The method of claim 1, Wherein the ?rst portion of the
data packet comprises an application marker that separates
What is claimed is:
1. A method, comprising:
receiving a high dynamic range (HDR) image;
processing the received HDR image, Wherein the received
HDR image comprises (a) a coded baseline image that
complies With the Digital Cinema Systems Speci?cation
of the Digital Cinema Initiative (DCI Systems Stan
sending the computed data packet in a bit stream that
conforms to the DCI Systems Standard.
2. The method of claim 1 Wherein the coded baseline image
comprises a resolution of 2048x1080 (2K).
30
coded chroma residual image.
7. The method of claim 1, Wherein the second portion of the
data packet comprises an application marker that separates
the baseline data for the second color component and the
second coded chroma residual image.
8. The method of claim 5, Wherein the application marker
conforms to a speci?cation of the JPEG 2000 compressor/
decompressor (codec) of the Joint Photographic Experts
one or more HDR-enhancement images each provide
Group (JPEG).
enhancement information to allow reconstruction of an
9. An apparatus comprising a processor, Wherein the pro
cessor is con?gured to:
instance of the received HDR image using the baseline
image and the HDR-enhancement images;
35
prises:
a ?rst data set that relates to the coded baseline image for
the at least three color components; and
a second data set that relates to the HDR-enhancement
the Digital Cinema Systems Speci?cation of the Digital
Cinema Initiative (DCI Systems Standard), and (b) one
40
images,
Wherein each of the residual images is con?gured to
capture chroma information lost in creating the base
line image from the received HDR image, and
Wherein the ?rst coded chroma residual image is embed
ded Within a ?rst portion of the data packet that also
three color components, Wherein the coded baseline
image and the one or more HDR-enhancement
45
HDR image using the baseline image and the HDR
enhancement images;
compute a data packet, Wherein the data packet comprises:
50
a second data set that relates to the HDR-enhancement
images, Wherein each of the HDR enhancement
images comprise:
55
portion of the data packet that also includes baseline
data for a third color component;
Wherein computing the data packet comprises:
a ?rst coded chroma residual image,
a second coded chroma residual image, and
a coded ratio image,
Wherein each of the residual images is con?gured to
capture chroma information lost in creating the base
line image from the received HDR image, and
Wherein the ?rst coded chroma residual image is embed
ded Within a ?rst portion of the data packet that also
generating a tone-mapped image based on the
received HDR image;
generating the coded ratio image, the coded ratio
image comprising luminance ratio values gener
ated by dividing luminance values of the received
includes baseline data for a ?rst color component, the
HDR image With luminance values of the tone
applying the luminance ratio values to the received
HDR image to create a re-mapped image; and
a ?rst data set that relates to the baseline image for the at
least three color components; and
second coded chroma residual image is embedded
Within a second portion of the data packet that also
mapped image;
images each provide enhancement information to
alloW reconstruction of an instance of the received
includes baseline data for a ?rst color component, the
includes baseline data for a second color component,
and the coded ratio image is embedded Within a third
or more HDR-enhancement images are created from the
received HDR image;
Wherein the coded baseline image comprises at least
Wherein the HDR enhancement images comprise:
a ?rst coded chroma residual image,
a second chroma residual image, and
a coded ratio image,
receive a high dynamic range (HDR) image;
process the received HDR image, Wherein the HDR image
comprises (a) a coded baseline image that complies With
computing a data packet, Wherein the data packet com
65
second coded chroma residual image is embedded
Within a second portion of the data packet that also
includes baseline data for a second color component,
and the coded ratio image is embedded Within a third
US 8,891,863 B2
11
12
ated by dividing luminance values of the received
portion of the data packet that also includes baseline
data for a third color component;
HDR image With luminance values of the tone
mapped image;
Wherein compute the data packet comprises:
generate a tone-mapped image based on the received
HDR image;
generate the coded ratio image, the coded ratio image
comprising luminance ratio values generated by
5
determining the ?rst coded chroma residual image
and the second coded chroma residual image based
on the re-mapped image and the tone-mapped
dividing luminance values of the received HDR
image With luminance values of the tone-mapped
image; and
image;
sending the computed data packet in a bit stream that
conforms to the DCI Systems Standard.
11. The apparatus of claim 9, Wherein the at least three
apply the luminance ratio values to the received HDR
image to create a re-mapped image; and
determine the ?rst coded chroma residual image and
the second coded chroma residual image based on
color components comprise:
an X' color component;
a Y' color component; and
a Z' color component.
the re-mapped image and the tone-mapped image;
and
send the computed data packet in a bit stream that conforms
to the DCI Systems Standard, Wherein the HDR image is
distributed.
10. A non-transitory computer-readable storage medium
having stored thereon computer-executable instructions for:
12. The apparatus of claim 9, Wherein the coded baseline
image comprises a resolution of either 2048x1080 (2K), or
20
receiving a high dynamic range (HDR) image;
processing the received HDR image, Wherein the received
HDR image comprises (a) a coded baseline image that
complies With the Digital Cinema Systems Speci?cation
25
of the Digital Cinema Initiative (DCI Systems Stan
4096x2160 (4K).
13. The apparatus of claim 9, Wherein the third portion of
the data packet comprises an application marker that sepa
rates the baseline data for the third color component and the
coded ratio image, and Wherein the application marker con
forms to a speci?cation of the JPEG 2000 compressor/de
compressor (codec) of the Joint Photographic Experts Group
(JPEG).
dard), and (b) one or more HDR-enhancement images
are created from the received HDR image;
Wherein the coded baseline image comprises at least three
color components, Wherein the coded baseline image
applying the luminance ratio values to the received
HDR image to create a re-mapped image; and
14. The computer-readable storage medium of claim 10,
Wherein the coded baseline image comprises:
30
and the one or more HDR-enhancement images each
provide enhancement information to allow reconstruc
tion of an instance of the received HDR image using the
an X' color component;
a Y' color component; and
a Z' color component.
images comprise:
15. The computer-readable storage medium of claim 10,
Wherein the coded baseline image comprises a resolution of
either 2048x1080 (2K), or 4096x2160 (4K).
16. The computer-readable storage medium of claim 10,
Wherein the third portion of the data packet comprises an
application marker that separates the baseline data for the
third color component and the coded ratio image, and Wherein
the application marker conforms to a speci?cation of the
JPEG 2000 compressor/decompressor (codec) of the Joint
a ?rst coded chroma residual image,
a second coded chroma residual image, and
a coded ratio image,
17. The method of claim 6, Wherein the application marker
conforms to a speci?cation of the JPEG 2000 compressor/
baseline image and the HDR-enhancement images;
computing a data packet, Wherein the data packet com
35
prises:
a ?rst data set that relates to the baseline image for the at
least three color components; and
a second data set that relates to the HDR-enhancement
images, Wherein each of the HDR enhancement
40
Photographic Experts Group (JPEG).
Wherein each of the residual images is con?gured to
capture chroma information lost in creating the base
line image from the received HDR image, and
Wherein the ?rst coded chroma residual image is embed
ded Within a ?rst portion of the data packet that also
45
includes baseline data for a ?rst color component, the
50
second coded chroma residual image is embedded
Within a second portion of the data packet that also
includes baseline data for a second color component,
and the coded ratio image is embedded Within a third
portion of the data packet that also includes baseline
18. The apparatus of claim 9, Wherein the ?rst portion of
the data packet comprises an application marker that sepa
rates the baseline data for the ?rst color component and the
coded ratio image, and Wherein the application marker con
forms to a speci?cation of the JPEG 2000 compressor/de
compressor (codec) of the Joint Photographic Experts Group
(JPEG).
55
data for a third color component;
Wherein computing the data packet comprises:
generating a tone-mapped image based on the
received HDR image;
generating the coded ratio image, the coded ratio
image comprising luminance ratio values gener
decompressor (codec) of the Joint Photographic Experts
Group (JPEG).
60
19. The computer-readable storage medium of claim 10,
Wherein the ?rst portion of the data packet comprises an
application marker that separates the baseline data for the ?rst
color component and the coded ratio image, and Wherein the
application marker conforms to a speci?cation of the JPEG
2000 compressor/decompressor (codec) of the Joint Photo
graphic Experts Group (JPEG).
*
*
*
*
*
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement