Personalizing 3dtv viewing experience

US 20120287233A1
(19) United States
(12) Patent Application Publication (10) Pub. N0.: US 2012/0287233 A1
Wang et a].
(54)
(43) Pub. Date:
PERSONALIZING 3DTV VIEWING
(52)
NOV. 15, 2012
US. Cl. ................................. .. 348/42; 348/E13.003
EXPERIENCE
(76) Inventors:
Haohong Wang, San Jose, CA
(US); Glenn Adler, Redwood City,
CA (Us)
(57)
ABSTRACT
A method for personalized video depth adjustment includes
receiving a video frame, obtaining a frame depth map based
on the video frame, and determining content genre of the
video frame by classifying content of the video frame into one
or more categories. The method also includes identifying a
(21) Appl. No.:
13/519,565
(22)
PCT Filed:
Dec- 29, 2009
(86)
PCT NOJ
PCT/Us2009/069704
user vieWing the video frame, retrieving depth preference
§ 371 (0)0)’
(2)’ (4) Date:
information for the user from a user database, and deriving
depth adjustment parameters based on the content genre and
the depth preference information for the user. The method
further includes adjusting the frame depth map based on the
Jun‘ 27’ 2012
_
_
_
_
depth adjustment parameters, and providing a 3D video frame
Pubhcatlon Classl?catlon
(51)
for display at a real-time playback rate on a user device of the
Int. Cl.
user. The 3D video frame is generated based on the adjusted
H04N 13/00
(2006.01)
frame depth map.
RECEIVING VIDEO FRAMES
Q
ARE THE
VIDEO FRAMES 2D
FORMAT?
5%
V
II
GENERATING FRAME DEPTH MAPS
REcONsTRucTING FRAME DEPTH MAPS
BASED ON THE VIDEO FRAMEs
BAsED ON THE VIDEO FRAMES
50$
E
DETERMINING A CONTENT GENRE BASED ON
CONTENT OF THE VIDEO FRAMES
5E
I
PROVIDING THE FRAME DEPTH MAPS AND
THE CONTENT GENRE
512
Patent Application Publication
cop
Nov. 15, 2012 Sheet 1 0f 7
M59w5:0 3 @
vEOwZ 5
US 2012/0287233 A1
w1omSw:o wli
a
M20:
Ewszm
Mom6n5=> 236 2 §
mkzwo a
m3
Patent Application Publication
Nov. 15, 2012 Sheet 2 0f 7
E3065 ml:
00F
US 2012/0287233 Al
Patent Application Publication
00?
US 2012/0287233 A1
Patent Application Publication
Nov. 15, 2012 Sheet 4 0f 7
m2.
cow
$82A0
US 2012/0287233 A1
Patent Application Publication
Nov. 15, 2012 Sheet 5 0f 7
US 2012/0287233 A1
RECEIVING VIDEO FRAMES
Q2
ARE THE
VIDEO FRAMES 2D
FORMAT?
5%
II
GENERATING FRAME DEPTH MAPS
RECONSTRUCTING FRAME DEPTH MAPS
BASED ON THE VIDEO FRAMES
BASED ON THE VIDEO FRAMES
5%
E
DETERMINING A CONTENT GENRE BASED ON
CONTENT OF THE VIDEO F RAMEs
510
i
PROVIDING THE FRAME DEPTH MAPs AND
THE CONTENT GENRE
512
FIG. 5
Patent Application Publication
US 2012/0287233 A1
Nov. 15, 2012 Sheet 6 0f 7
START
5E
RECEIVING FRAME DEPTH MAPS AND A
CONTENT GENRE OF VIDEO FRAMES
602
I
IDENTIFYING A USER’S IDENTIFICATION
604
DOES THE USER INPUT
DEPTH ADJUSTMENT?
606
V
V
UPDATING THE USER'S DEPTH PREFERENCE
INFORMATION IN USER DATABASE BASED ON THE
RETRIEVING THE USER'S DEPTH
PREFERENCE INFORMATION FROM USER
UsER’s INPUT AND IDENTIFICATION
DATABASE BASED ON THE IDENTIFICATION
M
612
I
I
DERIvING DEPTH ADJUSTMENT PARAMETERS
DERIVING DEPTH ADJUSTMENT PARAMETERS
BASED ON THE CONTENT GENRE AND THE UsER’s
BASED ON THE CONTENT GENRE AND THE
UPDATED DEPTH PREFERENCE INFORMATION
USER'S DEPTH PREFERENCE INFORMATION
m
ADJUSTING THE FRAME DEPTH MAPS BASED ON
THE DERIVED DEPTH ADJUSTMENT PARAMETERS
61.6
I
APPLYING DEPTH-IMAGE BASED RENDERING
ALGORITHM To THE VIDEO FRAMES BASED ON THE
ADJUSTED FRAME DEPTH MAPS
618
I
PROVIDING MULTI-VIEW VIDEO FRAMES FOR 3D
DISPLAYING
620
FIG. 6
Patent Application Publication
US 2012/0287233 A1
Nov. 15, 2012 Sheet 7 0f 7
START
ZQQ
DETECTING A USER AT A USER DEVICE
m
RECOGNIZING
THE USER?
Z0_4
I
PROMPTING THE USER TO SELECT A USER
IDENTIFICATION RETRIEVED FROM USER
DATABASE
RETRIEVING THE UsER‘s
IDENTIFICATION FROM USER
DATABASE
7_1_Q
_7_0§
I
Is A
RETRIEvE THE UsER’s DEPTH PREFERENCE
INFORMATION FROM USER DATABASE BASED
UsER
IDENTIFICATION
SELECTED?
ON THE IDENTIFICATION
7Q8
m
II
PROMPTING THE USER TO ENTER OR
AUTOMATICALLY ASSIGNING A USER
IDENTIFICATION
7’I_4
I
I
PROVIDING THE USER'S DEPTH PREFERENCE
INFORMATION
720
ASSOCIATE DEFAULT DEPTH
PREFERENCE INFORMATION AT THE
UsER DEVICE WITH THE UsER
IDENTIFICATION
11g
I
STORE IN UsER DATABASE THE
DEFAULT DEPTH PREFERENCE
INFORMATION IN ASSOCIATION WITH
THE USER IDENTIFICATION
11g
FIG. 7
Nov. 15, 2012
US 2012/0287233 A1
PERSONALIZING 3DTV VIEWING
EXPERIENCE
real-time playback rate on a user device of the user. The 3D
video frame is generated based on the adjusted frame depth
map.
TECHNICAL FIELD
[0001]
The present disclosure relates to methods and sys
tems for personaliZing 3DTV viewing experience.
[0006]
Another example in accordance with the present
disclosure includes a device coupled to receive a video frame.
The device includes a depth map obtaining module to obtain
[0002] Nowadays the consumption of digital media has
been changed rapidly from the typical “TV in the living
a frame depth map based on the video frame, and a content
classi?cation module to determine content genre of the video
frame by classifying content of the video frame into one or
more categories. The device also includes a user detection
module to identify a user viewing the video frame, and an
room” to ubiquitous access. A typical home entertainment
analysis module to derive depth adjustment parameters based
system may now contain more than one TV, and the scope also
extends to PCs and mobile TV systems such as mobile
tion retrieved from a user database. The device further
BACKGROUND
phones, PDAs, and portable players. Various efforts have
been made to provide a user with capabilities to personaliZe
the multimedia content according to the user’s preferences.
Personalization enables the user to access the multimedia
content seamlessly with various devices and networks. Fur
ther, seamless user experiences can be provided despite vary
ing device and network characteristics.
[0003] An example of prior art content adaptability and
personaliZation using MPEG-7 and MPEG-21 standards is
disclosed in B. L. Tseng, C. Y. Lin and J. R. Smith, Using
MPEG-7 and MPE G-21 for Personalizing VideO, IEEE Mul
timedia, January-March 2004. MPEG-7 is a multimedia
metadata description standard to allow searching for material
that is of interest to users. MPEG-21 is a rights repressions
standard that de?nes a multimedia framework to enable trans
parent and augmented use of multimedia resources across a
range of networks and devices used by different communities.
In the MPEG-7 standard, a user preference can be described,
and in the MPEG-21 standard, a usage environment can be
speci?ed with user pro?les, terminal properties, and network
characteristics. As an example, a user agent pro?le in the
wireless access protocol (WAP) speci?es a device pro?le that
covers software and hardware platforms, browser informa
tion, and network characteristics, so that the same visual
content would be shown differently (e.g., color vs. black/
white, or high resolution vs. lower resolution) at various
mobile devices depending on the conditions of display siZe,
battery status, computational capability, and so on. The user
on the content genre and the user’s depth preference informa
includes an automatic depth adjustment module to adjust the
frame depth map based on the depth adjustment parameters,
and a rendering engine to provide a 3D video frame for
display at a real-time playback rate. The 3D video frame is
generated based on the adjusted frame depth map.
[0007] It is to be understood that both the foregoing general
description and the following detailed description are exem
plary and explanatory only and are not restrictive of the inven
tion, as claimed.
BRIEF DESCRIPTION OF THE DRAWINGS
[0008]
FIG. 1 illustrates a block diagram of an exemplary
system.
[0009] FIG. 2 is a block diagram illustrating an embodi
ment of the exemplary system of FIG. 1.
[0010] FIG. 3 is a functional diagram illustrating an exem
plary process ?ow in the embodiment of FIG. 2.
[0011] FIG. 4 illustrates an exemplary process ?ow of real
time personaliZation of 3DTV viewing experience.
[0012] FIG. 5 is a ?owchart representing an exemplary
method of frame depth map generation and video content
classi?cation.
[0013] FIG. 6 is a ?owchart representing an exemplary
method of personaliZed depth adjustment.
[0014] FIG. 7 is a ?owchart representing an exemplary
method of retrieval of user depth preference information.
preference enables ?ltering, searching, and browsing so that
DETAILED DESCRIPTION
the genre of user favorable content could be ranked and
recorded.
[0004]
The personaliZation issue for three-dimensional
television (“3DTV”) has not been well studied yet, as 3DTV
is a recent advance and deployment of 3D displays is in an
early stage.
[0015]
Reference will now be made in detail to the exem
plary embodiments, examples of which are illustrated in the
accompanying drawings. Wherever possible, the same refer
ence numbers will be used throughout the drawings to refer to
the same or like parts.
SUMMARY
[0005] An example in accordance with the present disclo
sure includes a method for personaliZed video depth adjust
ment. The method includes receiving a video frame, obtain
ing a frame depth map based on the video frame, and
determining content genre of the video frame by classifying
[0016] Exemplary embodiments disclosed herein are
directed to methods and systems for 3DTV personalization,
which dynamically adjusts depths of objects in a scene to
satisfy a user’s depth perception. The user’s perception of 3D
content is related to a depth structure of the scene, which is
also re?ected by disparity maps of a left/right view. The user’s
depth sensation might be different for different persons, con
content of the video frame into one or more categories. The
tent, display siZe, image resolution and viewing distances, but
method also includes identifying a user viewing the video
the depth sensation is constant for the same user for similar
viewing conditions. In some embodiments, a user interactive
frame, retrieving depth preference information for the user
from a user database, and deriving depth adjustment param
depth mapping algorithm can be utiliZed to help adjust a depth
eters based on the content genre and the depth preference
information for the user. The method further includes adjust
map of a scene, and a speci?ed depth can be entered via an
on-line learning mechanism to update a user database. With
ing the frame depth map based on the depth adjustment
the user database, systems disclosed herein can dynamically
adjust the depth map of a scene according to playback content
parameters, and providing a 3D video frame for display at a
Nov. 15, 2012
US 2012/0287233 A1
to satisfy the user’s preference. In some embodiments, the
user depth preference information stored in user database 1 10
disclosed systems can handle a multi-user scenario as Well.
With those stored in local user databases on user devices
[0017] FIG. 1 illustrates a block diagram of an exemplary
system 100. Exemplary system 100 can be any type of system
112-114.
[0022] User database 110 is one or more hardWare storage
devices for storing structured collections of records or data of
user depth preference information. The structured storage can
that provides video content over a local connection or a net
Work, such as a Wireless network, Internet, broadcast net
Work, etc. Exemplary system 100 can include, among other
things, 2D or 3D video content sources such as a video stor
age medium 102, a media server 104 and/or netWork 106, a
home entertainment center 108, a user database 110, and one
or more user devices 112-114. The one or more user devices,
for example, user device 112 can be connected to home enter
tainment center 108 via a netWork 107, and can have one or
more external displays 116-118. Each user device can also
have a user database.
[0018]
Video storage medium 102 can be any medium stor
ing video content. For example, video storage medium 102
can be provided as a video CD, DVD, Blu-ray disc, hard disk,
magnetic tape, ?ash memory card/drive, volatile or non-vola
tile memory, holographic data storage, and any other storage
medium. Video storage medium 102 can be located Within
home entertainment center 108, local to home entertainment
center 108, or remote from home entertainment center 108.
[0019]
Media server 104 can be a computer server that
receives a request for video content from home entertainment
center 108, processes the request, and provides video content
be organiZed as a set of queues, a structured ?le, a relational
database, an object-oriented database, or any other appropri
ate database. Computer softWare, such as a database manage
ment system, may be utiliZed to manage and provide access to
the data stored in user database 110. User database 110 may
be located Within home entertainment center 108, local to
home entertainment center 108, or remote from home enter
tainment center 108. Some of user devices 112-114 may have
their oWn user databases storing user depth preference infor
mation. User database 1 1 0 may be synchroniZed With the user
databases of user devices 112-114.
[0023]
The user depth preference information stored in user
database 110 and/or the user databases of user devices 112
114 may include, but is not limited to:
[0024] (1) Information about each user consuming the
home entertainment system. For example, the information
may include but is not limited to, the user’s identi?cation and
group members if the user is a user group consisting of one or
more individual users. The user’s identi?cation may include
one or more of face pictures, recorded voices, a name entered
by the user, and/ or other information identifying the user.
[0025] (2) Information about user devices 112-114 and
to home entertainment center 108 through, in some embodi
ments, netWork 106. For example, media server 104 can be a
Web server, an enterprise server, or any other type of computer
server. Media server 104 can be a computer programmed to
their displays. For example, the information may include, but
accept requests (e.g., HTTP, or other protocols that can ini
erences can be con?gured by the user, or automatically gen
tiate a video session) from home entertainment center 108
erated based on the information about user devices 112-114
is not limited to, display screen siZe, resolution, and other
information about the devices and displays.
[0026] (3) Each user’s depth preferences. The depth pref
and to serve home entertainment center 108 With video con
and their displays, information about video such as video
tent. Also, media server 104 can be a broadcasting facility,
content categories and video resolution, vieWing distances,
historic depth preferences, and other factors.
such as free-to-air, cable, satellite, and other broadcasting
facility, for distributing digital or non-digital video content to
home entertainment center 108 through, in some embodi
ments, netWork 106.
[0020] Networks 106 and 107 can include any combination
of Wide area netWorks (WANs), local area netWorks (LANs),
or Wireless netWorks suitable for packet-type communica
tions, such as Internet communications, or broadcast net
[0027] The depth preference information for each user may
be stored in a lookup table. In the lookup table, the user’s
depth preferences may be searched and retrieved based on the
information about the user, the information about user
devices 112-114 and their displays, the information about the
video, and/or vieWing distances, etc.
[0028]
User devices 112-114 are hardWare devices such as
Works suitable for distributing digital or non-digital video
computers, PDAs, cell phones, laptops, desktops, broadcast
content.
tuners such as standard or mobile television sets, or any
[0021] Home entertainment center 108 is a hardWare
device such as a set-top box, a computer, a PDA, a cell phone,
a laptop, a desktop, aVCR, a Laserdisc player, a DVD player,
blue ray disc player, a broadcast tuner, or any electronic
electronic devices capable of playing video. User devices
112-114 may include softWare applications that alloW the
device capable of playing video and managing content play
back for various devices. Home entertainment center 1 08 may
include softWare applications that alloW center 108 to com
municate With and receive video content from a data netWork,
e.g., netWork 106, or local video storage medium 102. Home
entertainment center 108 may, by means of included softWare
applications, transform received video content into digital
format, if not already in digital format. Home entertainment
center 108 may transmit video content to user devices 112
114. Home entertainment center 108 may also communicate
With user devices 112-114 to share user depth preference
information and update user database 110 With user pro?les
for those Who consume the home entertainment system. In
addition, home entertainment center 108 may synchroniZe
devices to communicate With and receive video content from
home entertainment center 108. The communication may be
through a data netWork e. g., netWork 107. User devices 112
114 may also include a softWare video player that alloWs the
device to play video. Examples of softWare video players
include Adobe Flash Video Player, Microsoft WindoWs
Media Player, RealPlayer, or any other player application.
Some ofuser devices 112-114 may be located local to home
entertainment center 108, or remote from home entertain
ment center 108. If there is no home entertainment center 108
and there is only one user device at home, the user device
itself can be home entertainment center 108. Further, some of
user devices 112-114 can have user databases storing user
depth preference information.
[0029]
User devices 112-114 may have different capabili
ties. A device can be a normal 2D TV playback device, so it
Nov. 15, 2012
US 2012/0287233 A1
does not have the capability to playback 3DTV content in 3D
mode. In some embodiments, a device can have powerful
capabilities. For example, the device may be capable of
240-by-l35) for accelerated background tracking and depth
map estimation. By tracking moving objects in the loWer
resolution frames, module 210 may segment objects pre
detecting and/ or recognizing a user With its digital camera or
sented in each of the loWer-resolution frames into the back
voice recognition utilities. It may also be capable of alloWing
the user’s interactions to manually con?gure or specify his/
ground and foreground objects. Next, the background and
her depth sensation preference, and dynamically adjust the
mation process.
depth sensation based on an automatic depth adjustment algo
[0036] Module 210 may generate a background depth map
based on, among other things, background structure analysis
and background depth map estimation. Various methods may
be used in the background structure analysis. For example,
such analysis may include detecting a vanishing point and
rithm. The device may even have intelligence to model/ learn
the user’s depth preference based on the user’s manually
con?gured preference and/or historic depth preference infor
mation in a user database.
foreground objects are subjected to separate depth map esti
[0030] In other embodiments, a device may not support the
user interactions to manually con?gure or specify the custom
vanishing lines of the background frame based on the seg
iZed depth preference information, but may dynamically
adjust depth sensation by obtaining instructions from the
most distant point from an observer, and the vanishing lines
represent a direction of depth increase. The vanishing lines
home entertainment center, Which ports updated user data
base and noti?es Which group of users is currently vieWing a
video program. The user detection task may be performed by
other devices in the home entertainment system.
[0031] In still other embodiments, a device may support the
user interactions to manually con?gure or specify the depth
converge at the vanishing point. A region of the background
frame having the greatest number of intersections is consid
ered to be the vanishing point, and the main straight lines
passing through or close to the vanishing point are considered
to be vanishing lines. If no vanishing point is found, a default
vanishing point, also referred to herein as a convergent point,
on top of the background frame is used as the vanishing point
and a default vanishing line is a vertical line running from top
to bottom of the background frame and passing through the
default vanishing point. Other methods knoWn to those
preference information, and may dynamically adjust the
depth sensation according to video content being played.
[0032] Some of user devices 112-114, for example, user
device 112, may have one or more displays 116-118. Displays
116-118 are display devices for presentation of video content.
For example, displays 116-118 may be provided as television
sets, computer monitors, projectors, or any other video dis
play devices. Displays 116-118 may have different screen
siZe and resolution. Displays 116-118 may be located Within
user device 112, local to user device 112, or remote from user
device 112.
[0033] FIG. 2 is a block diagram illustrating user device
112 in greater detail Within exemplary system 100. For sim
plicity, FIG. 2 only illustrates home entertainment center 108,
user database 110, user device 112, and display 116. The
illustrated con?guration of user device 112 is exemplary only,
and persons of ordinary skill in the art Will appreciate that the
various illustrated elements may be provided as discrete ele
ments or be combined, and be provided as any combination of
hardWare and softWare.
[0034] With reference to FIG. 2, user device 112 includes a
depth map generation module 210. Depth map generation
module 210 can be a softWare program and/or a hardWare
device that generates 3D video depth maps from 2D video
frames. The methods for generating the 3D video depth map
may be, for example, methods for real-time 3D video depth
mented background objects. The vanishing point represents a
skilled in the art may also be used to determine the vanishing
point and vanishing lines of the background.
[0037] Based on the information provided by background
structure analysis, a background depth map may be derived.
For example, With the detected vanishing point and the van
ishing lines, module 210 may generate a depth map of the
background accordingly. For example, module 210 may gen
erate different depth gradient planes With the vanishing point
being at the farthest distance and the vanishing lines indicat
ing the direction of receding depth. Module 210 may then
assign a depth level to every pixel on the depth gradient
planes. Module 210 may additionally perform calibration
steps, and ?nally derive the background depth map.
[0038] Also, module 210 may generate a foreground depth
map based on, among other things, foreground skeleton depth
estimation and foreground depth map estimation. Skeleton
depth estimation includes object skeletoniZation. Such skel
etoniZation may be performed by decomposing a foreground
object shape into a skeleton de?ned as connected midpoints
betWeen tWo boundary points in the horizontal direction, and
determining distances of the boundary points from the skel
map generation by background tracking and structure analy
eton in the horiZontal direction. The object boundary can be
recovered from its skeleton and distance data. The skeleton
sis. The methods may include receiving a 2D video frame
points are connected in the vertical (y-axis) direction, Which
having an original resolution, doWnscaling the decoded 2D
facilitates processing.
[0039] For foreground depth map estimation, it is assumed
that a foreground object is typically oriented vertically Within
video frame into an associated 2D video frame having a loWer
resolution, and segmenting objects present in the doWnscaled
2D video frame into background objects and foreground
objects. The methods may also include generating a back
ground depth map and a foreground depth map for the doWn
scaled 2D video frame based on the segmented background
and foreground objects, and deriving a frame depth map in the
original resolution based on the background depth map and
the foreground depth map.
[0035] For example, depth map generation module 210
may receive 2D video frames in original resolution (for
example, 640-by-480), and doWnscale the 2D video frames
into an associated set of loWer-resolution frames (for example
a scene, so that frontal skeleton points of the object have the
same depth as a bottom point of the skeleton. To reduce
computational complexity, module 210 may obtain the skel
eton by scanning the foreground object and ?nding a middle
point of the horiZontal scan-line segment Within the object.
The bottom point of the skeleton is on the boundary of the
foreground and background, and its depth Was previously
determined. Thus, module 210 may determine the depth of
the bottom point of the skeleton based on the depth of its
neighboring background, and determine the depth for all
skeleton points because they have the same depth. Also, the
Nov. 15, 2012
US 2012/0287233 A1
depth of boundary points of the foreground object may be
readily determined because the boundary points share the
same depth With their neighboring background. The depth of
the boundary points may be adjusted for a better 3D effect.
[0040] For each horizontal scan-line segment in the fore
video frames into different categories. A user preference of
the depth sensation may be highly correlated to the genre of
the video content that the user is vieWing. Therefore it may be
useful for module 230 to automatically classify the video
content into categories so that the user preference can be
modeled progressively in response to more user interactions
ground object, With the depth for both the skeleton point (the
middle point) and the boundary points having been deter
mined, module 210 may interpolate internal points (betWeen
that personaliZe. For example, the content can be classi?ed
the skeleton point and the boundary points) on the scan-line
neWs, and so on. A speci?c program, for example, a sports
program, can be further analyZed and broken doWn into
segment With a Gaussian distribution function. For each inter
according to program type, such as drama, Wildlife, sports,
semantic meaningful shots by grouping video frames into
nal point, tWo Weights can be generated from the Gaussian
function depending on the distances from the internal point to
the skeleton point and to the boundary points. Module 210
may then derive the depth for the internal point through a
and so on, may be used to further classify the content into
non-linear interpolation process. Using this approach, the
additional categories.
foreground thickness effect is enhanced to further strengthen
the 3D depth effect. Based on the determined points and
depths, module 210 may generate the foreground depth map.
[0041] Further, module 210 may derive a frame depth map
shots such as strokes in tennis video program. After that,
loW-level features, such as motion, color, human face, texture,
for each video frame by fusing background and foreground
[0044] Optionally, user device 112 may utiliZe a user detec
tion module 240 for user detection and/ or identi?cation. User
detection module 240 is a hardWare device having a softWare
program to detect and/ or identify a user currently vieWing a
video program played on user device 112. The detection can
depth maps in the original resolution. Module 210 may fuse
be based on, for example, an image of the user’s face, the
the foreground and background depth maps in the original
resolution and re?nes the depth continuity for the original
user’s voices, the user’s interactions With user device 112, or
other mechanisms. The softWare program at module 240 may
identify the user based on vision-based face detection and
resolution image. The frame depth map may be derived
through an interpolation ?ltering process based on desired
recognition, speech recognition, or other algorithms. Also,
computational complexity. A variety of choices for interpo
user detection module 240 may receive the user’s remote
lation may be used. For example, When implementing one
solution to duplicate depths in the doWn-scaled map to result
in an upscaled depth map having a higher resolution, a linear
interpolation may be chosen to use a Weighted average depth
value from its neighboring pixels in the same scan-line to ?ll
controller inputs, keypad inputs, or other interactions to
detect and/or identify the user.
[0045] If module 240 identi?es the user, it may retrieve the
user’s identi?cation. If module 240 does not identify the user,
it can create a neW identi?cation based on an image of the
these positions in the upscaled depth map. More complicated
user’s face, the user’s voice, and/or the user’s interactions
?lters such as bilinear or bicubic interpolation solutions may
also be used. To achieve a better effect for a currently pro
cessed frame, module 210 may retrieve more than one neigh
With user device 112. In some embodiments, if user device
112 does not include module 240 or module 240 fails to
identify the user, a default user can be identi?ed as the current
boring 2D video frames in original resolution and their cor
vieWer. The default user can be, for example, the one using
responding depth maps.
user device 112 most often. Further, user detection module
240 can be locatedWithin user device 112, local to user device
112, or remote from user device 112.
[0042]
A depth map reconstruction module 220 can be
provided as a softWare program and/ or a hardWare device to
reconstruct or recover frame depth maps of the 3D video
[0046]
frames. Any disclosed depth map reconstruction method can
be utiliZed by module 220. For example, depth map recon
struction may involve computational stereo for determining a
provided as a softWare program and/ or a hardWare device to
3D structure of a scene from tWo or more images taken from
A manual depth adjustment module 250 may be
provide a user interface for the user to manually con?gure,
such as by inputting or selecting, his/her depth preferences.
The manually con?gured depth preferences may be associ
distinct vieWpoints. A single 3D physical location projects to
ated With the user’s identi?cation and provided for depth
a unique pair of image locations in tWo observing cameras. As
a result, given tWo camera images, if it is possible to locate the
image locations that correspond to the same physical point in
space, then it is possible to determine its three-dimensional
location. Computational stereo may include calibration, cor
respondence, and reconstruction processes. The calibration
adjustment, and may also be stored in user database 270.
process is for determining camera external geometry such as
relative positions and orientations of each camera, and cam
era internal geometry such as focal lengths, optical centers,
and lens distortions. The correspondence process is for deter
mining the locations in each camera image that are the pro
jection of the same physical point in space. The reconstruc
tion process is for determining 3D structure from a dense
disparity map based on knoWn camera geometry by matching
Depth adjustment is further described beloW.
[0047]
A user preference analysis module 260 may be pro
vided as a softWare program and/or a hardWare device to
derive depth adjustment parameters based on the user’s his
toric depth preferences, information about user device 112
and display 116, the user’s manually con?gured depth pref
erences, the content genre of the video content, vieWing dis
tances, and/or other information. After the user makes a ?nal
con?guration of the preferred depth adjustment, module 260
utiliZes a learning mechanism to study the user’s inputs based
on, for example, but not limited to, one or more of content
information such as content category and content rendering
resolution being currently vieWed, current user vieWing the
pixels in one image With their corresponding pixels in the
content, information about display 116 such as screen siZe
other image.
and resolution, and normaliZed translation and scaling
[0043]
parameters for depth adjustment.
A content classi?cation module 230 can be provided
as a softWare program and/or a hardWare device to receive
video frames and de?ne a content genre by classifying the
[0048] User preference analysis module 260 may model the
depth adjustment parameters With a mixture Gaussian model
Nov. 15, 2012
US 2012/0287233 A1
for each vector of content/user/display settings. Module 260
can model intensity values of each vector as a mixture of
Gaussian distributions. In such case, each vector intensity is
represented by a mixture of K (K is a pre-de?ned constant
value) Gaussian distributions, and each Gaussian distribution
is Weighted according to the frequency With Which it repre
sents a certain cluster of parameters. Based on comparisons
betWeen distances from a current vector intensity value to
3D representation based on images of an object and corre
sponding depth maps. To achieve a better 3D effect for a
currently processed frame, depth-image rendering engine
290 may utiliZe one or more neighboring video frames and
their adjusted depth maps.
[0053] DIBR algorithms may include 3D image Warping.
3D image Warping changes vieW direction and vieWpoint of
an object, and transforms pixels in a reference image of the
means of the mo st in?uential Gaussian distributions and asso
object to a destination vieW in a 3D environment based on
ciated thresholds that are highly correlated to the standard
deviations of Gaussian distributions, module 260 can deter
mine to Which cluster of parameters the vector of content/
depth values of the pixels. A function can be used to map
pixels from the reference image to the destination vieW.
Depth-image rendering engine 290 may adjust and recon
user/ display settings corresponds.
struct the destination vieW to achieve a better effect.
[0049] User preference analysis module 260 may also uti
liZe normaliZed translation and scaling parameters for depth
[0054] DIBR algorithms may also include plenoptic image
modeling. Plenoptic image modeling provides 3D scene
adjustment. The normalized translation can be a function
information of an image visible from arbitrary vieWpoints.
mapping one depth range to another depth range. For
example, to adjust a depth range [10, 100] to [0, 200], a
scaling function (to map the range distance from 90 to 200)
plus a translation function (to map the starting point from 10
to 0) can be applied to achieve the desired result. User pref
The 3D scene information may be obtained by a function
based on a set of reference images With depth information.
These reference images are Warped and combined to form 3D
representations of the scene from a particular vieWpoint. For
an improved effect, depth-image rendering engine 290 may
erence analysis module 260 may maintain a lookup table in a
adjust and reconstruct the 3D scene information. Base on the
user database 270 for searching current depth adjustment
parameters based on content/user/di splay settings. Automatic
depth adjustment can be conducted With the depth adjustment
may generate multi-vieW video frames for 3D displaying.
parameters.
plary process How for personaliZing 3DTV vieWing experi
[0050] User device 112 may optionally include a user data
base 270 for storing a structured collection of records or data
of users’ depth preference information. The structured stor
ences in exemplary system 100. It Will noW be appreciated by
one of ordinary skill in the art that the illustrated process How
can be altered to delete steps, change the order of steps, or
age can be organiZed as a set of queues, a structured ?le, a
include additional steps.
relational database, an object-oriented database, or any other
appropriate database. Computer softWare, such as a database
management system, may be utiliZed to manage and provide
access to the data stored in user database 270. User database
270 may be located Within user device 112, local to user
device 112, or remote from user device 112. User database
270 may be synchronized With user database 110 through
home entertainment center 108.
[0051]
An automatic depth adjustment module 280 may be
provided as a softWare program and/ or a hardWare device to
execute depth adjustment by changing frame depth maps for
a current scene, Which may include one or more video frames.
During depth adjustment, Whether manual or automatic, the
scene depth structure may be maintained. Module 280, as
Well as module 250, may not change the depth order of the
objects in the video frames. A user, through module 250,
and/ or module 280, may not micro-manage the scene by
moving single objects in each frame because the task Would
become impractical for a typical movie With more than
3D scene information, depth-image rendering engine 290
[0055]
FIG. 3 is a functional diagram illustrating an exem
[0056] After receiving (302), e.g., through network 107,
video frames from home entertainment center 108, user
device 112 can direct the video frames to different modules,
depending on the format of the video frames. Each video
frame can include a unique identi?er (frame ID) for later
retrieval and association purpose. In some embodiments, the
video frames can be stored in a storage for later processing.
[0057] If the video frames are in 2D format, user device 112
may pass (not shoWn) the video frames to depth map genera
tion module 210 to generate frame depth maps. After that,
module 210 can transfer (304) the frame depth maps along
With the associated video frames to content classi?cation
module 230.
[0058] If the video frames are in 3D format, user device 112
may pass (not shoWn) the video frames to depth map recon
struction module 220 to reconstruct or recover frame depth
maps. After that, module 220 can transfer (306) the frame
depth maps along With the associated video frames to content
classi?cation module 230.
15,000 frames. A depth map adjustment strategy disclosed
[0059] Alternatively, the depth map generation/reconstruc
herein is to map an object depth range in the scene to a neW
tion and the content classi?cation may be performed in a
parallel manner. For example, user device 112 can transfer the
video frames to module 210 or 220 for depth map generation
or reconstruction, and to module 230 for content classi?ca
tion. The generated or reconstructed frame depth maps are
range With linear or non-linear mapping functions. For
example, changing the depth range from [0, 1.0] to [0.2, 2.0]
can push objects farther and increase the depth distances
among objects. The mapping functions can strongly in?uence
depth distances among objects as Well. Non-linear functions
(for the user’s manual adjustment) can achieve uneven depth
distances among objects and thus a predictable and control
lable effect.
associated With the corresponding video frames. The associa
tion may be based on the frame IDs. In some embodiments,
the generated or reconstructed frame depth maps may be
stored in association With the video frames in a storage for
Ware program and/or a hardWare device that receives adjusted
later processing.
[0060] After receiving (3 02, 304, or 306) the video frames,
frame depth maps and video frames and applies depth-image
based rendering (“DIBR”) algorithms to generate multi-vieW
content classi?cation module 230 may determine content
genre based on content classi?cation. The content genre is
video frames for 3D display. DIBR algorithms can produce a
associated With the video frames. The association may be
[0052]
A depth-image rendering engine 290 may be a soft
Nov. 15, 2012
US 2012/0287233 A1
based on the frame IDs. In some embodiments, the content
genre may be stored in association With the video frames in a
[0066] Based on the adjusted frame depth map and the
corresponding video frames received (320) or retrieved from
storage for later processing. Content classi?cation module
230 provides (308) the content genre for further processing.
a storage, depth-image rendering engine 290 applies DIBR
algorithms to generate multi-vieW (3D) video frames With
[0061] If a user detection module 240 is available, module
240 may detect a user currently vieWing video program, iden
tify the user, and/ or obtain the user identi?cation. User detec
tion module 240 may query user database 270 to identify the
user and determine the user identi?cation, and provide (312)
adjusted 3D effects, as described above. To achieve a desired
the user identi?cation to module 250. In some embodiments,
module 240 may not be available to user device 112 but may
be available to home entertainment center 108. In that case,
module 240 may query user database 110 to identify the user
and determine the user identi?cation. Home entertainment
center 108 may then send (302) the user identi?cation to user
device 112.
[0062] The identi?ed user may specify his/her depth pref
erences about the video frames being vieWed, through manual
depth adjustment module 250. Module 250 may retrieve
(314) his/her historic depth preferences and provide them to
the user for selection or modi?cation. Based on video content
such as genre and resolution, information about user device
112 and display 116 such as screen siZe and resolution, and
historic personal depth preferences, the user may manually
input depth preferences or select from one of historic depth
preferences. Module 250 may also provide linear or non
linear depth mapping functions for the user to map a depth
range to another depth range. After the user inputs or selects
depth preference, module 250 provides (316) the user’s inputs
or selection to module 260.Also, module 250 may store (314)
the user’s inputs or selection in association With the user
identi?cation in user database 270.
[0063] User preference analysis module 260 derives depth
adjustment parameters based on information provided by
modules 210, 220, 230, 250, and user database 270. Module
260 may retrieve (318) from user database 270 the user’s
historic depth preferences and information about user device
112 and display 116. Module 260 may also receive (308) from
modules 210, 220, and/or 230 the video frames, frame depth
maps, and video content information such as content genre
and resolution. Further, module 260 may receive (316) from
module 250 the user’s manually entered or selected depth
preferences. Alternatively, the information from other mod
ules may be stored in one or more storages and module 260
may retrieve the information from the storages. After having
derived the depth adjustment parameters, module 260 may
provide (320) the parameters to module 280. In addition,
module 260 may store (318) the derived depth adjustment
3D effect for a currently processed frame, depth-image ren
dering engine 290 may adjust the 3D video frame based on
one or more neighboring video frames and their correspond
ing adjusted depth maps. Depth-image rendering engine 290
provides (324) the generated video frames to display 116 for
3D displaying.
[0067]
The systems and methods disclosed herein can also
handle multi-user scenario. In some embodiments, a user may
be a user group including one or more individual vieWers. If
the user is a user group, modules involving the user’s infor
mation, such as modules 240, 250, 260, 270, and 280 and
database 270, Work in similar Ways to that for an individual
vieWer. Information for the user group can be retrieved, pro
cessed, and stored in similar Ways to that for an individual
vieWer.
[0068]
Moreover, an individual vieWer’s depth preference
information can be obtained based on the group’s depth pref
erence information. The basic assumption is that a ?nal depth
inputs/ selection for a user group Would be tolerable for all
vieWers in the group, but this inputs/selection Would be
counted in a loW-Weighted manner in the training process as
it may not re?ect the best choice for each vieWer in the user
group. The user group’s inputs can be mainly valuable for a
member user Who has feW statistics, e.g., historic depth pref
erences, in user database 270. This user may choose not to
manually con?gure his/her depth preferences. The user
group’s inputs/selection may be the only information avail
able to determine this user’s depth preferences.
[0069] Furthermore, this user may be a member of several
user groups, and this user’s depth preferences may be
obtained by, for example, a Weighted sum based on each user
group’s depth preferences and the user’s participation in
determining the group’s depth preferences, because the sta
tistics shoW that this user is not so sensitive to the depth
sensation. For example, user A may be a member of group I
consisting of users A, B, and C, a member of group II con
sisting of users A, D, and E, and a member of group III
consisting of users A and F. UserA’s depth preferences can be
obtained by a Weighted sum, Which can be determined, for
example, by summing group I’s depth preferences times 1/3
(userA equally participated in determining the group’s depth
the user’s manually con?gured depth preferences and/ or the
preference With other group members), group II’s depth pref
erences times 1/3 (user A equally participated in determining
the group’s depth preference With other group members), and
group III’s depth preferences times 1/3 (userA did not actively
participate in determining the group’s depth preference as the
derived depth adjustment parameters.
other group member did).
parameters in association With the user identi?cation in user
database 270. Further, module 280 may update (318) the user
depth preference information in user database 270 based on
[0064] In some embodiments, user device 112 does not
include user database 270. In that case, user device 112 may
obtain/ store (302) the user depth preference information
from/in user database 11 0 through home entertainment center
108.
[0065] After receiving (320) the depth adjustment param
eters, automatic depth adjustment module 280 applies the
parameters to the generated/reconstructed frame depth maps
[0070]
FIG. 4 illustrates an exemplary process How 400 of
personaliZing 3DTV vieWing experiences. It Will noW be
appreciated by one of ordinary skill in the art that the illus
trated process can be altered to delete steps, change the order
of steps, or include additional steps.
[0071] Incoming video content received by a user device
could be either in a 2D format (402) or in a 3D format (404,
406, and 408). For the former case, the user device can per
of the video frames to generate adjusted frame depth maps.
form depth map estimation (410) to generate frame depth
Then, module 280 provides (322) the adjusted frame depth
maps to depth-image rendering engine 290.
maps during a 2D-to-3D conversion process. For the latter
case, a depth map reconstruction process (412) is called. In
Nov. 15, 2012
US 2012/0287233 Al
(414) by the content classi?cation module 230 to de?ne its
appreciated by one of ordinary skill in the art that the illus
trated procedure can be altered to delete steps, change the
genre.
order of steps, or include additional steps. After an initial start
the meantime, the 2D or 3D video content can be processed
[0072]
The viewer information, such as a user oruser group
identi?cation, can be obtained through user detection (416).
For example, the viewer information can be obtained either
by a video camera for automatic detection, or by the user or
user group’s inputs. Otherwise, a default user or user group,
for example, the one with the greatest frequency of using the
system, can be identi?ed as the current viewer.
[0073]
A user preference analysis process (418) determines
the best setting for the current viewer according to content
genre and resolution, viewing condition such as display siZe
and resolution retrieved from user database 270, view dis
tances, and other information. The user device may then
step 600, user preference analysis module 260 of user device
112 receives (602) video frames, generated or reconstructed
frame depth maps, and a content genre of the video frames.
[0080] If available, user detection module 230 of user
device 112 detects and/or identi?es (604) a user or user group
currently viewing the video frames, and retrieves (604) the
user or user group’s identi?cation from user database 270 of
user device 112. In some embodiments, module 230 may be
not available to user device 112, and user device 112 may be
able to obtain the user or user group’s identi?cation from user
database 110 through home entertainment center 108. In
perform automatic depth adjustment (420) to execute the
other embodiments, if user database 270 is not available or
setting of depth preference by changing the frame depth maps
module 230 cannot recogniZe the user or user group, a default
user or user group’s identi?cation can be used.
for current scene, which may include one or more video
frames.
[0081]
[0074] On the other hand, when the user or user group
decides to intervene the depth adjustment process, the user or
(606) whether the user or user group manually inputs depth
user group can use a user interface to perform manual depth
adjustment (422) to specify a desired depth sensation. The
user device may perform user preference analysis on the user
or user group’s request, update the user or user group’s depth
preferences with a learning mechanism, and store the updated
depth preferences in user database 270. In addition, the user
device may apply the automatic depth adjustment (420) based
on the user or user group’s request.
[0075] Finally, the user device may apply a depth-image
based rendering algorithm (424) to the video scene based on
the adjusted frame depth maps, and render the 3D scene into
a number of views for displaying (116).
[0076] FIG. 5 is a ?owchart representing an exemplary
method of frame depth map generation and video content
classi?cation. It will now be appreciated by one of ordinary
skill in the art that the illustrated procedure can be altered to
delete steps, change the order of steps, or include additional
steps. After an initial start step 500, user device 112 receives
(502) one or more video frames from, for example, home
Manual depth adjustment module 250 determines
adjustment information. If yes (606-yes), user preference
analysis module 260 of user device 112 updates (608) the user
or user group’s depth preference information in user database
270 of user device 112 based on the manual inputs. At this
point in the method an optional veri?cation can be performed.
A user may adjust depth, try it for a while, and not be satis?ed.
Module 260 may update the user or user group’s depth pref
erence information in user database 270 if the user is satis?ed
with the perceived depth after viewing either for some period
of time without modifying the adjustment, or via interactive
veri?cation. Then, module 260 can utiliZe a learning mecha
nism to derive (610) depth adjustment parameters based on
the content genre, the video frame’s resolution, the user or
user group’s updated depth preference information, and other
information such as a viewing distance. The user or user
group’s updated depth preference information can include,
but is not limited to, for example, user device 112’s con?gu
rations, display screen siZe and resolution, the user or user
group’s manually inputted depth adjustment information, the
entertainment center 108. Then, user device 112 determines
(504) whether the video frames are in a 2D format or a 3D
format.
[0077] If the video frames are in a 2D format (504-yes),
user or user group’s historic depth preferences based on con
depth map generation module 210 of user device 112 gener
ates (506) frame depth maps based on the video frames. If the
analysis module 260 of user device 112 can retrieve (612) the
user or user group’s depth preference information from user
video frames are in a 3D format (504-no), depth map recon
struction module 220 of user device 112 reconstructs (508)
frame depth maps based on the video frames.
database 270 of user device 112 based on the identi?cation. If
user device 112 does not have user database 270, user device
112 can obtain the user or user group’s depth preference
[0078]
information from user database 110 through home entertain
ment center 108. Module 260 can derive (614) depth adjust
After receiving (502) the video frames, content clas
si?cation module 240 of user device 112 determines (510) a
content genre based on content classi?cation of the video
frames. The content genre can include one or more content
categories based on one or more levels of classi?cation. Alter
natively, steps 506 and 510 or steps 508 and 510 can be
performed in a parallel manner. Also, the received video
frames, the generated or reconstructed frame depth maps, and
the content genre can be stored in one or more storages for
later processing and retrieval. User device 112 provides (512)
the generated or reconstructed frame depth maps and the
content genre for further processing. The method then ends
(514).
[0079] FIG. 6 is a ?owchart representing an exemplary
method of personaliZed depth adjustment. It will now be
tent/user/display settings, and so on.
[0082] If the user or user group does not manually input
depth adjustment information (606-no), user preference
ment parameters based on the content genre, the video
frame’s resolution, the user or user group’s depth preference
information, and other information such as a viewing dis
tance. The user or user group’s depth preference information
can include but not limited to, for example, user device 112’s
con?gurations, display screen siZe and resolution, the user or
user group’s historic depth preferences based on content/user/
display settings, and so on.
[0083] Automatic depth adjustment module 280 of user
device 112 adjusts (616) the frame depth maps of the video
frames based on the derived depth adjustment parameters.
Then, depth-image rendering engine 290 of user device 112
can apply (618) depth image based rendering algorithms to
Nov. 15, 2012
US 2012/0287233 Al
the video frames based on the adjusted frame depth maps, and
provide (620) multi-vieW video frames for 3D displaying.
The method then ends (622).
[0084] FIG. 7 is a ?owchart representing an exemplary
method of retrieval of user depth preference information. It
Will noW be appreciated by one of ordinary skill in the art that
the illustrated procedure can be altered to delete steps, change
the order of steps, or include additional steps. After initial
start step 700, user detection module 240 of user device 112
detects (702) a user or user group Who is vieWing video
frames at user device 112.
[0085] If recogniZing the user or user group (704-yes), user
detection module 240 can retrieve (706) the user or user
group’s identi?cation from user database 270 of user device
112. Based on the user or user group’s identi?cation, user
preference analysis module 260 of user device 112 can
retrieve (708) the user or user group’s depth preference infor
mation from user database 270 of user device 112.
[0086]
If not recognizing the user or user group (704-no),
grammable processor, a computer, or multiple computers. A
computer program can be Written in any form of program
ming language, including compiled or interpreted languages,
and it can be deployed in any form, including as a standalone
program or as a module, component, subroutine, or other unit
suitable for use in a computing environment. A computer
program can be deployed to be executed on one computer or
on multiple computers at one site or distributed across mul
tiple sites and interconnected by a communication netWork.
[0092] A portion or all of the methods disclosed herein may
also be implemented by an application speci?c integrated
circuit (ASIC), a ?eld-programmable gate array (FPGA), a
complex programmable logic device (CPLD), a printed cir
cuit board (PCB), a digital signal processor (DSP), a combi
nation of programmable logic components and program
mable interconnects, a single central processing unit (CPU)
chip, a CPU chip combined on a motherboard, a general
purpose computer, or any other combination of devices or
modules capable of performing personaliZed depth adjust
user detection module 240 of user device 112 can prompt
ment disclosed herein.
(710) the user or user group to select a user identi?cation
[0093] In the preceding speci?cation, the invention has
been described With reference to speci?c exemplary embodi
ments. It Will, hoWever, be evident that various modi?cations
and changes may be made Without departing from the broader
retrieved from user database 270 of user device 112. If an
identi?cation is selected (712-yes), user preference analysis
module 260 of user device 112 can retrieve (708) the user or
user group ’s depth preference information from user database
spirit and scope of the invention as set forth in the claims that
270 of user device 112.
If an identi?cation is not selected (712-no), it may
folloW. The speci?cation and draWings are accordingly to be
regarded as illustrative rather than restrictive. Other embodi
be the ?rst time for the user or user group to use user device
ments of the invention may be apparent to those skilled in the
[0087]
112. User detection module 240 can prompt (714) the user or
art from consideration of the speci?cation and practice of the
user group to enter an identi?cation, or module 240 can
invention disclosed herein.
automatically assign (714) an identi?cation based on the user
or user group’s face pictures, voices, and/or other informa
tion. Module 240 can also treat the user or user group as a
default user or user group Who uses user device 112 most
often, and assign (714) the user or user group a default iden
ti?cation. User preference analysis module 260 of user device
112 can associate (716) default or generally accepted depth
preference information With the user or user group’s identi
?cation, and store (718) in user database 270 of user device
112 the default depth preference information in association
With the user or user group’s assigned identi?cation.
[0088] In some embodiments, user database 270 may not be
available to user device 112. In such embodiments, user
device 112 may obtain information from and/ or store infor
mation in user database 110 through home entertainment
center 108. In other embodiments, user detection 240 may be
available to home entertainment center 108 but not available
to user device 112. In such embodiments, user device 112
may obtain the user or user group’s identi?cation and depth
preference information from user database 1 10 through home
entertainment center 108.
[0089] User preference analysis module 260 of user device
112 provides (720) the user or user group’s depth preference
information for further processing. The method then ends
(722).
[0090] In some embodiments, a portion or all of the meth
ods disclosed herein may also be performed by a device that
is different from user device 112, and is located local to or
remote from user device 112.
What is claimed is:
1. A computer-implemented method for personaliZed
video depth adjustment, comprising:
receiving a video frame;
obtaining a frame depth map based on the video frame;
determining content genre of the video frame by classify
ing content of the video frame into one or more catego
ries;
identifying a user vieWing the video frame;
retrieving depth preference information for the user from a
user database;
deriving depth adjustment parameters based on the content
genre and the depth preference information for the user;
adjusting the frame depth map based on the depth adjust
ment parameters; and
providing a 3D video frame for display at a real-time play
back rate on a user device of the user, Wherein the 3D
video frame is generated based on the adjusted frame
depth map.
2. The method of claim 1, Wherein the obtaining the frame
depth map comprises:
generating, if the video frame is in a 2D format, the frame
depth map from the 2D video frame; and
reconstructing, if the video frame is in a 3D format, the
frame depth map from the 3D video frame.
3. The method of claim 1, further comprising determining
the depth preference information for the user, Which includes:
identifying information about the user, Which includes an
The methods disclosed herein may be implemented
identi?cation of the user and group members if the user
as a computer program product, i.e., a computer program
tangibly embodied in an information carrier, e.g., in a
machine readable storage device, for execution by, or to con
identifying information about the user device, Which
includes display screen siZe and/or resolution; and
trol the operation of, data processing apparatus, e.g., a pro
identifying depth preferences for the user,
[0091]
is a user group having one or more individual users;
Nov. 15, 2012
US 2012/0287233 A1
wherein the information about the user device and the
depth preferences for the user are associated With the
information about the user.
4. The method of claim 1, Wherein the identifying the user
comprises:
recognizing the user based on a user identi?cation distin
guishing the user from other users Who have used the
user device to vieW a video program,
Wherein the user identi?cation includes one or more of an
image of a face of the user, a voice of the user, and/or a
name of the user inputted by the user.
5. The method of claim 1, Wherein the identifying the user
comprises:
treating the user as a default user Who uses the user device
most often, if the user cannot be identi?ed.
6. The method of claim 1, Wherein the identifying the user
comprises:
identifying the user as a group of one or more individual
users.
7. The method of claim 6, Wherein the user is a member of
one or more user groups each including one or more indi
vidual users, the identifying the user comprising:
identifying the one or more user groups to Which the user
belongs.
8. The method of claim 6, Wherein the user is a member of
one or more user groups each including one or more indi
vidual users, the retrieving the depth preference information
comprises:
retrieving depth preference information for the one or more
user groups; and
obtaining the depth preference information for the user
based on the depth preference information for the one or
more user groups.
9. The method of claim 1, further comprising:
providing a user interface for the user to manually con?g
ure a depth preference; and
updating the depth preference information for the user in
the user database based on the manually con?gured
depth preference, after the user has veri?ed satisfaction
With the manually con?gured depth preference.
10. The method of claim 9, further comprising:
deriving the depth adjustment parameters based on the
content genre and the manually con?gured depth pref
erence.
11. The method of claim 1, further comprising:
updating the depth preference information for the user in
the user database based on the depth adjustment param
eters.
12. A device coupled to receive a video frame, the device
comprising:
a depth map obtaining module to obtain a frame depth map
based on the video frame;
a content classi?cation module to determine content genre
of the video frame by classifying content of the video
frame into one or more categories;
a user detection module to identify a user vieWing the video
frame;
an analysis module to derive depth adjustment parameters
based on the content genre and the user’s depth prefer
ence information retrieved from a user database;
an automatic depth adjustment module to adjust the frame
depth map based on the depth adjustment parameters;
and
a rendering engine to provide a 3D video frame for display
at a real-time playback rate, Wherein the 3D video frame
is generated based on the adjusted frame depth map.
13. The device of claim 12, Wherein the depth map obtain
ing module comprises:
a depth map generation module to generate, if the video
frame is in a 2D format, the frame depth map from the
2D video frame; and
a depth map reconstruction module to reconstruct, if the
video frame is in a 3D format, the frame depth map from
the 3D video frame.
14. The device of claim 12, Wherein the user detection
module comprises one or more of:
a vision-based face detection and recognition module to
detect and recogniZe the userbased on an image of a face
of the user;
a speech detection and recognition module to detect and
recogniZe the user based on a voice of the user; and
a manual input module to accept manual inputs of the user
through a remote controller or a keypad and to recogniZe
the user based on the manual inputs.
15. The device of claim 12, Wherein the user detection
module is con?gured to:
identify the user as a user group including one or more
individual users.
16. The device of claim 12, Wherein the user detection
module is con?gured to:
identify one or more user groups to Which the userbelongs,
Wherein the user is a member of the one or more user
groups each including one or more individual users.
17. The device of claim 12, Wherein the analysis module is
con?gured to:
retrieve the depth preference information for one or more
user groups, Wherein the user is a member of the one or
more user groups each including one or more individual
users; and
obtain the depth preference information for the user based
on the depth preference information for the one or more
user groups.
18. The device of claim 12, Wherein the user database is
con?gured to store the depth preference information for the
user, and
the depth preference information for the user includes
identi?cation of the user and group members if the user
is a user group, display information including at least of
display screen siZe and resolution, and historic depth
preferences manually con?gured by the user or auto
matically generated.
19. The device of claim 12, further comprising:
a manual depth adjustment module to provide a user inter
face for the user to manually con?gure a depth prefer
ence.
20. The device of claim 19, Wherein the manual depth
adjustment module is con?gured to:
update the depth preference information for the user in the
user database based on the manually con?gured depth
preference, after the user has veri?ed satisfaction With
the manually con?gured depth preference.
21. The device of claim 20, Wherein the analysis module is
con?gured to:
derive the depth adjustment parameters based on the con
tent genre and the updated depth preference informa
tion.
Nov. 15, 2012
US 2012/0287233 Al
10
22. The device of claim 12, Wherein the analysis module is
con?gured to:
update the depth preference information for the user in the
identifying a user vieWing the video frame;
retrieving depth preference information for the user from a
user database based on the depth adjustment parameters.
deriving depth adjustment parameters based on the content
genre and the depth preference information for the user;
adjusting the frame depth map based on the depth adjust
23. A computer readable medium storing instructions that,
When executed, cause a computer to perform a method for
personaliZed video depth adjustment, the method compris
ing:
receiving a video frame;
obtaining a frame depth map based on the video frame;
determining content genre of the video frame by classify
ing content of the video frame into one or more catego
nes;
user database;
ment parameters; and
providing a 3D video frame for display at a real-time play
back rate on a user device of the user, Wherein the 3D
video frame is generated based on the adjusted frame
depth map.
Download PDF