Digital DECtalk Software, J300 Video and Audio Adapters J300 Family, DECsafe Available Server Environment DECsafe Available Server, Parasight Technical Journal
Below you will find brief information for DECtalk Software, J300 Video and Audio Adapters, DECsafe Available Server Environment, Parasight. DECtalk Software provides natural-sounding, highly intelligible text-to-speech synthesis. It is available for the Digital UNIX operating system on Digital's Alpha-based platforms and for Microsoft's Windows NT operating system on both Alpha and Intel processors. The J300 Family of Video and Audio Adapters offer a versatile color-map rendering system, and the subsystem design decisions made to keep product costs low. The DECsafe Available Server Environment provides high availability for applications running on Digital UNIX systems. It describes the ASE design for detection and dynamic reconfiguration around host, storage device, and network failures, and key design trade-offs that favored software reliability and data integrity. Parasight is Encore Computer Corporation's new real-time debugging tool. It contrasts Encore Computer Corporation's set of debug and analysis tools for real-time applications with conventional UNIX tools.
Advertisement
Advertisement
Digital
Technical
Journal
I
AU D IO AND VIDEO T ECHNOLOG IES
UNIX AVAILABLE SERVERS,
REAL- T I M E DEBUGG ING TOOLS
Volume 7 Number
I <)<)5
4
Editorial jane C. BLlkc, JVLl1L1ging Editor
Kathleen M. Stetson, Editm
Helen
L.
Patterson, Editor
Circulation
Cnherine M. Phillips, Administrator
Dorothea B. C1ssady, Secretary
Production
Terri Autieri, Pmduction E d ito r
AnneS. Kat1.dl, ·ryrogpphcr
Peter R.
\Voodbury, Illustrator
Advisory Board
Samuel H. fuller, Chairman
Richard W. Beane
Donald Z
Harbert
William R. Hawc
Richard
J.
1-!ollingswmth
Willic1m A. L1ing
Richa1·d
Alan
F l
.ary
G.
Ncmerh
!'au line A. Nist
Robert M. Supnik
Cover Design
The concept t(Jr the cmn grap hic is dcri 1 eel fmm the Video Odvsse\· scr ee n S cl
\
" cr arpli cation, which allows uscr.s to displav fu\1rnorion ,·ideo images var on their screens in a i etv of modes. The screen saver appl i c a
tion wcJS built to test rhat p ress o ll'cJrc i
; a s o ft11 the h111ction:Jiitv s and renderers i n t o
· c r rc cJrchitecturc
�1 o fs
· id e
�mit�·ing o com soft m cr
LK e
. 'This soltwarc-onlv approach to digit.1l 1ideo is one o f the to p ic s in the teature section Audio :1nd
Video Technol ogies in this issue.
The cover wc 1s designed by l.ucinda 0'01cill ofDigiral's D e si gn
Group. Our rhanks go to cluthms Vict<H· Bc1hl c111d l'cllll <_;.Juthicr f(H· prov i dinp; the screen saver, and to author
Bill Hallahan t(Jr the DEC:t;J\k Sotrw:rrc synthetic speech
SIKCtmgrcJnl used in the cover gr.1phic.
Correction fo1· Vol. 7 No. 3 Cover Design
Description
I n describing rhc cover of the prc\'ious issue, vol. 7 no. to p r ope r
3, we neglected l y credit the sources of the cover imap;es. The visualiutions 011 the ti·ont and b:1ck covers were cre:Jtcd lw jon:1th:1n Shade usi ng the comput.Hional resources of the
Sc111 Diego Supercomputer C:cmcr. \Vc thank )onat!J.rn cmd the C:elltcr ti>r the usc of these im;Jges.
The f)(qital
Tl'ch u ic a l
{o u m a i is a retereed journ a l published quarterly by Digital
Equipment Catvoration, 30 Porter Road
L)0
2
/D lO, Litderon, Massachusetts
0 l 4
6
0
.
Subscriptions to the.fouma/ are $40.00
( non
-
U
.S. $60) tor tour issues and
$75.00
(non-U.S. $115) for eight issues and must be prepaid in
U.S. funds. University and c o lle g e professors and Ph.D. students in the electrical engineering and computer science t ielcls rece iv e c ompl i ment ar y sub scriptions upon request. Orders, inquiries, and address changes should be sent to the
/Jigito/ [echnical[ourna/at the published by address. Inquiries can also be sent elec tronically ro dtj@digital .com. Single copi e s and back issues arc available for $16.00 each by calling DECclirect at 1-800-DIGITAL
( 1-800-344-4825
)
.
Recenr back issues of thej o u rnal are also available on the lmernet at http:/ jwww.digital.com/info/DTJ/home. html. Complete Digital Internet listings can be obtained by sending an electronic mail message ro [email protected].
Digital employees may order subscriptions through Readers Choice bv entering vrx
P ROFILE at the system prompt.
Comments on the content of anv paper arc
11·elcomed and mav be sent to the mana gi n g editor at the published-lw or network
:1ddress.
Copvright
© 1996
Digital Equipment
Corporation. Copving 11ithout fee is per mitted pro,·ided that such copies arc made t(Jr usc in educational institutions bv burltl' membas and are nor distributed tor com mercial advanragc. Abstracting with credit ship is permitted.
The int(mnation in the journal is subJeCt to dJ
J
< n g c without notice and should nor be consnued as a commitmem bv Digitcll
Equipment Corporation or by the compa nies herein r eptTse n ted
.
Digital Equipmem
Corporation assumes no responsibility t(Jr any errors that may appear i n thcjourl/al.
ISSN 0898-901X
Documentation Number EY-U002E-Tj
Book production was done by Quantic
Communications, Inc.
· r h c t c l ll o 11
· i n g arc tradem.1rks of
Equipmem
Corporation:
AccuLook,
AccuVideo, Alphcl(;cneration,
AlphaStation, DEC:, DEC: OSI'/1,
DECchip, DEC:sak, DEC:st:Jtion,
DECtalk, Digital, the Dil;II't\1.\ogo,
Digital C:\IX, FuiiVicko, OpcnV,viS,
PDP,
RZ,
TURBOchanncl,
UI T
R
I
X
,
;111d VMSclustcr.
C-Cube and CI.550 :1re tr:1dem:1rks of
C C ube Micmsysrems.
I-IA/6000, IBM, Powerl'C, c111d PS/2 arc registered trJdcmarks of
Inr nn a ti o nal
Business Machines C o r po r a ti o n.
1-lewlctt-Packard a n d
HI' c11·e regist ere d trademarks and Switch
O v e r UX i s a mark of f-lewlcrr-!';1ckard
Co mp.111y. trade
INDEO is a registered trackm;Jrk and
Pentium is a trademark of !nr
c \ Corpor a tio n .
Micmsoft is a re g ist ere d n·ademark and Video tcJr
Windows, Windows, and Windows NT
:1 r e trademarks of Mic r o so ft C orp o r a t i o n.
MIPS, R3000, and
R4000 arc registered trademarks ,,f Ml l'S Techn o l og ies, Inc.
MotoroLJ is a
1-c gistc r c d trademark of
Motorola, Inc.
:sJI'S is a registered tradcm,Jrk ;llld
SI'ARC:clusterl is a trademark of
Su n lvlicros�'srcms, Inc.
ORAC:I.J-:7 is a tra de mark o f
Or;Jcl e
Co rp o r a t i o n.
Parasight is .1 trademark of Encore
Compute!· Corporation.
QuickTime is a trcJdemark of A pp l e
Comrutcr, Inc.
Sl'EC:�J, Sl'EC:int, ;Jnd Sl'FC:mark .1re trcJdcmarks of the St a mh r d Pcrt(>rnlance
EvalucHion Co uncil. l'NIX is .1 rcp;istcred trademc1rk in the l
'nitcd States and other counn·ies, licensed exclusivck through X /O pe n
C o mp; l ny
L t d.
X Win do w System is a trademark of the
MasscKhusetts Institute ,,t
.. lc ch n o logy.
Contents
Foreword Robert A. Ulichnn·
AUDIO AND VIDEO TECHNOLOGIES
DECtalk Software: Text-to-Speech
Technology and Implementation
The J300 Family of Video and Audio Adapters:
Architecture and Hardware Design
The J300 Family of Video and Audio Adapters:
Software Architecture
Software-only Compression, Rendering, and
Playback of Digital Video
Integrating Video Rendering into Graphics
Accelerator Chips
William I. Hallahan
Kcnncrh W. Correll and Robert A. Ulichncy
Paranwir Bahl p,ll·anwir Bahl, PaulS. Gauthier, cmd
Robert A. Ulichncv
D. Seiler and R o b e rt
A. UlichiJCI'
.
UNIX AVAILABLE SERVERS, REAL-TIME DEBUGGING TOOLS
Technical Description of the DECsafe
Available Server Environment LJIITCnce S.
Cohen and
John
H. WilliJms
Parasight: Debugging and Analyzing Real-time
Applications under Digital UNIX i'vlichael Palmer and )efti·cy
M.
Russo
3
89
10 l
20
34
S2
76
Di)2,it.11 Teclmic<li )ourn.1l Vol. 7 l\:o. 4 199:>
Editor's
Introduction
This issue's opening section features audio and video technologies that exploit the power of Digital's 64-bit
RISC Alpha systems. Papers describe new software and hardware designs that make praerical such applications as text-to-speech conversion and ti.JIJ motion video on the desktop. A sec ond set of papers shifrs the tows to the UNIX environment with discus sions ofbigh-availability services and of Encore Computer Corporation's new real-time debugging tool.
The opening paper tor the audio and video section references an audio technology that physicist Stephen
H:nvking uses to convert the text he types to highly intelligible syn thetic speech. Recently, engineers h:ll'e ported this mature 10-yearold hardware technology, called
DECtalk, ro text to-speech sofrware.
Bill Hallahan explains that the com putational power of Digital's Alpha systems now makes ir possible tor a software speech synthesizer to simul taneously convert many text streams to speech without overloading a work station. After revie11�ng relevant speech terminology and popular synthesis techniques, he describes OECtalk
Sofrware multithreaded processing and the new t e xt to -s peech applica tion programming interf.1ce for
UNIX and
NT workstations.
Video technologies-full-motion video on workstations-also capital ize on the high performance of Alpha systems. In the first of four papers f(JCused on digital video, Ken Correll and Bob Ulichney present the J300 video and audio ad<lpter architecture.
To improve on past full-motion \'ideo implementations, designers sought to allow video data to be treated rhe same as am· other data type in a
\\'Ork station. The authors review the J300 features, including a versatile color map rendering system, and the sub system design decisions made to keep product costs low.
Victor Bahl then presents the ]300 software that controls the hardware.
The challenge for sofrware designers was ro obtain real r i me performance fi·om a non-real-time operating sys tem. A description ofrhe video sub system highlights the video library and an innovative use of queues in achieving good performance. This software architecture has been imple mented on Open VMS, Windows
NT, and Digital
UNIX platforms.
A third paper on video technology looks Jt delivering video without spe cialized hardware, that is, a sofrware only architecture for general-purpose computers that provides access to video codecs and renderers through a flexible application programming inrerf.1ce. Again, taster processors make a softwar e only solution possi ble :It low cost. Authors Victor Bah I,
Paul G:wrhier, and Bob Ulichney prdace the paper with an overview of industry-standard codecs and compression schemes. They then discuss the creation of the software video library, its an:hitecture, and its implementation of video render ing that parallels the ]300 hardware.
The final paper in the audio and video technologies secrion explicitly raises the question of what features are best implemented in hardware and what in software. The context ti:>r the question is a gr<lphics accelerator chip design that integrates traditional synthetic graphics features and video image display f<::atures-umil now, implemented separately. Larry Seiler and Bob Ulichney describe the video processing implemenn.:d differently in two chips, both of which offer sig nificantly higher performance with minimal additional logic.
The common theme of our second section is the UNIX operating system.
Larry Cohen and John Williams pre sent the DECsafe Available Server
Environment (ASE), which provides higb availability tor applications run ning on Digital
UNIX systems. They describe the ASE design for detection and dynamic reconfigur<ltion around host, storage device, and nenvork fail ures, and review key design trade-offs that favored software reliability and data integrity.
Mike Palmer and Jeff Russo then contrast Encore Computer Corpora tion's set of debug and analysis tools for real-rime applications, called
Parasight, with conventional
UNIX tools. They examine the Ceatures that are critical in an effective real-time debugging tool, for example, the abil ity to att3ch to a running program and to analyze several programs simul taneously. A description follows of the Parasighr product, which includes the features necessary for real-time debug and analysis in
J set of graphi cal user interface tools.
Upcoming in our next issue are papers on a variety of topics, includ ing Digital
UNIX clusters, eXcursion for
NT,
<lnd network services.
Jane C. Blake
Nlcmaging Editor
2 Digiral Technical )ournJI Vol. 7 No.4 1995
Foreword
Robert A. Ulichney
Senior Consullinp, h:np,ineer
J<esearch and Ad�·unced
Decelopment.
Cambric(u,e
Research Lah
"Can you dig it ... New York State
Throughway's closed, Man. Far out, Man," announced a young Arlo
Guthrie in rhe vernacular on the stage ar Woodstock in 1969. Reading these words may evoke a mental picture of rhe evem, but it sure is a lot more hm to hear :md see Arlo deliver this mes sage. Audio and video technology is the featured theme of this issue of the
Digital Technicai.Jounzctl.
Four years before Arlo's traffic report, in rhe year that a young Digital
Equipment Corporation introduced the PDP
8, an interesting forecast was made. Gordon Moore, who was yet to co-found Intel, asserted in a lit tle-noticed paper that the power and complexity of the silicon chip would double every year (later revised to every 18 months). This prediction has been generally accurate for
30 years and is today one of the most celebrated and remarkable "laws" of the computer industry.
While we enjoyed rhis exponential hardware ride, there was always some question about the ability of applica tions and software to keep up. If any thing, rhe opposite is rrue. Software has been described as a gas that imme diately fills rhe expanding envelope of hardware. Ever since the hardware envelope became large enough to begin to accommodate crude forms of audio and video, the pressure ofrhe software gas has been great indeed.
Digitized audio and video represent enormous amounts of data and stress the capacities of real-rime processing and transmission systems.
Digital has particip;1red in expand ing rhe envelope and in tilling ir; irs hardware performance is record breaking and its audio and video tech nologies are srare-ofrhe-arr. Looking specifically at the tour categories into which computer companies segment audio and video technologies, Digital is making contributions in each of these: analysis, synthesis, compression, and input/output.
MIT's Nicholas Negroponre believes that practical analysis, or interpretation, of digitized audio and video will be the next big advance in the computer industry, where noth ing has changed in human input (key board and pointing device) since, well, the Woodstock era. Digital is actively investigating methods tor speaker independent speech recognition and, in the area of video analysis, means to automatically detect, track, and recognize people.
The synthesis of still and motion video, more commonly referred to as computer graphics, has traditionally been a much larger area of focus tlun the handling of sampled video. Syn thesis of audio, or text-to-speech conversion, is the topic of one of the papers in this issue; DECtalk is largely considered to be the best such synthesis mech:mism available.
W hen audio or video data are rep resented symbolically,
:IS is the case after analysis, or prior to synthesis, a most efficient form of compression is implicitly employed. However, the task of storing or transmitting the raw digitized signal can be overwhelm ing, especially at high sampling rates.
Compression techniques are relied upon to ease the volume of this datJ in two ways:
( l) reducing statistical
Digital Technid Journal Vol. 7 No.4 1995 3
4 redun,hncy and (2) pruning data that ll'ill not be noticed bv exploiting ll'hat is kno"'n about hum:m percep tual systems. In this climate ofinreroper abilitv and open svstems, Digital recognizes the imporrance of adher ing to accepted st;mdards for audio
;lnd video compression versus the promotion of some propriet;lrv reprcscnt<ltion.
Th�.: last category is that of l/0.
Audio :-tJH.J video input require a means f(>r signal acquisition and
;lnalog-to-digital conversion. The focus here is on preserving the integ ritY of the signal Js opposed to inter preting the data. Proper rendering is needed for good-quality output,
;llong ll'ith digit;ll-to-;malog con
\'ersion. �or both ;ludio and video, trade-ofrs must be made to accom modJtc the highest degree ofs:�mpling resolution in time and amplirude.
Digital is a leader in the ;lrea of video rendering with our AccuVideo technologv, aspects of which aJT described in p:1rr in three papers in this issue. Video rendering incorpo rates ;111 processing that is required to tailor \·ideo to a p;1rricular target dis play. This includes soling and filter ing, color adjustment, dithering, and color-space com·crsion from video's
J u mi nance-chromi nance represen ration to R.CB. In irs most general form, Digital's rendering tedmology ll'ill optimize displav qualit\' �i\·cn an)' number of available colors.
The c.1rliest form ofAccuVidco
:�ppeared in a 1989 testbed, known ime rn
; l lh
· as Picror. This led to the widely distributed research prototype called )video in I 991. )video w;lS a TUR130ch;mnel bus option with
J PEG compression and decompres sion and \\·as the first prototvpe to combine dithering with color-space conversion. }\·ideo was the basis for design of the Sound & Motion J 300 product, which incJ uded a rem:1rbblv improved dither method. A f(>llow-on ro }300 is a PCJ-bus \·crsion c1lled ruiiVideo Supreme.
In products th;lt render RG B data instead of video, Digit<ll's rendering tcchnolog\' is reterrcd to :1s Accu Look; except tix this one difkrcnce, the rest of the rendering pipeline is idcntic:li ro AccuVideo. Accu Look products include gr;lphics options tor ll'ork stations: ZLX-E
(S�B+) designed for the TU RBOchanncl and ZLXp-1-:
(TGA) designed as an cntr\·-in·cl product r(Jr the PCl bus.
Accu Video rendering is a key tcarurc in the DECchip 21130
PC graphics chip and in the TGA2 high-end workstation graphics chip.
While noted f(>r irs high image qual it\', A.ccuVideo is also cfticientk implemented in software; it is av:�il able as p:-�rr of a tool kit with evcrv
Digital UNJX, Open VMS, Jnd
Windcms �T platfim11. vVith Moore's law on the loose, ir can be <lrgued that hardware imple mentations oh·ideo rendering arc not justified
;ls so!tw;1rc-only versions grow in speed. Although toci:ly's pro cessors c:-�n indeed handle the pl:l\' bxk of video bv both decompressing and rendering at :1 qu:�rrer of hill size, little is lett f(Jr doing anvthing else.
Moreo\·cr, users will want. to scale up the dispby sizes, <lnd perh;lps add multiple \'ideo stre;1ms-and still be
:-�ble to usc their processors to do other things. for rhe nc;lr term, h�lrd
\\·are \·ideo rendering is justified.
The five p:1pcrs that make up rhc
;ludio and video technology theme of this issue :tre bur a sndl sampling of the \\·ork under \\;1\. in rbis are:t at Digital; look tor more p<lpers to t(>lloll' in subsequent issues of this
Journal.
As rhc audi,, .md \·ideo g.Js continues to till the ever-expanding lurdwarc cn\·clope, \\·c look t())'ll'<lJ-d to :111 enriched and more natural experience with computing de' ices.
Arlo's Woodstock p;lls \\'Ould likely
�1grce that this sounds lik e more ti.1n.
Vol. 7 0\io. 4 1995
I
William I. Hallahan
DECtalk Software:
Text-to-Speech
Technology and
Implementation
DECtalk is a mature text-to-speech synthesis technology that Digital has sold as a series of hardware products for more than ten years.
Originally developed by Digital's Assistive
Tech nology Group (ATG) as an alternative to a character-cell termi nal and for telephony applications, today DECtalk also provides visu ally handicapped people access to information.
DECtal k uses a digital formant synthesizer to simulate the human vocal tract. Before the advent of the Alpha processor, the com puta tional demands of this synthesizer placed an extreme load on a workstation. DECtalk
Software has an application program ming interface (API) that is supported on multiple platforms and multiple operating systems.
This paper describes the various text-to-speech technologies, the DECtalk Software architecture, and the API. The paper also reports our experi ence in porting the DECtalk code base from the previous hardware platform.
Du ring the past ten years, advances in computer power have created opporruniries for \'Oice input and our pur. Many major corpor;nions, including Digita l , provide database access through the telephone. The advent of Digital's Alpha prou.:ssor has ch:mged the economics of speech synthesis. I nstead of an expen sive, dedic:1ted circuit card that supports onlv ��single ch annel of symhesis, svstem developers can usc an
Alpha- b:1sed workstation to su pport many channels simul taneously. In add ition, since text-to-speech con version is a light load tor an Alpha processor, appli ca tion developers can tl·eely integrate text to speech i nto their products.
Digital's DECtalk SoiT\\"are provides natu ral -sound ing, highly inteJligible text-to-speech syntl1esis. It is available tor the Digital UNIX operating system on
Digital's Alpha-based platt(mns and tor Microsoft's
Windows NT operating system on both Alpha and
Intel processors. DEC:talk Software provides an easy ro-use application program ming intert:1ce
(API) that is fu lly integLH<.:d with the computer's audio subsystem.
The text-to-speech code was ported ti·om the software t(>r the DEC:talk
PC card, a lm·dware product m�1de bv
Digital's Assisrive Technolot,')' Group . This software
COIJStiturcs over 30 man ye:1rs of develo�1111enr effort
:1nd contains approximatdy 160,000 lines of C pro grammi ng language code.
This paper begins by d iscussing rhe k:nun:s of
DEC:ral k Software and bridly describing the various text-to-speech t<.:dmologies. It then presents a descrip tion of the DECtalk Software architecrurc and the
API . Final ly, the paper relates our cxperiwce in port ing the DEC:talk code b<lSe.
Features of DECtal k Software
The DECtalk Software development ki t consists of a shared li br:trv (a dvnamic l i nk librarv on Windows
NT), a link librarv, a header tile that defines the sym bols and nlllctions used by DECralk Software, SJ111ple applications, and sample source code that demo n stt·ates t h e A P I .
Di�;iral Tcd111iGtl )oum.1l
Vol.
7
No.4
!995
DECtal k Software supports nine preprogrammed voices: tou r male, t(Ju r female, and one child's voice .
Both the API and in-line text commands can control the voice, the speaking rate, and the audio volu me.
The volume command supports stereo by provid ing i ndependent control of the left and right cbannds.
Other in-line commands play wave audio files, gen erate singl e tones, or generate dual-tone multiple frequency (DTMf) signals for telephony applications.
Using the tex t to-spe e ch API, applications can play speech through the computer's audio system, write the speech samples to a wave audio tile, or write the speech samples to buffers supplied by the appl ication.
DECtalk Software produces speech in 3 audio formats:
16-bit pulse code modulation (PCM) samples at an ll
,025-hertz ( Hz) sample rate, 8-bit PCM samples at an 1 1,02 5 - H z sample rate, and J.-L law encoded 8-bit samples at an 8,000-Hz sample rate. The first two f(x mats are standard multimedia audio formats for per sonal computers (PCs). The last format is the stand ard encod ing and rate used tor telephony appl ications.
The API can also load a user g enerated dictionary that def-ines the pron unciation of application-specific words. The development kit provides a window- based too l to generate these d ictionaries. The kit also con rai ns a window-based application to speak text and an electronic mail-notitlcation program. Sample sou rce code includes a simple window-based application that speaks text, a command l ine application to speak text, and a speech -to-memory sample program .
The version o f D ECtal k Software for Windows NT also provides a text-to-speech dynamic data exchange
(DOE) server. T h is server i ntegrates with other appli cations such as Microsoft \Vord . Users can select text i n a Word document and then proofread the text merely by clicking a button. This paper was prootre:td usi n g DECtalk Sofhvare running a native version of
Microsoft Word on an AlphaStation workstatio n .
Speech Terms and DECtalk Software
Hum:m speech is produced by the vocal cords i n the larynx, the trachea, the nasal cavity, the oral cavity, the tongue, and the lips. Figure l shows tbe human speech organs. The glottis is the space between the vocal cords. For voiced sounds such as vowels, the vocal cords produce a series of pulses of air. The pu lse repetition frequency is cal led the glottal pitch. The pulse train is reterrcd to as the glottal waveform. The rest of the articulatory organs filter this waveft>rm.'
The trachea, in conjunction with the oral cavity, the tongue, and the l ips, acts l i ke a cascade of resonant tubes of varying widths. The pulse energy reflects backward and forward i n these organs, which causes energy to propagate best at certain fr-equencies. These are called the f()fmant ti-equencies.
NASAL
CAVITY
Figu re
1
The Speech Organs
LARYNX
VOCAL CORDS
TRACHEA
The primary discrimi nation cues for different vowel sounds are tbe values of the first and second tormant frequency. VO\vels are either t!-ont, mid, or back vow els, depending on the place of articulation. They arc either rounded or unrounded, depending on the posi tion of the lips. American English has 12 vowel sounds. Diphthongs are sou nds that change smoothly from one vowel to another, such as in hoy, bow, and bay. np,
Other voiced sounds include the nasals m, 11, and
(as in ing).
To produce n asals,
�1 person opens the velar tlap, which con nects the tl1roat to the nasal cavity.
Liqu ids are the vowel-like sounds
I and r.
Glides are the sounds y
(as i n you) and w
(as i n we).
Breath passin g through a constriction creates tur bulence and produces unvoiced sounds. fand s are unvoiced sounds cal led ti·icat.ives. A stop (also called
:1 pl osive) is a mome ntary blocking of the breath stream tcJIIowed by a sudden release . The consonants p, h. t. d, 1?., and g arc stop consonants. Openin g the mouth and exhaling rapidly produces the conson ant h. The h sound is called an aspirate. Other conso nant s such asp. t, and k frequen tly end in aspiration, especially when t hey start a word.
An affricativc is a stop i mmediately followed by a fricative. The English sounds ch (as in chett'andj(as i njar) are at1i-icatcs.
These sounds arc all American English phonemes.
Phonemes are the smallest u nits of speech that d istin guish one uttera nce from another in a particular language2
An allophone is an acoustic manifestation of a phoneme. A particular phoneme may have many al lophones, but each <lllopho nc (in context) will sound l ike the same phoneme to a speaker of t h e lan guage that detines the phoneme. A n other way of say ing this is, if two sounds have differen t acoustic man i testations, b u t the use of eit her one does not change the meaning of an u tterance, then by defini tion, they arc the same phoneme.
6
DigiLll Technical )oumcli
Vol.
7
No.4 1995
b bet ch chi n d debt dh thi s el bottle en button f fin g guess hx head jh g i n k
Ken
I let m n nx p met net sing pet red sit sh shin t test th thi n v vest w wet yx yet z zoo zh azure
Phones are the sets of all phonemes and allophones tor all languages. Linguists have developed an interna tional phonetic alphabet ( I PA) that has symbols for almost all phones. This alphabet uses many Greek letters that are difficult to represent on a computer.
American li ngu ists have developed the Arpabet phoneme alphabet to represent American English phonemes using normal ASCII characters. DECtalk
Software supports both the IPA symbols tor American
English and the Arpabet alphabet. Extra symbols are provided that either combine certain phonemes or speci�r certain al lophones to al.low the control of fine speech features. Table 1 gives the D ECtalk Software p honemic symbols.
Speech researchers often use the short-term spec trum to represent the acoustic manifestation of a sound. The short-term spectrum is a measure of the h·equency content of a windowed (time-limited ) por tion of a signal. For speech, the time wi ndow is typi cally between 5 mil liseconds and 2 5 milliseconds, and
Table 1
DECta lk Software Phonemic Symbols
Consonants Vowels and Di phthongs a a Bob a e bat a h but ao bought aw bout ax a bout ay bite eh be ey bake i h bit ix kisses iy beat
OW boat oy boy rr bird u h book uw lute yu cute
Allophones dx rider l x electric q we eat rx oration tx Latin
Silence
_
(underscore) the pitch frequency of voiced sounds varies from 80
Hz to 280 Hz. As a result, the time window ranges tl-om slightly less than one pitch period to several pitch periods. The glottal pitch frequency changes very litrle in this interva l . The other articulatory organs move so l ittle over this time that their ti ttering effects do not change appreciably. A speech signal is said to be stationary over this interval .
The spectrum h as two components for each fre quency measured, a magnitude and a phase shift.
Empirical tests show that sounds that have identical spectral magnitudes sound similar. The relative phase of the individual frequency components plays a lesser role in perception. Typically, we perceive phase differ ences only at the start of low frequencies and only occasional ly at the end of a sound. Matchi ng the spec tral magnitu de of a synthesized phoneme (al lophone) with the spectral magnitude of the desi red phoneme
( ta ken from hu man speech recordings) always improves intelligibility 3 This is the synthesizer calibr3tion technique used tor DECtal k Software.
A spectrogram is a plot of spectral magnitude slices, with frequency on the y axis and time on the
.x axis.
The spectral magnitudes are specified either by color or by saturation for two-color plots. Depending on the time interval of the spectrum window, either the pitch frequency harmonics or the tormant structure of speech may be viewed . It is even possible to ascertain what is said tl·om a spectrogram. Figure 2 shows spec trograms of both synthetic and human speech for the same phrase . The tormant frequencies are the dark regions that move up and down as the speech organs change position. Fricatives and aspiration are charac terized by the presence of high fi:equencies and usually have much less energy than the fclrmants.
The bandwidth of speech signals extends to over
10 ki lohertz ( k Hz) although most of the energy is confined below 1 ,500 Hz. The m inimum in telligible bandwidth for speech is about 3 kHz, but using this bandwidth, the quality is poor. A tel ephone's band width is 3.2 kHz. The DECtalk PC product has a speech bandwidth just under 5 kHz, which is the same
<lS the audio bandwidth of an AM broadcast station .
The sample rate of a digital speech system must be at least t\vice the signal bandwidth (and might have to be higher if the signal is a bandpass signal ), so the
D ECtalk PC uses a 10-kHz sample rate . This band width represents a trade-off berween speech quality and the amount of calcu lation (or CPU loading). The
DECtalk Software synthesizer rate is l l ,025 Hz, which is a standard PC sample rate. An 8-kHz rate is provided to support telephony applications.
People often perceive acoustic events that have different short- term spectral magnitudes as the same phoneme. For example, the k sound i n the words kill
Digital Techni(JI Journal Vol.
7
No. 4 1995
7
Figure 2
T"·o Spectrograms of rile
Crrerancc
" Line up at the sncen door." Tile upper spectrogl·c\111 is the au thm's �pee ell.
The l o\\·cr spcctrogr:un is sYnthetic speech produced b1· DECt:llk Soft\\·an:. and cool have ver y diHcrent magnitude spectra . An
American percei1·cs the two spectra as the same sound;
IH)II CI'cr, the sounds :1rc very d i frc re n t to someone tt·om Saudi Arabia. A Japanese person does nor per ceive :my d i fference between the words car and call.
To clll Engl ish spea k e r, the r and the I sound difkrcnt c1·cn though they have nearlv identical magn itu d e spcctLl . The I sou nds in the 1\'0rds w/1 :md /eo/arc d i f krcnt spectra l ly ( acoustical l v ) b u t ha1·e the ScllllC sound. Thus they arc the same phoneme in English .
Several a llophones a re requ i red to represent t h e r� phoneme. Most consonant phonemes req u i re sc1-cral difkrcnt : tllophoncs because the vol\'cl sounds nnt to them cha nge their acoustic manifestations. This dkct,
Gll lcd coarricu lation , occurs because it is often un nec essary rc>r the articulatory orga ns to reach the ti nal position used to generate a phoneme; they merdv need to gesture to\\'ard the tined position . Another type of coarriculation is part o f the grammar of :�
L111gu:1gc . For example, the phrase dou ·/ you is often pronou nced doctil choo.
A l l allophones that n:prcscnt rhc phon eme k arc prod uced lw closing the 1·e l u m and then suddenly opening it and releasing the broth strc:�m. Speakers of the English la nguage perceive all these J!l ophoncs as the s:1mc sou nd, which suggests that synthesis may be modeled by an articulato ry model of speech prod uc tion . This model ll'ould p resun1Jblv handle coarricula tion efkcts that are not due to gram m:1r. I t is currcnrlv nor known how to consistently determine speech org�1n positions ( or control strategies) directly h·om acoustic speech data, so articu latory models have had l ittle success tor text-to-speech synrhcsis 4
For English, the v oicing pitch provides cues to cbusc boundaries and mcJ ning. ChJnging the ti·c qu cncv of the 1 i brarion of the I'OCll cords ,·aries rhc pitc h . I ntonation is the shape of the pitc h l'ariarion across
:1 d:tllse. The sentence " Ti m is leaving.'' is pro nou nced differently than "Tim is leavi ng>" The l :ntcr torm req u i res difkrcnt intonation, depending on
11 hcthcr the inrcnt is to emphasize th:1 t it is "Tim" ll'ho is le:wing, or tbar "leaving" is 11·h ar Tim i s ro do.
A word or p h rase is stressed bv increasing irs pitc h , amplitude, o r duration, o r some com bination of these .
In tonation incl udes p i tc h ch<lngcs due ro stress and norm<ll p i tch \'ariarion :�cross a clause . Correct i n toJJ:l rion is not J!II'J\"S possible because i t requires speech understanding. O ECrJik Software pcrt(>nllS an an<llysis of clause structure that i ncludes tl1c t(mn c lasses of both words and pu nctuation and then applies a pitc h contour to a clause . ·rhc t(mn class defini tions include sym bols tor the p:urs of speech ( :� rticlc, adjecti1·c, adve rb, conju nction , noun, preposition, verb, etc . ) and sym bols to indic:�tc i f the 1vord is a number, Jn abbreviation, a homograph, or a special word ( requir ing special propricrar\' processing ) . for the scnrcncc,
"Tim is lcal'i ng)" rhc question mark u uses D FCr:1lk
Software to raise the ti na! p i tch, bur no stress is pur on
"Tim" or " leaving." Neutral i ntonation sometimes sounds boring, but at least it docs not sou nd foolish.
Text-to-Speech Synthesis Techniques
Early attempts at text- to-speech synthesis assembled clauses hy conc:ncnJting recorded words. This tech nique produces cxtrcmelv unnatura l-sou nding speech.
8
Di�iral Tcchnicll )ound Vol.
7 No. 4 1 <)<)5
f n continuous speech , word durations are ofi-en short ened and coarriculation efkcrs can occur between adja cent words. There is also no way ro adjust the intonation of recorded words. A huge word database is req uired , and words th;n :tre nor in rhe d:trabase cannot be pro nounced . The resulting speech sou nds choppy.
Another word concatenation technique uses record ings of the �i.>rmanr patterns of words . A �i.m11ant synthesizer smoorhes �(>rnl:tllt transitions ar the word bou ndaries. A 1·:-triation of this technique uses linear prcdictiw coded ( LPC) words. An ad1·a ntagc of the t(>rmanr symhcsizer is that the pitch and du r:ttion of words may be varied . UntiJrtu nately, since the p honeme bound aries within a word are difficu l t to determine, the pitch and d u r;nion of the individual phonemes un nor be ch:mged . This technique also requires a L1rge database. Again, a word can be spoken on ly if it is in the database. In genera l , the quality is poor, although this technique has been used with some success ro speak numbers.
A popular technique today is ro store actu:tl speech segments that contain phonemes and p honeme pairs.
These sp..:cch segments, known as diphones, are obtained trom recordings of human speec h . They con rain all coarricularion drects that occur tor a particular language. Diphones are concJtcnatcd to produce words and sentences. This solves the coa rticu larion problem, bur it is impossibk to accuratelv modi�)• the pitch of a tly s<.:gmenr. The intonation :tcross a clause is gencr
:tl ly incorrect. Even worse, the pitch varies trom s<.:g mcnt to segment within a word . The resulting speech sounds unn<Hural, u nless the system is spea king a phrase that tht di phones came from ( this is a dc1·ious marketing plov ) . Nevertheless, d iphone synthesis pro d uces speech that is fai rly i mclligible. D i phonc syn thesis requir<.:s rc btively l ittle compute power, but it is memory intensive. American Engl ish requi res :tpprox imately 1 ,500 dip hones; diphonc synthesis wou ld have to provide a large database of approximatelv
3 m<.:ga
DECtalk Software uses a d igital tc>rmant symhesizcr.
The syn thesizer in put is derived from phonemic sym bols instead of stored t<.mn:tnt patterns as in
:1 conve n tional t<.m11ant svmhesizer. I ntonation i s based on clause structure. Phonetic rules detcrmi nc coarticu la tion cfkcts. The synrhcsizer req uires only two rabies, one t()r each gend<.:r, to map al lophonic variations of each phoneme to <Koustic events. Modification of vocal tr:tct parameters in the synthesizer allows the system to gmcratc multiple voic<.:s without a signiticant increase in stor•lge requircmcms. (The D ECtJI k code and data occupy less than 1 . 5 m<.:gabyres. )
Poor-q uality spetch is d i fticu l t to understand and causes tatigue. Linguists usc standard phoneme recog nition tests and comprehension tests to measure the i ntell igi bility of synthetic speech . The D FCta l k tamilv of prod ucts achieves the highest test. scores of a l l text to-speech systems on the market.; Visually handi capped individuals prdcr D ECtalk ovn all other text-to-speech systems.
How DECta/k Software Works
DECulk Software consists of eight processing threads:
( 1 ) the text-queuing thread, (2) the command parser,
( 3) the letter-to-sou nd converter,
( 4) the phonetic and prosodic processor, ( 5 ) the vocal traer model
( V T M ) thread, ( 6) t h e :tudio rhn.::td ,
( 7 ) t h e synchronization thread , and ( 8 ) the timer thread. Tht tex t, VTM, audio, synchronization , and rimer threads ::1re nor part of the DECta l k PC soft,vare (the D ECtalk PC VTM is on a sptcial Digi t:tl Sign<li Processor) and have been addtd to DECtalk Sofuvare. The audio th read creates the timer thread 1\'htn the text-to-speec h S\'Stem is i nitialized . Since tht audio thread docs not usuallv open the audio dcvic..: u nt i l a suftici<.:IH number of audio samples are qu..:ucd, rhe timer th read serves to t(lrcc the audio to play in case any sampks have been in the qu eu<.: too long. The DECta l k Software th reads pert(xm serial processi ng of data as sholl'n in Figu re
3.
CALLBACK FUNCTION FOR UNIX MESSAGE FOR WI NDOWS NT
POLL AUDIO
POSITION
KEY:
CALLBACK FUNCTION FOR UNIX. MESSAGE FOR WINDOWS NT
1
COMMAND
PARSER
THREAD
SYNCHRON IZATION EVENT
,
...
SYNCHRONIZATION
MESSAGES
LETIER·TO·
SOUND
THREAD
,
...
PROSODIC
THREAD
�
�
�
���
ONIZATION f-
PHONETIC AND
PROCESSING
_
•
,
, ...
VOCAL TRACT
MODEL THREAD
�'";""'"
�
AUDIO THREAD
APPLICATION
�
ASCII TEXT
TEXT QUEUING
THREAD
-·
I
ASCII TEXT ASCII TEXT PHONEMES VTM
COMMANDS
SPEECH
SAMPLES
INDICATES PIPES
Figure 3
The
DEC:r;llk S o ft
\\'arc
Archir..:crurc tor \.V
i n d
O\\'S
NT
Digital Tcchnicc1l Journal
Vol .
7
No. 4 1 995
9
Multithreading al lows a simple a n d efficient means of throttling data in m ultistage, real -time systems.
Each thread passes its output to the next thread through pipes. Each thread has access to nvo pipe han d les, one for input and one for output. Most th reads consist of a main loop that has one or more calls to a read_pipe fu nction tollowed by one or more calls to a write_ pipe function . The write_pipe function wil l block processing and suspend the th read if the speci fied pipe does not have enough free space to receive the specified amount of data. The read_ pipe function will block processing and suspend the thread if the specified pipe docs not contain the requested amount of data. Thus a n active th read will eventually become idle, either because there is not enough i nput data, or because there is no place to store its output.
The pipes are implemented as ring buffers. The ring bu ffer item cou nt is protected by mutual-exclusion objects on the Digi ta l
U N I X operating system and by critjcal sections on the Windows NT operati ng system.
The pipes are created at text- to-speech initialization and destroyed during shutdown . The DECtalk Software team implemented these pipes because the pipe cal ls supplied with the Digital
UNIX and Windows NT operati ng systems are f()f interprocess communication and arc not as efficient as our pipes.
The DECtalk Software threads all used different amou nts of
CPU time. The data bandwidth increases at the output of every thread between the command thread and the
VTM thread. Since the
VTM produces audio samples at a rate exceeding 1 1 ,025 samples per second, it is no su rprise that the
VTM uses the most
CPU time of all threads. Tab le 2 gives the percentage of the total application time used by each thrc<�d when the Windows
NT sample application "say" is continu ously speaking a large text tile on <�n Alpha
AX P
1 5 0
PC product. The output sample rate is 1 1 ,025 Hz.
Note that the "say" program main thread blocks and uses virtually no
CPU time after queuing the text block. These percentages have been calculated from times obtained using the Windows
NT ped(xmance monitor tool.
Because the data bandwidth i ncreases at the output of successive threads, it is desirable to adjust the size of each of the pipes ring bu ffers. If one imagines that all the pipes had an i n fi nite length (and the audio queue was intinite ) and that the operating system switched thread context only when the active thread yielded, then the text thread wou ld process al l the
ASCII text data before the letter to-sound thread wou ld ru n .
Likewise, each successive thread would r u n t o comple tion before the next thread became active. The system l atency would be very high, but the thread switch ing woul d be minimized . The system wou ld use 1 00 percent of the CPU until all the text was converted to audio, and then the
CPU usage would become
Table 2
DE Ctalk Software Th read Loading
T h read
Appl ication
(say.exe)
Text queue
Command parser
Letter-to-sound processing
Prosod ic and phonetic process ing
Vocal tract model
Audio
Sync hron ization
Timer
System
Percentage of Total
Application CPU Time
1 .0
0.2
1 .4
2.4
1 8.3
7 1 . 9
2.9
0.0
0.0
1 .9 very low as the audio played out at a fixed rate.
Alternatively, if all the pipes are made verv short, the system latency is low. I n this case, a l l but one of the thre<Jds will become blocked i n a very short time and the startup transient i n the
C P U loadi n g wil l be mini mized. Untortunately, the threads wi ll constantly switch, resulting in poor efficiency. What is needed is a trade-off benveen these nvo extremes.
For a speciticd latency, the opti m u m pipe si zes that m i n imize m e mory usage for a given efficiency are in a ratio such that each pi pe contains the same temporal amount of data . For example, let us assu me that 64 text characters ( requ iri n g 64 bytes) are i n the command thread . They p roduce approxi mately 100 phonemes (requiring 1 ,600 bytes) at the output of the l etter-to-sound thread a nd approximately 750
VTM control commands ( requiring 1 5 ,000 byres) at the output of the prosodic and p hon etics thread. I n such a case, the size o f the input pipes for the com mand, l etter- to-sound, and prosod ic and p honetic threads cou l d be made 64, 1 ,600, ;:md 1 5 ,000 bytes, respectively, to minimize pipe memory usage t(>r the speci tied latency. ( A l l numbe rs are h ypothetica I . )
The pipe sizes i n DECta l k Software actua l ly increase at a s l ightly faster rate than necessary. We chose the
E:�ster rate because memory usage is not critical since a l l the pipes Jre small relative to other data struc tures. The size of the VTM input pipe is the m ost critical : it is the largest pipe because it supports the largest data bandwidth.
The Text Thread
The text thread's only purpose is to buffer text so the appl ication is not blocked during text processi ng.
An application using text-to-speech services calls the TextToSpeechSpeak API fu nction to queue a null-
1 0
Digital T�chnical Journal
Vol. 7 No. 4 1995
terminated text strin g to the system. This API function copies the text ro a bu ffer and passes the buffer ( using a special message structure ) to the text thread . This is done using the operatin g system's PostMessage function for vVindows
NT and a thread-sat-e linked l ist for Digital
U NIX .
After the text th read pipes the enti re text stream to the command thread, it ti·ees the text buff-er and the message structure.
The Command Processing Thread
The command processing th read parses in- l i ne text commands. These com mands control the text-to speech system voice selection, speaking rate, and audio volume, and adjust many other system stare parame ters. For OECtalk, most of these commands <lre of the torm [ : command <parameters> ] . The strin g " [ : " specifies rh:n a command stri ng �allows . The stri ng "]" ends a comma nd. The toll owi ng stri ng ill ustrates sev eral in-line commands.
[ :nb][ :ra 200] My name is Betty.
[ :play aud io.wav]
[ :dial 5 5 5- 1 2 1 2 ] [ : tone 700 1 ,000]
This text will select the speaker voice �()r " Betty," select a speaking rare of 200 words per minute, speak the text "My name is Berty. " and then p!Jy a wave audio �i le named "audio.wav." Final ly, the DTMF tones for the number 5 5 5 - 1 2 1 2 are played followed by a 700-Hz rone �or l
,000 milliseconds.
Because the text-to-speech system may be speaking while simultaneously processing text i n the command thread, it is necessary to synchronize the command pro cessing with the audio. The DECtal k
PC product ( from which we ported the code) did not perform synchro nization unless the application placed a special string bdore the vol u me command . For DECralk Software, asynchronous control of all fu nctions provided by the in-line commands is already available through the text-to-speech
APT cal ls. For this reason, the DECtalk
Software in-line commands are all synchronous.
The D ECt:tlk command [ :vol ume set 70] wil l set the audio volume l evel to
70. Synchronization is per formed by i nserting a synchronization sym bol in the text stream . This symbol is passed through the system u ntil it reaches the VTM thread. When the VTtvl thread receives a synchronization sym bol , it pipes a message to the synchronization thread . This message causes the synchronization thread to signal an event as soon as all audio ( that was queued before the message) has been played . The volu me control code in the com mand thread is blocked until this event is signaled . The synchronization thread also handles commands of the form [ : i ndex mark 1 7 ] . Index mark commands may be used to send a message value ( i n this case 1 7 ) back to an application when the text up to the index mark command has been spoken.
The command thread passes control messages such as voice selection and speaking rate to the letter-to soun d and the prosodic and phonetic processing threads, respectively. Tone commands, index mark commands, and synchronization symbols are format ted i nto messages and passed to the letter-to-sound thread . The command thread also pipes the input tex t string, with the bracketed command strings removed, to the letter-to-sound thread .
The Letter-to-Sound Thread
The letter- to-sound ( LTS) thread converts ASCTI text seq uences to phoneme sequences. This is done using a rule- based system and a dictionary tor exceptions. It is the single most complicated piece of code i n a l l of
DECtalk Sot-i:ware. Pronu nciation of English language words is complex. Consider the dif'ferent pronu ncia tions of rhe stri ng ough in the words rough, through, hough. thought. dough, cough, and hiccough -"
Even though the
LTS thread has more than 1 ,500 pronun ciation rules, it requires an exception dictionary with over 1 5 ,000 words.
Each phoneme is actually represented by a structure that contains a phonemic symbol and phonemic attri butes that include duration, stress, and other propri etary tags that control phoneme synthesis. This is how allophonic variations of a phoneme are handled. In the descriptions that tal low, the term phoneme refers either to this structure or to the particular phone spec i�ied by the phonemic sym bol in this structure.
The
LTS thread first separates the text stream into clauses. Clause separation occurs in speech both to encapsulate a thought and because of our limited lung capacity. Speech run together with no breaks causes the listener (and the speaker) to become fatigued . Correct clause separation is important to achieve natural into nation. Clauses are deli neated by commas, periods, exclamation marks, question marks, and special words.
Clause separation requires simultaneous analysis of the text stream. For example, an abbreviated word does not end a clause even though the abbreviation ends in a period . If the text stream is sufficiently long and no clause delimiter is encou n tered , an artificial clause boundary is inserted into the text stream.
After clause separatjon, the LTS thread perkmns text normalization . For this, the LTS thread provides spe cial processing rules tor numbers, monetary amounts, abbreviations, rimes, in-line phonemic seq uences, and even proper names. Text normalization usually refers to text replacement, but i n many cases the LTS thread actually inserts the desired phoneme sequence directly into irs output phoneme stream instead of repl acing the text.
The LTS thread converts the remai ning unprocessed words to phonemes by using either the exception dic tionary or a rule-based " morph" lexicon . (The term morph is derived from morpheme, the minimum u nit
Digiral Technical fournal Vol. 7 No. 4 1 995
I I
o f meaning tor a hl11guage . ) B r combining these t\:vo
:1pproaches, me mory utilization is minimized . A user ddinable d ictiona ry may also be loaded ro detlnc application-specific terms. During this conversion, the
LTS thread assigns one or more f-(xm class�.:s to each word . As memioncd previously, form class ddinitions include symbols For abbreviations and homographs.
A homograph is a word that h:�s more than one pro nunciJtion, such as a/tenwte or console. DECta l k
SoFtware prono,Jnces most abbreviations a n d homo gra phs correcrly in context. An alternate pronu ncia tion of a homograph may be fort.:ed bv inserting the in-line command [ : pron alt] in fi·ont ot· the word.
D ECralk Sorhvarc speaks the phrase " Dr. Smith lives on Smith Dr." correctly, as " Doctor Smith lives on
Smith Drive ." It uses the correct pron unciation of the homograph liws.
Before applying ru les, the
LTS
thread pcrt(mns a di ctionary lookup f-or each unprocessed word in a clause . If the lookup is successfu l , the word's f(m1l classes and a stored p honeme seq uence are extracted from the dictionary. Otherwise, the word is rested for an English suftl x, using a su fri x ta ble. If
:1 s u nl x is found , sometimes the form class of the word can be in terred . Suffi x rules are applied , and the dictionary lookup is repeated with the new suff-Ix-stripped word .
For example, ti1L word test ing r�.:quires the rule, locate the suHix ing and remove it; whereas the word C/1'/a�)IZ ing req uires the rule, locate rht.: suffix ing and replace it with e. The su fti x rules and the dictionary lookup are recursive to handle words that end in m u l tiple suffixes such as endless!)'.
If the word is not in the d ictionary, rbe
LTS
th read performs a decomposition of the word using morphs.
D ECtal k uses <I morph table to look up the phonemic representation of portions of words. A morph :tl\\'a\'S maps onto one or more English words and can be represented by a letter string. Morphs generally consist of one or more roots that may contain aftl xcs and suf fixes. AJthough ne\\' words m:�y tl·equcn th' be added to a language, new morphs :1re rarelv added . Thcv arc essential ly sound groupings that make up many of the words of a Lmguage . DEC:ta l k contains a table with hu ndreds of morphs and their p honemic representa tions. Either a single character or a set of characters that resu lts in a single phoneme is rcrcrred to as <I grapheme. Thus this portion of the lcrrcr-ro-sound conversion is referred to as the grap heme-to-phoneme translator. Figu re 4 shows the architectu re of the l
. .TS thread .
J\tiorphcmcs are abstract grammatical units and \\'ere originally ddined to descri be words that can be s�.:g me nted , such as tall. tollc1; and tallest. The word tallest is made ti·om the morphemes tall and est. The word tccnt decomposes i n to the morphemes go and
PAST.
Thus a morpheme does not nccessari lv m:�p directly onto :1 derived word . Many of the pronuncia tion rules arc based on the morphemic represe ntations of words.
Manv morphs have multiple phonemic representa tions that
G i l l depend on t.:ithcr word or phonemic con text. The correct phonemic symbols a1T determined by morphophonemic rules. For cx:�mplc, plur<11 words th<H end in the morpheme s arc spoken bv appending ei ther the s. the z. or the eh z plural morphemes (expressed as Arpabcr phonemic sym bols) :�t the end ofrhe \\'ord . '
Which allomorph i s used depends o n the tim! phoneme of the word . A l lomorphs arc morphemes with altern:t tc phonetic t(>rr11S. For :llH>thcr example requiring a morphophone mic rule, consider the t-ina l phoneme of t h e word the when pronouncing "the apple," and "the boy."
After :�ppl\'ing manv morphophonemic ru les to the phonemes, the
LTS
thread pertim11s s\'l l abitication,
:�pplies stress to cert:�in syllables, Jnd pcrr<.mns allo phonic recod ing of the phoneme stream. The
LTS
TEXT -
CLAUSE
SEPARATION
-
TEXT
NORMAL IZATION
-
DICTIONARY
.--
-
GRAPHEM E-TO·
PHONEME RULES
L
SYLLABIFICATION f.--
STRESS
-
ALLOPHONIC
S U BSTITUTION
-
P H O N E M E S
Nore that the gr<lpheme-to-phom:me rules �re used onlv if the dictionary .loo kup bi ls.
1 2
Fig ure 4
Block Diagram of the Letter· to-Sound Processing Thread
Di�ir.tl Technical )ournJI Vol. 7 No. 4 1 995
thread groups phone mes into syllables, using tables of legal phoneme clusters and special rules. The syllab ification must be accurate, because the ITS th read applies stress berween syl lable boundaries.
The
LTS
thread then assigns either primary stress, secondarv stress, or no stress to each syllahk. The stress ru les are applied i n order. Thev assign stress only to syl lables that have nor had stress previously assigned. These rules rake i nto account the nu mber of syllables in a word �•nd the positions of aftixes that
\\Tt-c t()lJnd d u ring morph decomposition of a word.
Al lophonic rules arc the last r u l es the
LTS
thread applies to the phoneme stream . These arc really pho netic rules. Most a l l ophonic rules are described as foll ows: "if phoneme A is fol l owed bv phoneme B, then modi t\· ( o r d e lete ) phoneme A ( or B ) . " Most all ophonic ru les arc not applied across morpheme bou ndaries. These rules hand le many specific cases; for example, the p i n the word spit is aspirated, whereas the p in rhe word pi! is not. The s phoneme modifies the articu lation of rhc p. The s phoneme is differe nt in the words slop and s/ rC!el because the r sound is antici pated and mod i fies the s in the word street. This l ast example is called distant assimiL:nion .
The
LTS
thread passL·s the p honemes that include du rations and lexiul information to the prosodi c and phonetic processing thread . Tone, dial, index mark, and svnch ronization messages arc passed unmod i fied through the
L TS
th rcKi .
The Phonetic and Prosodic Processing Thread
The phonetic and prosod ic processing ( P H ) thread, shown in Figure 5, converts the phoneme stream to a series of vocal tract con trol comma nds. Both prosodic ru les and addi tional phonetic rules arc applied to the input phoneme stream -" Prosody refers to clause based stress, intonation, and voice quality in speech .
'Norcis �•rc stressed to add meaning to a clause. Stress is achieved by i ncreasi ng one or more of either the pitch, the duration, or the amplitude of an ttttenmce . The phonetic rules handle coarticu lation effects and adjust phoneme d urations based on the t<m11 class, rhc cl ause posi tion, and the speaking rate . One example is a rule th:tt increases the duration of the final stressed p honeme i n a clause . Addition�•! context-dependent phonetic coarticu lation ru les can adjust the d u rations ofphoncn��:s or delete them .
PHONEMES
PHON E MES
Fig ure
5
The
Phom:ric and l'rmod ic Pnxess i n g; Thread
VTM
CONTROL
COMMANDS
The correct application of stress, l i ke intonation, requires understanding, so DECtalk Sof'rware gener ally applies syllabic stress only as part of an i n tonation contour across a c l ause. Intonation contours are gen erated by tixed rules. In most clauses, the pitch rises J.t the start of the clause and t:llls at the end of the clause .
This basic t()rm is changed tor questions, prepositional phrases, exc!J mations, compound nouns, and n u m bers. This intonation i s also changed based on the syl labic stress assigned by the
LTS thread . The PH thread can also process pitch control symbols that are pi<Ked i n - l ine with text. These pitch commands are parsed in the command thread and pass through the
LTS thread .
The P H thread uses each phoneme symbol and its context to generate anv a llophonic variation of the phoneme. The resulting al lophone sym bol indexes i nto one of two tables, one table tor each gender. E:1ch allophone symbol indexes a set of parameters that includes voicing source amplitude, noise source ampl i tude, formant frequencies, and formant bandwidths.
These, along with voicing source pitch �md a number of fi xed speaker-dependent parameters, make up the
VTM parameters. A new set of parameters is generated tor every 6 .4 milliseconds of speech . The
VTM thread uses these parameters, which are collectively cal l ed J voice packet, to generate the speech waveform.
In addition to sending voice packets to the
VTM thread, the PH thread can send a spe:�ker packer to select a new speaking voice. The voice is selected either by an in-line text command or by the appl ication cal l ing a specific A P I fu nction . T h e PH thread has fixed tables of parameters tor eac h voice . There are many voice parameters, but some of the more interesting ones include the gender, the average pitch, the pitch range, the :tsseniveness, the breathiness, and the for mant scale factor. The gender is used by some of the
PH rules and bv the PH thread to select the table used to generate voice packets. The average pitch and the pitch range are used by the PH thread to set the pitch characteristics tor the
VTM
's \'Oicing source.
The assertiveness parameter sets the rate of fal l of the pitch at the end of a clause. A high assertiveness t:Ktor res u l ts i n a n emphatic voice. The breath iness parameter sets the amou nt of noise that is mixed with the \'Oiced path signal . The formant scale factor eftcc tively scales the size of the speaker's trachea.
Tone, dial, index mark, and synchron ization nJCs sages arc p:�ssed un modi fied through the PH th read .
The Vocal Tract Model Thread
The Vocal Tract Model
( VTM ) th read processes speaker packets, voice packets, tone messages, and synchronization messages. Speaker packets set the speaker-voice-dependent parameters of the
V T M .
Digir·,1l Tec hnical journal
Vol.
7
No.4 199S
1 3
One of these, the formant scale factor, is multiplied by the first, second, and third tormant frequencies in each voice packet. Other parameters include the values for the frequencies and bandwidths of the fourth and fifth formants, the gains tor the voiced path of the
VTM, the ti·ication gain for the unvoiced path of the
VTM, the speaker breathiness gain, and the speaker aspiration gain.
Each voice packet produces one speech fi·ame of data. The output sample rate tor D ECtalk Software is either 8 ,000 Hz or 1 1 ,0 2 5 Hz. For each of these sample rates, a fi·ame is 5 1 and 71 samples respectively.
Each voice packet i ncludes fi-equencies and band widths for the first, second, and third formants, the nasal antiresonator frequency, the voicing source gain, and gains f()f each of the paral lel resonators. Figure 6 shows the basic architectu re of the
VTM .y The VTM, in conjunction with the
PH rules, simulates the speech organs.
The
VTM consists of two major paths, a voiced path and an unvoiced path . The voiced path is excited by a pulse generator that simu lates the vocal cords. A num ber of resonant filters i n series simulate the trachea.
These cascaded resonators simu late a cascade of tu bes of varying widths. 10 A nasal filter in series with the res onant tube mode l simulates the dominant resonance and antiresonance of the nasal cavity. 1 1 The cascade resonators and the nasal filter complete the "voiced " path of the
VTM.
Unvoiced sounds occur as a result of chaotic turbu lence produced when breath passes through a con striction . This turbulence is difficult to model. In our approach, the
VTM matches the spectral magnitude of filtered noise with the spectral magnitude of the desired unvoiced phoneme ( al lophone). The noise source is realized by fi lteri ng the output of a uniform distribution random number generator. Unvoiced sounds contain both resonances and antiresonances.
Another approach to obtain an appropriate fi·e quency characteristic is to fi lter the noise source signal using a series of parallel resonators. A consequence of
DIFFERE NTIA TED
PULSE
GENERATOR
PITCH AND GAIN
GAIN
NOISE SOURCE
VOICED PATH
FILTERS
FORMANTS,
BANDWIDTH,
AND GAINS
UNVOICED
PATH F I LTERS
Figure
6
Basic Archi tecture of rhe Vocal Traer Model
SPEECH putting resonators in parallel is to create antircso naoces. The positions of these antiresonances are dependent on the parallel formant frequencies, but it has been empirically determ ined that this model pro vides more than enough degrees of freedom to closely match the spectral magnitude of any unvoiced sound .
The noise source generates fricatives, such as s. plosives, such as p, and aspirates, such as h. The noise source also contribu tes to some voiced sounds, such as h, J<, and z.
The noise source ou tput may also be added to the input of the voiced path to produce aspiration . To gen erate breathy vowels, the parallel formant fi·equencies are set equal to the cascade fclrmant t!·equencies.12
The radiation characteristic of the lips approximates a differentiation (derivative) of the acoustic pressure wave. Since a l l the fi lters in the
VTM are linor and time invar ian t, the rad iation effects can be incorpo rated in the signal sources instead of at the output.
Therd(Jre the glottal source (pulse source) prod uces differentiated pulses. The differentiated noise signal is the filtered first difference of a uniform-distri bution random number generator.
The DECtalk Software
VTM (also known as the
Klatt Synthesizer) is shown in Figure
7.
The italicized terms arc either speaker-dependent parameters or con stant values. All other parameters are updated every frame. Depending on the system mode, the audio samples generated tor each fl-ame arc passed to the output routine and subsequently are either queued to the audio device, written to a wave audio file, or writ ten to a buffer provided by the application. Alter gen erating a speech frame, the
VTM code increases the audio sample coun t b)' the frame size. This cou nt is sent to the synchronization thread whenever a syn chronization symbol or an index mark is received by the
VTM thread. The count is reset to zero at startup and whenever the text-to-speech system is reset.
Tone messages are processed by the
VTM thread.
Tone messages are for single tones or
DTM t-: signals.
Each tone message includes two fi·eq uencies, two amplitudes (one tor each h-cquency), and one duration.
For a single tone message, the amplitude for the second frequency is zero. Tone synthesis code generates tone frames and queues them to the output routine. The first
2
milliseconds and the last
2
milliseconds of a tone signal arc multiplied by either a rising or a filling cosine-squared shaping function to limit the out-of band pulse energy. Each tone sample is synthesized using a sinusoid look-up tablc.1'
The Synchronization Thread
The synchronization thread is idle un less the
VTM thread t(xwards a synchronization symbol message or an index mark message. Bot h messages contain the current audio samp le count. The i ndex mark message
14
Digiral
Tct"hnic1l J ournal
Vol .
7 1o.
4
1 995
PITCH
VOICING
TILT
VOICING
GAIN
Fz
SPEAKER BREA THINESS x BR EATHINESS
DIFFERENTIATED
NOISE
NOISE
MODULATION
0.5 OR 1 . 0
SPEAKER
FA/CATION
SPEAKER
FRICATION
SPEAKER
FRICATION
SPEAKER
FRICATION
SPEAKER
FRICATION
KEY:
F FREQU ENCY
8
BANDWIDTH
A AMPLITUDE
G GAIN
Note: 11al icized terms are either speaker·dependent parameters or constant values.
All other parameters are updated every lrame.
SPEA KER FRICATION
" FRICATION
Figure 7
The DECtalk Software Vocal Tract Model (a lso known as the KJatr Synthesi zer) also contains an index mark number from 0 to
99.
After receiving one of these messages, the synchro nization thread periodically pol ls the audio thread until the indicated audio sample has been played. If the message contained a synchronization symbol, an event is set that unblocks the command thread. If it is an index mark message, the synchronization thread sends the index mark nu mber back to the application .
For the Digital
U N I X operating system, this number is returned by calling a callback fu nction that the appli cation specifies when D ECtalk Sofu.vare is started . For the Windows NT operating system, the Send Message function is used to return the index mark number to the application . The message is sent to a window procedure specified by the window handle that is pro vided when the text-to-speech system is started.
The Audio Thread
The audio thread manages al l activities associated with playing audio through the computer's sound hard ware .
An audio
API insulates DECta lk Software from the differences between operating systems. The audio
API communicates with the audio thread . The VTM thread ca lls an audio
API queuing function that writes samples to a ring bu ffer that is read only by the audio thread. The audio thread opens the audio device afi:er approximately 0.8 seconds of audio samples have been queued and closes the audio device when there are no more samples to play. If the number of audio samples i n the queue is too small to cause the audio device to be opened , and the How rate ( measured over a 1 00millisecond interval) into the audio ring buffer is zero, the timer thread will send the audio th read a message
Digital Technical Journal
Vol . 7 No. 4 1995 15
that causes the audio device to open and start playing audio. When audio either starts or stops playing, a message is sent to the application.
For the Digital U N IX operating syste m , the audio thrc1d is an interbce to the lmv-lcvcl audio fu nctions of the M u ltimedia Services t(>r Digital U N I
X
( M M S ) prod uct. M l\r!S provides a server t o play audio and
,·ideo.
For the \Vindows
NT operating svstem, the i mple mentation also uses the system low-level :l lldio func tions, but these fu nctions inrcrbce directly with a system audio dri, er. The audio API prm·idcs
C 1 fXlbi li ties to pause the audio, resu me paused audio, stop audio fi·om playing and cancel Jli queued audio, gct the
Judio volume level, set the audio volume l evel, get the number of audio sampl e s p laved, get the ;�.udio r(m11<1t, and set the audio rormat. An i n- line plav command can be used to p lay audio ti les. D ECt,llk Sotiware uses the get r(>rmat and set ronnat audio capabilities to dynami cally change the audio format so it can play an audio file that has
J rormat different rrom the torm;lt generated bv the
VTM.
DECtalk Software API
In the mid-l 980s, researchers at Digital's Cambridge
Research La b ported the DECtal k tnt-to- speech
C language- based code to the ULTR1X operating syste m . The command, L'TS , PH, and VTJ\!1 portions of the system were difterent processes. The pipes were implemented using sta n dard UNIX r;o h:llld les, stdin and stdout. These, along with an audio dri\'Cr process, were combined into a command proced ure. This
Ta ble 3
DECta l k Software CPU Loadin g versus Processor SPE Cmarks
System
Alpha AXP
1 50 PC
Alpha Station
2 50 4/266 workstation
DEC 3000
Model 800 workstation
DEC 3000
Model 900 workstation
AlphaStation
400 4/233 workstation
AlphaStation
600 5/2 66 workstation
XL 590 PC
Clock
(M Hz)
1 50
266
200
275
233
266
90
Processor
Alpha
2 1 064
Alpha
2 1 064
A l pha
2 1 064
Alpha
2 1 064A
Alpha
2 1 064A
Alpha
2 1 1 64
Pentium
Secon dary
Cache
( M B)
5 1 2
2,048
2,048
2,048
5 1 2
2,048
5 1 2
SPECi nt92 SPECfp92
80.9 1 1 0 . 2
1 98.6
1 38.4
230.6
1 57.7
288.6
Unkn own
262.5
1 88.6
2 64 . 1
1 83 . 9
428.6
N/A system lacked many of the rules and katures found in DECtalk Software todav, but it did demonstrate that real-time speech svntiKsis was possible on a work station. Bdore this time, D EC:talk req uired speciali zed
Digital signal- processing l
1:1 rdware t(_n r<:al -time oper ation . 14 On a D EC:sta tion Model 5000/25 work station, the text-to s peech impl e m e m ation used 65 percent of t he CP U
.
If the output sa mple rate of this system had been raised ti·om 8,000 Hz to I I ,02 5 H;,, the h ighest-q uality rate provided by DECt:dk Software, it would h:l\'e l oaded approximately 89 percent of the C P U . Wor kstation text-to-speech S\'nthcsis, while possible, was still \'cry expensiYe .
The powe r of the Alpha C P U has changed this.
Today, many copi es of D ECtalk SoftwJrc can run simultaneously on Alpha- b:�sed systems. Speech S)' n thcsis i s now a
,· i abl c mu ltimedia form. This change created the need for a text-to-speec h A P I . Table 3 shows the D ECtalk Software CPU lo:td t(x \'arious computers.
On
Alpha systems, the perf-ormance of D EC:talk
Software depends primari l�· on the S PEC:mark rating of the computer. A Jesser considerJtion is the sec ondary cache size . System bus bandwidth is not a lim iting factor: The com bined data rates f(n the text, phonemes, and audio are ntremeh· low relati\'C to modern bus speeds, e\'cn when running the maxim um nu mber ofrea l-time text-to-speech processes that the processor can support.
The A I' I we h
J v e developed is the rcsul t of collabo ration between several organi zations within Digita l : the Light and Sound Group, t h e Ass istiYe Tech nol o b '
Group, the CJm bridge Resc:trch LJb, and the Voice
Audio
Rate
(kHz)
1 1 , 025
1 1 ,025
1 1 , 025
1 1 , 025
1 1 ,025
8,000
1 1 ,025
Total CPU
Load (%)
8
2 .4
5
3
3
24
1 6
Dig;iral Tcchnic1\ )mii'Jlill
Vol .
7 No. 4
1 995
and Telecom Engineering Group. vVc had two basic requ i rements: We wanted the API to be easy to usc and ro work with any text-to -speech svstem . VVh i le creating the API, we defined i nterfaces so that future improvements to the text-to-speech engine would not requ ire any A PI C;:tiiS to be changed . ( Customers ri·own on prod uct updates that req uire rewriting cod e . )
Some decisions were contrm-crsia l . Some contri butors kit that the text-to-speech svstem should return speech samples o n lv i n memory bufkrs, and the application should shoulder the burden of i n terracing to the workstation's audio su bsystem. The other approach was to support the standard workstation audio ( which is platform depend ent) and to prol'ide an API call that switched the SI'Stem i nto
�l speech- to memory mode. We selected the latter approach because it simplifies usage for most appl ications.
The API Functions
The core tex t - to-speech A l) J functions arc the
TextToSpcechStanup ti.mction, the TextToSpccchSpcak ti.mction, and the TcxtToSpccchShutdown ti.mction.
The simplest application might use only these three functions.
AJ I applications using text-to-speech must call the
TextToSpccchStartup function. This function creates all the f) ECta lk system threads and passes back a han die to the text-to s peech system. The handle is used in su bsequent text-to-speech API calls. The stJrtup func tion is the onll' API function that has different argu ments r(n the Digi t<l l U N I X ;:md the Windows NT operating systems. This is necessary because the asyn cbronous reporting mechanism is a cal lback function rcJr Digital U N I X and is a system message r(Jr Windows
N T .
The Tcx tToSpeechS hutdown h.mction h·ces a l l
Sl'stem resources and shuts down t h e threads. This would normally be called when closing tht.: ::�pplication .
The Tcx tToSpt.:cchSpcak fu nction is used to q ueue text to the svstc m . ! fan entire c lause is not queued, no output \\·i l l ot.:cur un til tht.: c lause is completed bv q ueuing addi tional text. A special TTS_FO RC:E para meter may be supplied in the ti.mction call to torce a clause boundary. The
TrS_FORC:E parameter is nec essary tcx appJ iutions that have no control ol'er the text source :md rhus can not guarantee that the final text forms a complete c lause.
The tcxt-to-spt.:cch API pr01·ides three audio output control hmctions. These pause the audio output
(TextToSpt.:cch Pausc ) , resu me output ati:cr pausing
(TcxtToSpccchRcsumc ) , and reset the tnt - to-speec h svstem (TcxtToSpeec hRcst.:t). The reset fu nction dis cards all queued text and stops all audio output.
The text-to-speech API •1lso provides a special syn chronization fu nction (TcxtToSpccchSync ) that blocks until all prcviouslv queued tnt has been spoken . This
A PI call ma�· not return tcx davs if a sufficit.:nt amount of text is queued. ( Index 111�1rks pr<widc non blocking synchroniz�Hion . )
Tbe A P I supplies fu nctions to both load
(TextToSpt.:cchLoad UserDictionary) and unload
(TcxtToSpccch UnloadUscrDictionan·) an applicnion cletined diction�uy The dictionarv contains words and thei r p honemic representations. The de1·elopcr creates a dictionary using a window-based user-dictionary too l . This tool can speak words and their phonemic representations. lt can also com·ert text sequences ro phonemic scqu t.:nces. This last kature hTcs the cln·cl oper from ha1·ing to memorize and usc the D ECtalk
Soti:ware pho!Jt.:mic symbols.
Additional functions se lect the speaker voice, con trol the speaking rate, control the language, dctt.:rmine the system capabilities, and return status. The status
API function can indicate if the svstem is currcmly speaking.
Special Text-to-Speech Modes
DEC:tal k Soti:w:trc has three spcci::�l modes: the speech to-wave tile mode, the log-rile mode, and the speech to-memory mode. Each mode has two complcmcn t:try calls, one to enter the mode and one to exit.
\Vhen i n the speec h -to-wave ti le mode, the system writes all speech samples to a wave audio rile The tile is closed when exiting this mode. This is useful on slower
I ntel systems rhat cannot perform real-time speech syntbesis. The log- tile mode c:tuscs the system to write the phonemic sym bol output of the LTS thre:-�d to a ril e . The l ast mode is the spcec h - ro- me morl' mode.
Mer entering this mode, the applicnion uses a special
API cal l to supply the tnt-to-speech system with memory bufkrs. The text -to speech system writes synthesi zed speech to tht.:sc bu ffers and returns the bu ffer to the application . Tht.: buffers arc retu rned using the same mechanism used tor index marks, a call back fu nction on the D igital U N I X opcr:-�ting sys tem and a system message on the VVindows NT operat ing system . These bufkrs mav :tlso retu rn index marks and phont.:mic svm bols and tht.:ir durations. lf tl1t.: text tO-speech system is in specc h - to- memorv mode, cal l ing the reset fu nction ca uses all buffers to be returned to the application.
Porting DECtalk Software
The
DEC:talk
PC code used a simple asst.: m bll' lan guage kernel to manage the th reads. The existence of threads on our target p latr<.mm simplified porti ng the code. The thread fu nctions, signals ( such as condi tions or e\·cms), and mutual t.:xcl usion objects :� rc dif rl:rcm f(x the Digital U N I X and the Wind ows NT operating svstcms. Since these fu nctions occu r mai n ly in the pipe code and the audio code, we maintain d i frerent versions of code r(x e::�ch svstcm . The message -passing mechanism r(Jr Windo11·s NT Ius no
Vol . 7 l'o. 4 1995 1 7
equivalent on Digital U N I X; therdore part o f the API code had to be difterent. The command, LTS, and
P H threads arc all common code tor Digital U NIX and Windows
NT.
Most of the
VTM thread is also common code.
Porting the code tor each thread req uired putting conditional statements that dctinc th read entry points into each module t()r each supported operating system.
We also had to add special code to each thread to sup port our API cal l that resets the text-to-speech system .
The reset is the most complicated API operation, because the data piped between threads is in the torm ofvariable-lcngth packets. During a reset, it is incorrect to simply discard data within a pipe because the thread that reads the pipe will lose data synchronization.
Therefore a reset causes each thread to loop and dis card all input data until al l the pipes arc empty. Then each thread's control and state variables arc set to a known state . In many complicated systems, n::setting and shutting down arc the most complicated parts of a control arc hitecture. System designers shou ld incorpo rate mechanisms to simpli�' these functions.
The VTM code is much shorter and simpler than the code i n either the L T S or the PH t h read, but it is by br the largest CPU l oad in the system . The
DECtalk PC hardware used a specialized Digital Signal
Processor ( DS !') tor the
VTM.
The research
VTM code (written in the C language ) was rewritten to be sample-rate independ e nt. The fil ters were all made in-line macros. With this new
VTM, the D EC:talk
Software system loaded an Alpha
AXP 1 50 PC product
3 1 percent. After rewriting this code using floating point arithmetic and then converting it to assembly l:mguage, D ECtalk Software l oaded the processor less th:�n 8 percent. (Both tests were condu cted at an
1 1 ,02 5 - H z output sample rate . )
There are several reasons a Hoating-poi nt VTM runs f:1stcr than an integer VTM on an Alpha system.
An integn VTM req uires a separate gain tor each !l iter to keep the output data within the filter's dynamic range.
For a Hoating-poim VTM, the gains of all cascaded filters are combined into one gai n . The increased dynamic range <ll iows com bining parts of some filters to red uce computations. Also, floating-point opera tions do not require addition�11 instructions to perform sca ling. The processor ach ieves greater i nstruction throughput because it can dual issue floating-point instructions with integer instructions, which are used tor pointers, indices, and some loop counters. Finally, the current generation of Alpha processors performs some fl oating-point oper:-ttions with less pipeline
1:-ttcncy than thei r equivalent integer operations ( note the SPECtp92 and SPECint92 ratings of the current
Alph:� processors listed in Table
3 ) .
The integer
VTM is bstcr than the tloating-point
VTM on Intel processors, so we maintain two versions of the VTM . Both versions support multiple sample rates. The pitch of the glottal source and the frequen cies and bandwidths of the tilters arc adjusted for the output sample rate. When necessary, the filter gains ::trc adj usted . These extra calculations do not add much to the tota l time used by the VTM because they are per t(1rmcd only once per ti-ame.
Possible Future Improvements to DECtalk Software
The Assistive Technolot,'Y Group continues to improve the l etter-to-sound rules, the prosodic rules, and the phonetic ru les. Future implementations could usc object-oriented techniques to represent the dictionar ies, words, phonemes, and parts of the
VTM.
A larger dictionary with more symactic int<mnation can be added . There h<lS even been some discussion ofcombin ing the L
T
S and PH thre::tds to make more efficient usc oflcxical knowledge in PH . The glottal waveform gen erator can be improved . Syntactic parsers might prm·ide the information required tor more accurate i ntonation.
Someday, semantic parsing (text u nderstanding) may provide a m�1jor impro\'cment in synthetic speech i nto nation. Researchers both within and outside of Digital arc investig:lting these and many other areas. It seems l i kely that the American English version of DECtalk
Software wil l continue to improve over time.
Summary
D ECtal k Software provides natural -sounding, highly int elligible text-to-speech synthesis. It was developed to pcrtorm on the Digit:�!
UNIX
operating system on
Digital 's Alpha- based platt<mm and with Microsoft's
Windows
NT
operating system on both Alpha and Intel processors . It is based on rhe mature DECtalk
PC
hardware prod uct. DECtalk Software also provides an easy-ro-use API that allows applications to usc the work station 's audio subsystem ,
to
create wave auclio ti les, and to write the speech samples to application -supplied memory bufkrs.
An
Alpha-based workstation can run many copies of DECtalk Sothvare si multaneously.
D ECtalk Sothvare u ses a d ictionary and l i ngu istic rules to convert speech to phonemes. An application supplied d ictionary can override the dctault pronunci ation of a word . Prosodic and phonetic ru les modifY the phoneme's attributes. A vocal tract model synthe sizes each phoneme to prod uce a speech wavdorm .
The result is the highest-quality text to speech. The
Assistive Technology Group conti nues to improve the
DECtalk text-to-speech algorithms.
1 8
Digital Tc(hnicll j ou rn:ll Vol. 7 No. 4 1995
Acknowledgments
I wi s h to acknowledge and thank all the mem bers of the D ECtalk Software project and additional support staff. Bernie Rozmovits, our engineering project leader, was the visionarv tor this entire eft(Jrt. He con tributed most o f our sample appl ications on Windows
NT, and he also wrote the text-to-speech
ODE server.
Krishna Mangipudi , Darrell Stam , <lJKi Hugh Enxing implemented DEC:talk Software on the Digit1l U N I X operating svstem. Thanks to
Bill Scarborough who d i d a great j o b o n a l l of our docu mentation, particularlv the on-line help. Special thanks to
Dr. Tony Vitale and
Ed Bru ckerr, ti·om Digital's Assistive
Technology
Group. They both were instrumental in developing the D ECta l k funily of products and are continuing ro improve them. Without their eftixts and support,
DECtalk Sofuv;1re could nor ex ist. Tom Levergood and T. V. Raman at Digi tal 's Cam bridge Research Lab helped test D ECtalk Software and provided many sug gestions and improvements. Tha n ks also to the engi neering manager t(Jr Graphics and Mu ltimedia, Steve
Seu fert, who continues to support our efforts. Final l y, we are all indebted to
Dennis Klatt who was the cre ator of the
D
ECtalk speech synt hesizer and to a l l the other developers of the original D ECtalk hardware products.
References
I . C. Fam, A coustic 77teOJy u/ SjJel!ch Produclion
(The Nethe r b nd s : Mouton and Co. N .V., 1 960).
2 .
C.
Schmanclt,
\'r
J ic e Communicatiuu n·ith Computl!ro·
( New York: Van Nostrand Rein hold , 1 994 ) .
3 .
)
. Allen, M . H u n n icutt, an d D . Klatr, From Text to Speech · The 1Vfl'f'alk System (Cambridge, Mass . :
Cam bridge U nivnsitv Press, 1 98 7 ) .
4 . ). F l a na ga n
, SpeedJ A na(J'Sis. Sruthesis, aud Percep tioll, 2 d ed . ( New York: Springer-Verlag, 1 97 2 ) .
S .
D .
Pisoni,
H .
N usbaum, and B. Greene, " Perception of Synthetic S p e e c h Generated by R u l e ," Pruceedinp,s of the
11:1:'1:. vol . 73, no. 1 1 ( 1 985 ): 1 665-1 676.
6. A. Vit a le c1 11d M. Di,·av, "Algorithms for G r ap h e me
Phoneme Translation in preparation ). french a n d E n g li s h " ( i n
7 . V.
F r om ki n cmd R
. Rod man, A n
Intro d u c tio n lo
Lan8uage. 2d ed. ( New York: H olt, Rinehart, and
Wi nston ,
J 97R ) .
8 . D . Klarr, " Review o f Text-to-Speech Conl'ersion
English ," jollrllal <�/lh<' Aco11sticul Society tor ojA meriut. vol . 82, no. 3 ( 1 98 7 ) : 737-79 3 .
9 . D. KJarr, "Software tor a Cascade/Parallel Fonn<111t
Synthesizer," journal of the A coustical Societv of'
A meriw, vol. 67 ( 1980): 97 1-975.
1 0 . L. Rabinn :�nd B. Gold , 7heor)' ami Application o/
Digital Sl,� nal Processin�; ( London: Prentice Hall,
1 9 7 5 ) .
1 1 . L. Rabinu a n d
Speech Si/.!,llllls
R. Scbater,
D(�i!al Processiug of'
(
Lo nd o n
:
Prentice Hall,
I
978 ).
1 2 . D.
Klatt and l...
Klatt, "Analvsis, Synthesis, and Pcn.:ep ti o n of Voice Quality Variations among Fe male and
Male Talkers," Journal of' the Acoustical Societ v
<!!'
America, vol .
87, no.
2 ( 1 99 0 ) : 8 2 0 -857.
1 3 . J. Tierney, " D igi t a l frequ e n c v Svnthesizcrs," C:haprcr
V of Fre<JIIi!IILJ' Sl'llihesis· Techll iriiii!S aud Applica tions.
] .
Gorski - Popel , eel .
( New York: I E E E
Prc�s,
1 97 5 ) .
1 4 . E. 13 ruckerr, M. l'vli n ow, a n d W . Tcrschncr, "Three
Tiered Software c111d VLSI Aid Development System ro
R ead ·rcxt Aloud,"
1:'/ectrunic
(A pri l 2 1 , 1 98 3 ).
Biography
William 1. Hallahan
Bill
Hallahan is a member of the
L ig ht and Sou nd Grou p, parr of Software: Engi neering for the WoJ·kst<Hion B u si n ess
Segment . Previously he worked in the lmage, V o i c e
Vide o Group on signal -processing c l lg or i r h m s and rewri ting of the DECtalk vocal tract m o del . B c f
(
1 r e j
, oi
<ll1d the n i ng
Digital in 1 992, he was e m p l oy e d at Sande1·s Associates for
1 2 vears, where he de v e l o pe d and i mplcmenred algorithms that per for m e d si gn
.1
l analvsis, si g n a l ckmodu lation , cmd numerical methods.
Uni versity of
13 i l l received a fl.S.E.E. fi · om the
New Ha m p s h i r e in
19 80. He is co-author of a patent �pplication tor a sp e c i fic colo r
sp a c e conversion algorithm used in video multimedia applications.
Digital Technical journal
Vol .
7 No. 4 1995 19
I
Kenneth
W. Conell
Robert
A.
Ulichney
The
J300
Family of Video and Audio Adapters:
Architecture and
Hardware Design
The
BOO family of video and audio adapters provides a feature-rich set of hardware options for Alpha-based workstations. U n like earlier attempts to i nteg rate ful l-motion dig ital video with general-purpose computer systems, the architecture and design of
J300 adapters exploit fast system and
1/0 buses to al low video data to be treated l i ke any other data type used by the system, independent of the graphics su bsys tem . This paper describes the arch itecture used in
BOO products, the video and audio features supported, and some key aspects of the hard ware design. In particular, the paper describes a simple yet versati le color-map-friendly render ing system that generates high-quality 8-bit image data.
The ()\'Cral l arch i t ec tura l design goal r()r the J300 b m i h· o f \
· i d e o and audio ad a pters \L1S to p r m id c the h :1rd\\'are su pport nc c c s s a n
· to ai iO\\
. the integration of broadcast \'ideo into workstations. The three p ri m a r y objecti v es were as td low s
:
( 1 ) d igitized \'ideo data should be n·catcd the same as :111y o t h er data type in tlle system; ( 2 ) the video and the gra p h i c s su bsvs tem de si gns sho u l d b e c o mp leteh
· i ndepend e nt of e a c h other; and ( 3 ) Jn\· hard\\·arc d esigned sh o u l d be l ow cost.
Digital has imple me nted the
]300
:\rch i tecrurc in three p roducts
:
Sou nd & M otion } 3 00, FullVideo
S upr eme JPEC, and ful lVideo S u p r e m e .
'
Tl1c Sound
&
Motion
J300
( rc k r r cd to in this paper s im p ly as t h e
} 30 0 ) was the fi rst pr od u c t designed \ \
' it h this archi tecture a n d is th e pri man· focus of t h is paper. The
Fui iVideo Supreme
] PEG :111d FuJ IVidco S u p r e m e products arc based on the same d e sig n d:1 t:1 base as the
J 300.
Thev difkr ri
· o m the
}300 in the bus supported
( they support the per i ph e r a l componcllt interconnect
[ PC r J bus) and the lack of audio suppmr. Add ition:1l lv, the Fu iiVideo S u pr eme product docs not include hardware compression/ deco
The mp ress i on c in
: u i t rv .
J
3 0 0 b r i ngs :1 \\·ide ra nge or \·ideo :11ld
:
1 ll d i o capabilities to m
J chines based on Digital's
TU RBOchannel 1/0 intercon nect. An:1log b video r o a dc
1 s t can be digit i z ed, demod u lated , and rendered tor d i s p l ay on any graphics device . The } 3 0 0 prm·ides h:1rdware video compr es sion and de c om p r e ss ion to accelerate Jpplicuions such as videoconkrencing.
The
J300 supports analog broJdcast \·ideo o u t p ut ri
· o m either comp re ss e d o r unco mpr css e d \·ideo riles.
Au d io support i ncl u d e s
:1 gener a l
p ur p ose
, d i gital signal processor (
DSI)) to assist i n the real-time man a g c mellt of the audio streams and t(>r a d
\
' J nced pro cessing, such as compr e s si o n
, d ecompression, an d echo cancellation . Audio input and output cJpa bi include stereo :ulJ!og ljO , digital l it ies audio 1/0, an d a he:1d p hone/m icrop hone jack. Analog a u di o c:1n be digiti zed to
1 6 bits per sam ple at
:1 r�lte of u p to
48 ki lohertz ( kH z ) .
Th i s pap e r be g ins with a n 0\'e rYie\\. o f some tc rmi nology comrn on l y used in the field o fb r oad cJ s t video.
The paper then presems the e vo l ution and desi g n of the J 300 arc h i tectu re , i nc l u d ing Se\Tr:J! kel' enabling
2 0
Di�ital Tcclmic.1i ) umn;ll \'ol. 7 ]\;o. 4 1 99�
tech nologies and the logical 1·idco cbta paths a1·ailable.
Next ti:>llows a discussion of the hardware design phase of the project and the tradc-ofts made to reconcile expectation and implementatio n . Derailed descrip tions arc devoted to speci fic areas of the design, including the 1·idco 1/0 logic, the AccuVidco render ing path, and the 1·ideo and audio d i rect memorv access
( D M A ) i ntcrbccs.
Video Terminology Overview
Three fu ndamental standards are in use worldwide tor representing what is rdcrrcd to in this p<lpcr as broad c:tst 1·idco: the National ( U S ) Tclc1·ision Svstem
Committee
( NTSC ) recommendation, Phase Alternate
Line
( PA L ) , and Scqucmiel Cou lcur <ll'tC i\1lcmoire
( S E CA M ) .
The standards d i ftcr i n the n u mber of horizontal l ines in the d isplay, the vertical rdrcsh rate, and the method used f(>r e ncoding color i n t( mnatio n .
North America a n d Japan usc t h e 5 2 5 - l i ne, 60- hcrtz
( Hz )
N T S C f(mmt;
I'A L is used i n most of Europe; and
S ECAM is used pri marilv in France. Both the
PAL
<llld
S E CAM standards arc 6 2 5 · 1inc, 50-Hz SI'Stems.2
All three television standards split a n image or a ti·<1111t of video data into two tie lds, rekrred to as the even and the odd fields. Each tield conuins clltcrnate horizontal l i nes of the ri·amc.
The vertical rcti-csh rate cited in the pre�·ious p<lLlgraph is the ticld rate; the fr•1me r<lte is onc- h <llfofthat rate .
U n like computer displav systems that use red, green , ;md blue (
RG B ) signals to represen t color i n fi:mnation,
PA L and
S E C A M usc a l u minance chrominancc system, which has the three parameters
Y ( the luminance component), and [! and \/ ( the two chrominancc components ) .
NTSC uses a variation of
Y UV, where the (' and \ · component s arc rotated by
33 degrees and cal led
I and Q. YUV is related to RC B lw the f(>llowing comTrsion matri x : '
Y= 0.299 N + 0.587G + 0 . 1 14 13
U= - O . l 69 N - 0 . 3 3 1 G + 0.50013
1'= O . S OO N - 0 .4 1 9 G 0 . 08 l !J
A l l the d i fk rcnt stambrds limit the b<lndwidth of the chrominancc signal to between one-quarter and one-thi rd that of the l u minance signal . This limit is taken i n to account i n the digital representation of the signal and resu l ts in wiLlt is cal led 4 : 2 :2 YUV, where, f(>r every f()ur horizomally adjacent samples of Y there arc two samples of both U and \/. All three compo ncms arc sampled abo1·c the Nyquist r.uc in this t(x m•H with a signiticmt red uction i n the amoum of data needed to reconstruct the 1·idco i mage.
Various mod u l ation tec h n iques transti:m11 the sepa
Llte Y [!, and V com ponents i n to a single signa l , tvpi c:tl ly reterred to as composi te video. To increase the fidelity of video signals bv reducing the l u m i n ance chrom i nance cross ta I k caused by mod u b tion, the
S-Video standard h as been developed as an alternative.
S-Video, which rekrs to separate video, speci fics that the l u minance sign a l and the mod u la ted chromi nancc signal be carried on separJtc wires.
The J 300 incl udes hardware support for the Joint
Photographic Experts Group ( J PE G ) compression/ decompression standard .' J PEG is based on the discrete cosine transform ( DC:T) compression method f()l· sti l l ti·amc color images.
OCT i s a widcll· accepted method for image compression because it provides an efticicnr mechanism to el iminate components of the image that arc not eJsily perceived by c:tsual inspection .
Design History and Motivation
D igital arrived at the J 300 adapter design after consid eri n g several d igital video plavback architectures. The
Jvideo advanced development project, the implemen tation of one of the altenLHives, was instrumental in achieving the design goals.
Architectural Alternatives and Objectives
I n ]Jnuary 1 99 1 , snnal Digital engineering organiza tions collaborated to dctine the architecture of a h<ln1 warc seed project that cou ld be used to explore <1 workstation 's capability to process video data. The par ticipants klt that the kcv technologies required to explore the goal ofinregrating computers Jnd broadcast video were available. These enabling technologies liTre
1 .
The TU RBOchanncl high -speed l/0 bus, which was a standard on D igital workstations
2 . The anticipa ted acceptance of the
] P E G compression/ decompression standard and single chip implemen tations that supported that standard
3 . The development of a rendering svstcm ( now cal led the AccuVidco svstc m ) that cou ld map YUV i nput 1·alues into an 8 - bit color index using <1111' n u m ber of available colors with very good resu lts
We eval uated the three a l ternative appro;Khes shown in F igure l tor moving compressed video data from svstcm memory, f(>r decompressing and render ing the data, and, finally, f(> r moving the data i mo the frame b u rler.
The chroma key approach , sho\\'n i n Figure l a , d i tkrs l i ttle from previous work d o n e at Digit<li and was the pri mary architecture used bv the i n dustrv.
Severa l variations of the exact i mplementation arc i n use, but, basical l y, the graphics device paints a dcsig· natcd color into sections of the ti·ame bufttr where the video data is to appear on the d isplay. A comparator located bct\\'een the graphics frame b u tkr and the d is play device looks at the serial stream of dat:l coming from the graphics ri·anK bufkr and, when the data matches the chroma key (stored in a register ) , inserts the video data. As shown in Figu re l a, this <lpproach
Vol.
7
:-\u 4
IY% 2 1
SYSTEM
1/0
BUS t
I
DECOMPRESS
� 1
I
�
-R E _
D
_ R
Key Approach
( a ) C :hronu
H
CHROMA KEY
Lj
MONITOR
�6�
Tu
�
M
H
RAPHics coNmOL:Lffi -
- - - - 1
I
DECOMPREss
��
RENDER
H-1�
�� �
-
�
_
:
_
_ _ _ _ _ _ _ _ _ _ _ _ j
( b ) C raphics Comroller Approac h
.....
�I
MONITOR r----------------·�
�
To_R
22
Digit.ll Tecbnic1l Journal
(c)
Graphics Con rrol lcr-i ndcpendcm Approach
Figure
1
Digital Video Pbvback t\rc hirecrurcs relics on a spe c i al connection between the video de c o mpre ss ion block and the output of the g r ap hi c s device. While this approach off- loads the system 1/0 b u s , it treats video dac1 d iffer e n tly ti·om other data t y pes to be d i s p lay c d
.
In particu lar, the X Wi ndow
S ys t e m g r a p h i c a l wi n dowin g environmellt ILlS no knowledge of th e actual contcnts ofrhe vid c o \\'indow a t any giv e n t i me .
The graph i c s comrollcr approach , shown in Figure
I b, i nt e g ra te s t h e de c o m p rc s si o n te c h nolob')
' with the g rap h ics accelerator. Although this app roa c h has the po t c n ti a l of i ncu r r in g the lowest ove r al l system cost, i t fails in two important aspects. First, it does not expos e thc windowin g system to the video data. Second , since th e g ra ph ics controller and video l og i c arc in t e g r a ted , the user must �1ccept thc level of gr aphi cs perrcm11ance provided . No g r a p h i c s up g r ad c p a th cxists, so upgr
< ld
ing would r e q u i r c anothcr pr od uct d c vclopm e nt c ycl e
. I n c l udin g t h e vidco l o g i c across t he rangc of g ra phic s devices is not desirable, because such a dcsi g n forccs higher p r i ces rc)r uscrs who are no t intcrcstcd in t h e m:�nipularion ofbro:�dcast vid eo .
The third approach , shown in Figure l c, is much more rad ical. It places th c responsi bility o f m m
· ing each neld of video datJ to and from the decomprcssion/ r e nde r i ng o p t ion sq uar ely on the system . The systcm
I/0 b us m ust absorb not only the traffic g en e r ar cd bv thc movement of thc c ompr e ssed video to thc decom p re ss i o n h ardware but also th e movement of the
Vol . 7 No. 4 I \IYS d e com p rcssed vide o i m age ri-om rhc a c cclcra t o r back to system mcmory and b a c k again ovcr the sa m e bus to thc g ra p h i c s option. us
A cc c p t i n g the third a lt c rn at i v c architccture allowed to mcet thc threc i mportant o b j ec t iv e s ror t h e
1 . proj e ct
:
Thc workstation should be ablc video dat:� thc same as a ny o the r d to treat
1 t
� a t y pe . di g i t i ze d
2 . Thc inclusion of video capabil irics i n a wor k s t at i on should hc c om p l et subsystem use d . e l y i nde p e n d c nr o f the g r ap hi c s
3 . Any hardw�11-c option should b e low cost.
Thc original desi g n goals incl udcd :�udio
I/0, even though thc pro c essi ng power and bandwidth n ee d e d tor audio were br below th o s c r equ i rcd for \'ideo.
S i n c e uscrs who want video capabil i ty usually req u i re audio c apa bi lit y as weJI , audio suppo r t was included so that uscrs wou ld havc to get both audio and vid eo . buy o n l y one option to
This dcsign red uced the n umber of bus slot s u s e d .
The Jvideo Advanced Development Project
]video was m e nt h a r d rhc w a r namc given e s e ed p r o je to c t
. the a 1.h
·: m c ed develo
Actual desig p n work sta r t e d i n Fcbruary 1 99 1 ; p owc r on occurred in
S e p t e mbc r 1 99 1 . ]video has since bc c o m c a w i d e ly used rcscarch tool .
Table 1
The N i n e Video Flow Paths
I nput
Analog
�om pressed
.!,!_ncompressed
.------------------------------ Output ------------------------------,
Analog rom pressed
.!,!_ncompressed Qithered
C -> A
U --• A
A -> ( u
-· c
A -> U c
-· u
A -• D
C -> D
U --> D
Jvideo was an important advanced development project tor several reasons. First, it was the vehicle used to verif)r the tirst two project objectives. Second , it was the tlrst complete hardware implementation of the rendering circuit, thus verif),ing the image qual ity that was available when displaying video with tewer than
256 colors. Finally, it was d uring tbe development of
]video thJt the DMA structure and interaction with the system wJs developed and verified .
BOO Features
This section descri bes the various video paths sup ported in the as an example of video data tlow. The AccuVideo tilter-and-scale and dithering system designs used in the
J300 arc presented in detai l .
Video Paths
]300 and presents videoconterencing
Table l summarizes the nine fundamental video paths that the
]300 system supports . The input to the
J300 can come ti·orn an extern:d analog source or ti·orn the system in compressed or u ncomprcssed t(xm . The outputs include analog video and several internal tormats, i . e . ,
J PEG compressed, uncompresscd, or d ithered. Dithering is a technique used to produce a visually pleasant i mage whi le using f:1r less intcxmation than was avaibble in the original tonnat.
A conceptual Aow diagram of the major compo nents of the
J300 video system is shown in Figure 2 .
Physical l v, the frame store and the blocks to its left make up the video board . All the other blocks except for J PEG compression/decompression Me part of the
]300 application-specitic integrated circuit
(ASIC) .
(The
]300
Hardware I mplementation section pro vides details on this
ASIC.)
Both t h e upscale prior to the analog o u t block a nd the downscak: after the analog i n block scale the image size independently in the horizontal and vertical direc tions with arbitrary real-value scale factors. The filter and-downscale ti.1 1JCtion is handled by the Philips chip set, as described in the
J300
H:mlware Implementation section. The upscale block is
J copy of the Brcscnh:�m sryle scale circuit used in the filter-and -scale block.
The B resenham-stylc scal e circuit is extremely simple and is described in "Brcsenham-stylc Scaling," along with an i nteresting closed-torm sol ution tor finding initi:�l parameters.' The fi l ter-and-sc:�le block is part of the
J 300 rendering system . The
J300 supports arbitrary scaling tor either enlargement or reduction in both dimensions. We carefu l l y selected a tew simple, three-element horizontal til tcrs to be used in combi nation with sc1l ing; the til tcrs were small enough to be included in the
]300 ASIC.
The
J300 supports three sharpening nl ters that arc based on a digit<ll Laplacian:''
Low sharpness
Medium sharpness
H igh sharpness
( - 1/2
( - 1
( - 2
2
3
5
- lj2)
- 1
)
- 2 )
The
]300 also su pports t\vo low-pass or smoothing fi lters:
Low smoothing
High smoothing
( l/4
( 1/2 l/2
()
1 /4 ) l/2)
Sharpening is perto rmcd bd(lre scaling tclr enlarge ment and after scaling tclr reduction. Smoothing is a lways pertcm11ed bctclre scaling (as a band limiter) t(x reduction and after scaling ( as an interpolator) tor enlargement.
I
ANALOG OUT
1-+-1
UPSCALE
I
I t
FRAME
STORE I
SYSTEM
1/0
BUS
Figure 2
J 300 Video Flow
Digital Tcch nicJI journal Vol .
7
No. 4 1 995
23
24
The second part of video rendering occurs in the d ither block. The AccuVideo Rend e r i ng section pro vi de s derails on this block.
The I/0 bypass skips over the video rendering blocks when undithered uncompresscd output is req uired .
VVhcn uncompresscd d igital video in used as input, the
I/0 bypass is also used. DMA B rhus passes d i t h e r ed or uncompressed output and uncomprcssed input.
Co m p re ssed i nput :md compressed ourpur arc p a s se d through DMA A . The J PFG com p r e s si o n/ d ecompression block handles :�II co m p r es s ion of our pur and decompression of i nput. T h e combination of t h e t\\'O D!v!A chan nels allows high data rates bccJusc both dl<lnnels '1rc often used i n para l l e l .
Videoconferencing Application
A good i l lustrJtion of the video data tlow in J 300 is
:t v idc o c o nfer en cing application . Figure 3 sh ow s the tlow of analog
( A ), c ompre ss e d
( C:), and dithered ( D ) video data t o and ti
· o m memorv in
J svstem on a ne t work.
The application software controls the tlow of data between memory and the d ispl:�y and network devices. The J300 h
, m.hv
a r e must pert(mn two fu nda mental operations:
I .
C:tptu re the loul 'malog signal, compress the chta,
:md send it to memory, :md in paLlllcl, d ither the data and send it to memory. The solid arrows in figure
3
denote the compress, send , and ' icw
pa ths.
2 . Rccci\'C a remote compressed ,-ideo stream ti·om memory, decompress and d ither the data, and send it back to memory. The dashed <ltTows in Figure
3 de n o t e the recci,·c, decompress, :tnd \'iew paths.
Figure
3 d e m o n s t ra t es the unique graphics con troller independence of the
J300
''rchi tecture, ''s shown in Figure 1 c . In assessing the aggregate video data rraftic, it is important to keep in mind that the c
• c t c
I
MEMORY t
D
I
J300
C
I t
•
D D D t
A
$
Notes: Dashed arrows represent the receive, decompress, and view paths
( C -> D). Solid arrows represent the compress, send, and view paths
(A -> C , A -> D). The symbols A, C. and D stand for analog, compressed. and dithered data.
Figure 3
Vidcocon fc.:renc i ng Applic1rion d ithered d ata is 8 bits per pixel, and the compressed data is '1pproximarcly l . S bits per pixe l . For example, consider a videocon terence with
1 1 participants, where each person's workstation screen d isplavs the i mages of the other 1 0 participants, cKh in a
320by-
240
p ix c l window and with a rdi-esh rare of
20 Hz.
T h e bus traHic req u i red t(x each window is twice the compressed image size p l u s twice the decompressed i mage size, i . e . , (2 X 320 X 240 X
1 .5 )
-'-
8 Lwtcs +
( 2 X 320 X 240) byr es = 1 82 .4 ki l o b yt es ( kB ) per window. The tOtal b:�ndwidth wou l d be 1 82 .4 kB X
1 1 windows X 20 H z = 40. 1 megabytes ( M B ) pn second , which is wel l within the achiC\·::tblc bandwidth of borh TUR.BOchanncl and PCI buses.
These two o per a t i o n s through t he
J
300 c o n ce p tual tlow diagr:tm of
Figure
2 are shown explicitly in
Figure
4 tor tbe capture, compress, and dither paths, and i n
Figure
5 t()r rhe decompress :tnd d i ther path .
I n F i g ur e
4,
,·ideo cht:t is captured through rhc atlJ!og in block and buffered in the ti·amc store block. The fi·amc srore then sends the data in parallel to the
J
PEG compression/decompression path, and to the filter, scale, and d i ther path , each of which sends the data to its own ded icated DMA port.
In Figure 5, com pressed datJ enters DMA A, is
J P EC..; d e c o m p re s s ed using the km1e store JS a buftcr,
:�nd is sent ro the h i rer, sc1le, :tnd d i ther path , w here it is output through DMA B .
Figures
4
and 5 i l l ustrate three
of r
h c n i ne
po
s
s
i ble
Yideo p:uhs shown
i n
T:tble
l . ft
is srraightfon, ard
ro sec
h
o
w the other six paths tlow
t
h
r
o
u
gh the block. diagram of Figure
2 .
AccuVideo Rendering
Digit,l l 's AccuVidco method of ,·ideo renderi ng is used in the
]300 :�nd in o t h e r p ro d u c t s.
'
J300 render i ng is represented in Figure 2 by the ti ltcr-and -sctlc block and by the d ither block. The t(JI Iowing fc:�ru rcs arc su pported:
• H i g hq u ali t y d i t h e r i ng
•
Selectable n u m ber of colors from
2 ro 2 5 6
•
•
Y UV - to- R G B conversion w i r h conrrollcd ottt-ofbou nds m a p pi n g
Brightness, contt·,lst, ;llld satucnion control
•
Colm or grav-sc:�le output
•
•
Two-dimensional ( 2 - D ) scaling ro ;my s i ze
S harpening and smoothing conrrol
The
;J ig; or i thm t(lr me:�n- prcscn·ing m u lrilcYcl d ithering is descri bed b y
Ulic1Jt1ey in "Video
Rendering."" Mean preserving den o t e s that the macroscopic average in the o u tput i m a g e is mai n rained across the entire range o f i n pu t
,.,, l u e s.
F i g ur e
6 depicts the version of t h e dithering a lgorithm used t(lr the single c o m po n en t } in t he
] 300 prototYpe,
}video.
Vol . 7 No. 4
1 '.195
�----1
FRAME
STORE r
- -
I
I
I
1
I
JPEG
COM PRESSION/
DECOMPRESSION
.-
- - - - - - - ,
J3oo
V I D EO
I
I
I
I
I
I
SYSTEM
I/O BUS
Figure
4
Capture, Compress, ;HJd Dither J>,uhs
JPEG
COMPRESSION/
DECOMPRESSION
FRAME
STORE r
- -
1
I
I ..-�----,
I
I
L..------J
,...- - - - - - - -
..,
J3oo
VIDEO
I
I
I
I
I
I
SYSTEM
1/0 BUS
Figure
5
Decom press and Dither Path
Y
--R-
Y ADJUST
LUT
256 BY 9 BITS y
3
3
8
Y DITHER
MATR I X
64
BY
8
BITS
�
Figure 6
Dither Componcms of rhc ) \·ideo Prototvpc
To quantize with a simple shift register and still main tain mean preservation, a p:�rticular gain that happens to have a v;l l ue between
1 and
2 m ust be imparted to the input.'' This gain is i nc l uded i n the adjust look-u p table
( LUT), thus adding a bit to the data width of the i nput value to the ditherer.
In the case of the
V
( l uminance) component, the efkct of brightness and contrast can be con tro l led by dynamically ch;mging and Jo:�ding the contents of this adjust LUT. Satu ration comrol is a contrast-like map ping controlled on the
U and V adjust L UTs.
The least signi ticant bits of the horizontal and verti cal address (x,y) of the pixel index the d ither matrix.
I n the Jvideo prototype, we used an 8 by 8 recursive tessellation :�rrav.' Because the size of the array was so snLlll, all the components in Figure 6 could be encapsu lated with a single l 6 K-bv-4-bit random access memory ( RA M ) . This implemen ration is not the least expensive, but it is the easiest to build and is q uite appropri<Jte tor a prototype .
Figure
7 illustrates the } video dither svstem. The number of dither levels and associated color adj ust ment are designed in software and lo<lded i nto each of the 16K-bv 4 bit
LUTs for r; U, and V Each compo nent outp
;
Its trom
2 to 15 dithered levels. The three
4-bit dithered va lues are used as J collective <lddress to a color convert LUT, which is a 4K- b,·-8-bit RAM .
Loaded into this
LUT is the conversion of each
YUV triplet to one of N RGB index values. The gener ation of rhis LUT i ncorporates the state of the displ ay server's color map at render ti me. Although this
Jpproach is much more efticiellt than a direct algebraic conversion known as dcmatrixi ng, an arbitLlril y com plex mapping of out-of range values o n take place because the ta ble is built off l i ne. Another paper in this issue of the
Journal. "Software-only Compression,
Rendering, and Playback of Digital Video," presents details on this Jpproach .'
Perhaps the central characteristic ofAccuVideo ren dering is the pl easing nJture of the dither patterns generated. VVe are able to obtain such patterns because we incorpor:�re dither m:�trices developed using the void-and-cluster method . '" These matrices Jrc 32 by
Vol 7 No. 4 l '.)95 25
y
X y
8
3
3 u
8
-
1-
v
8
-
�
Y DITHER AND
ADJUST LUT
1 6K BY 4 BITS
U DITHER AND
ADJUST LUT
1 6K BY 4 BITS
�
�
COLOR
CONVERT
LUT
P--
COLOR
INDEX
V DITHER AND
ADJUST LUT
1 6K BY 4 BITS
�
4,096 BY
8
BITS
Figure
7
/video Dirher Svstem
32 in e xte n t . Although surprisi ngly small for the complexity and seamlessness of the patterns produced, this size requires 1 0 bits of disp lay a ddr ess i n t
(
> rma t io n t o r i n d e x
W i ng . hi l e v e ry s i m p k to i mp lemen t , the si n gl e LUT approach used in the Jvideo system shown i n Figure 7 becomes u n a t tra c ti v e tor a matrix of this size because of the large memory requirement. Eight bits of i n p u t plus 10 bits of array add ress r e q u i res '
'
25 6 K-bit RAM for each color component; Jvideo's 8 by 8 dither matrix called for a more cost-effective 1 6K-bit RAM
.
The dither system de si gn used in the J 300 is shown in
F i g u re
8 .
The d e si gn is q uite simple, requiring only
RAM and th ree adders. We restricted the numbe r of
Uand V-d ithered le v el s ro alw ays be equa l . Such a res t ri c tio n allows the sha r i ng of a singl e dither matrix
RAM. The paper "Video Rendeting" provides details on the relationship berween the number of dithen.:d l e v e l s t(x each component, the number of bits shi fted, the nor mal ization of the dither matrix v al u es, the gain embed ded in the a d j u s t LUT, and the bit widths of the data paths.9 Note that the decision to usc RAM i n st ead of read -only m e ma m or y ( RO M ) f()f the adjust LUTs, dither tri ce s
, and color co n v e rt LUT permits c omp le t e tlcx ibility in selecting the number of dithered colors. a
When the video source is mono chrom e, or wh ene v e r monochrome d i sp l a y is d es i re d , a Mono Select mode al lows the Vch:111nel to be quamized t o up to 8 bits.
The a l go ri t h m used in the sofuva rc-only version of
AccuVideo exa ct ly parallels f i gu r e 8 -' "Integrating
Video Rendering into Graphics Accelerator Chips" de s cribe s variations of this :trchitecture t(x other pr o duc ts .' One d e s i g n always renders the same num ber of colors without a d ju s t m ent, in Ewor ot' very lm-\'
Y ADJUST
LU
T
256 BY 9 BITS
u ---!--
U ADJUST
LUT
256 BY 9 BITS
v ---!--
V ADJUST
LUT
256 BY 9 BITS
9
8 r
X � y �
Y DITHER
MATRIX
1 ,024 BY
8
BITS
9
9
8 r
X
�
5
UV DITHER
MATRIX y __,........ 1 ,024 BY
8
1
8
BITS
9 9
+
9
_I
Y SHIFT
I
8
V SH IFT
I
4
4
4 l
U S H I FT
4
�
MONO
SELECT
COLOR
CONVERT
LUT
--!-.-
COLOR
INDEX
4,096 BY
8
BITS
Figure
8
]300 Dither System
26
Digit<ll Tcchnic,ll journ�l Vol.
7 No. 4 1995
cost. Anorher pertcm11s Y UV-ro-RGB conversion first, to allow di thering to more than 256 colors. Note that with this design, for large numbers of outp u t colors, the memory requ ired t()r th e back-end color convert
LU'f design wou ld be prohi bitive.
BOO
Hardware Implementation
I mplementing the J 300 hardware design entailed making trade-ofts to keep down the costs.
This section presents the In<ljor trJde-ofts and then d iscusses the resu lting video and audio su bsystem designs, the built-in 1/0 test capabil ities, and the Veri log hardware description language design environment used .
Design Trade-offs
In August 1 99 1 , the Jvideo hardware design team presented to engineering management several cost red ucing design alternatives with the goal of wrning
Jvidco into a product. Alternatives ranged tl·om rct:tin ing rhe basic design (which wou ld require a short design time and would resu l t in the tastest time to market) to redesigning the board with minimal cost
JS the driving tactor ( which meant p u tting as m uch logic as possible into the ] 300
ASI C ) .
Management accepted the latter proposal , :llld design started in
]allll:l ry 1992.
The major design trade-ofts i nvolved in red ucing module cost centered around three portions of the design: the accelerator chip, the pixel representation, and the dither circuit. The design team evaluated d i f krenr
J
PEG hardware compression/ decompression accclerJtors in terms ofavaibbiliry, pertormancc, cost, and sched u l e risk. While various manufactu rers claimed to have cheaper parts avai lable within our design sched u l e constraints, the C:L550 c h ip from
C:-Cube Microsystems, the same chip used in the
J video syste m, had reasonable pedormancc and known idiosyncrasies. The designers decided to usc one CL5 5 0 chip instead oftwo, as was done in J video.
This meant that in videocon krencing applications, the chip wou ld have to be progr:1m nKd to compress the incom ing image and then reprogrammed to decom press the other i mages. The turnaround time of the programming req uired to implement the design change plus the compression time together accounted tc>r the pertonnancc penalty that the prod uct would p:1y tc>r including only one CL550.
To understand the impact on pedormance of using just one CL5 5 0 chip, consider that all 700 registers in the chip would have to be re loaded when changing the chip from compression to decompression and vice versa. Given a register write cycle of2 5 0 nanoseconds, the penalty is 1 7 5 microseconds. We esti mated the time to compress an image as the number of pixels in the uncompressed image ( the CL550 does occasionally stall during compression or decom pression, but we ignored this bet tc>r these calculations) times the period of the pixel rare . For an image size of 320 by
240 pixels and a pixel clock period of
66.67 nanosec onds, the time used t()r compression is 5 . 1 2 milli seconds. If the desired overa l l frame rate of all images on the screen is 20 Hz, then approximately 1 1 percent of the available time is given to compression ( ( 5 . 1 2 mill iseconds + 0.35 milliseconds) -:- 50 milliseconds ).
We judged this decrease in decompression pert(>r mance reasonable, since approx imately 30 percent of rhe early estimated cost of materials on t he J 300 was the CL5 50 and the associated circuits.
The second major area of savings came with the d ecision to use the 4 : 2 : 2 Y UV pixel representation in the ti·ame store, the CL550, and the input to the ren dering l ogic . This approach red uced the width of the fra me store and external data paths from 24 to 1 6 bits with no loss of f-idel ity in the i mage. The trade-off associated with this decision was that the design pre cluded the ability to directly capture video in 24 bit
RGB u nless the ASI C included a fu l l YUV-to - RGB conversion. The mJin thrust of the product was to accelerate i mage compression and decompression on what was assu med to be the largest market, i . e . ,
8 -bit gr<lphics s�'Stcms, by using the Accu Video rendering path . Since 24-bit RG B can be obtained from
4 : 2 : 2
Y U V pixel represen tation (which c a n b e captured directl y ) with no loss of image fidelity, we considered this hardware limitation to be m inor.
The third area of rrade-oft-s revolved around the implementation of the dither circuit and how much of that circuitry the AS I C : shou ld include. The rendering system on Jvidco was implemented entirely with
LUTs, J method that is inexpensive in terms of the random logic needed but expensive in terms of com ponent cost. Early on, the design team decided that including the 4K- by-8-bit color convert LUT inside the ASIC was not practica l . Placing the LUT outside the
ASIC requ i red using a minimal n u m ber of pins,
2 8 , and usi ng a readily available 8 K-by-8- bit static random-access memory ( S RAM ) allowed the u nused portion of the RAM to store the d ither matrix va lues.
Such a design reduced the amount of on-chip storage required tor dither matrix values to 32 by 8 bits.
The i mpact of req uiring dither matrix va lue ktches on a per- line basis added to the interl ine overhead
32 accesses tc)r the new dither matrix values or 1 6 pixel docks. The impact of the 16 added c locks on a line basis depends on the res u l tant displayed i mage size. lfrhe displayed im::tges are small, the impact is as much as 10 percent
( t c
> r
�l
1 60-by- 1 20-pixel image ) . It is uncom mon, however, tor someone to view video on a workstation at that resolution. At a more com mon d isplayed size of 640 by 480, the amount ofoverhe:Ki decreases to 3 percent.
Vol .
7
No. 4 lY95
27
Video Subsystem Design
The m a
j
o r elements of th e video subsystem d e s i gn are the ASI C, which is d esig n e d in the Verilog ha rd wa r e description language, the Philips digital video c h ip set, and the com pression/decompression circui try. This section di scuss e s the AS! C d e s i gn and some aspects of the vi d eo 1/0 circuit de s i gn
.
The
J300 ASIC o
The ]300 AS I C design included not n l y the video paths d iscussed earlier in the s e c ti o n
]300
F ea ture s but also :11! the control tor the v id e o
1/0 section of th e d e si gn
, a l l video ra ndom-Jccess memory ( V RAM ) control , the CL550 interface, access to t he d iag no sti c s
RO M , arbi tration with the audio c i r c u i t tor T U RBOchannel access, and the
TU R.BOchan nel int e rb ce .
F ig u re 9 shows a block diagram of the ]300 AS I C. Only the DMA section of the design is discussed further in this paper.
The DMA
interface built i n to the
AS I C is designed to facilitate the m o v e m e n t ofL1rge blocks of data to or t i · o m system memory with rn ini ma l interaction from the system . The chip supports two channels: the first is u se d tor
CL550 host port data ( co mp r e sse d video and register write data ) ; the second is used ft)r pixel datJ fl o w i n g to or fi·orn the r en de r in g circuit. Once s ta rt e d , each channel uses its map pointer register to access successive (address. length) pairs that describe rhc p h y sic al memory to be used in the operation . (The map po i n t e r r e g ist e r poi nts to t h e scatter/gather map in svs tc m memorv to be u se d
.
) The ASIC ti lls or emp ties the first buffer and then
Jutomatically fe t c h e s the next
(address. /enp,th) p a i r in the scatter/garber map and so on u ntil t h e o p era tio n is complete.
'vVhen a c om pressed i mage is transtl:rred into system memory, the exact length of the d a t a set is un known until the AS IC detects the end-of- image m arke r ti·om the CL5 5 0 . In this case, s ys t e m sofTware can rc:�d a le ngth r e g i st er to ti nd out exactlv how much d:�ta was rranstCrred.
There is no r e s tr i ct i o n on the n u m ber of
(address. leupJb) p a ir s included i n e:1ch sc:mc r/garher map.
Nc11· pairs can be assigned to each line of incoming video such that dei ntcrlacing even and odd video tlclds can be a c co m p l i s hed
JS the dat<l is mo v ed i n to system m emo ry.
S ince only the map poi nter re g i s te r needs to be updated between operations, system software can set up multiple b uffe r s, each with i ts associated scatter/ g ath e r map, ahead of time.
Video I n p ut and Output Logic circuit, s
The J 300 1·ideo [/0 h o w n in Figure 10, was designed using Ph i l i p s
Semiconductors' digit a l
1·ideo c hip set. ExpLlnarion of some a s p e c t s of the d e s ig n td lows.
The J300 uses the
Philips chip set to d igitize and decode input video.
The c h i p set consists o t
' the
TDA8708A and the TDA8709A,
:1s the analog-to d i g i t a l
(
A /
D
) converters, and the
Digital MultiStandard
D
e co d er
SAA7 1 9 1 , as the
( DJ\i!SD). This ch i p
TURBOCHANNEL
1/0 BUS
TURBOCHANNEL
INTERFACE A N D
DMA CONTROL
-
CL550 HOST
PORT DATA t
COMPRESSED
DATA FIFO
BU FFER
CL550
CONTROL t
J PEG
CONTROL
AUXILIAR Y
B U S INTERFACE t
AUXILIARY
I NT E R FACE
INTERNAL CONTROL BUS l
INTERRUPT
I
TIMER
CONTROL
I
CONTROL
REG ISTERS
......
PIXEL F I FO
BUFFER
11
DITH ERING
1/0 BYPASS f-
FORMAT
AND F I LTER
V I D EO
CONTROL t
VRAM
CONTROL
PIXEL t
CONTROL
I
VIDEO
TIMING AND
CONTROL
FRAME
BUFFER
CONTROL
PIXEL
TIMING AND
CONTROL
PIXEL BUS
COLOR CONVERT LUT
AUDIO TURBOCHANNEL
ARBITRATION
Figure 9
J 300
ASIC
Bl ock
Diagram
Vol . 7 No. 4 l995
1
CLOCK
GENER ATOR
SAA7 1 97 t
2
I..._.
VIDEO I N
CONNECTOR
I
I
� t t
2
�
3 1
NO
CONVERTER
TDA8709A
1
'
3
NO
CONVERTER
TDA8708A
I
!
I NTERNAL
LOOPBACK
DIG ITAL
MULTI STANDARD
DECOD ER
VIDEO
OVERLAY
DATA
+
• •
1 2 C SERIAL
+ CONTROL
LINE
SAA7 1 9 1 l
VIDE O
SCALER
SAA7 1 86
I
+
• •
1 2 C SERIAL
+ CONTROL
LINE
VI DEO BUS
!
FRAME STORE
E I G HT 256K-BY-4-BIT VRAMS
....
:
VRAM
CONTROL
VIDEO OUT
CONN E CTOR
I
I
ANALOG
MULTIPLEXER
Y
TDA8540 lc
+
COMPOSITE
• •
1 2 C SERIAL
+ CONTROL
L I N E
VIDEO
ENCODER
SAA7 1 99B l
VIDEO UPSCALE
LOGIC
CLOCK
-
G E N E RATOR
SAA7 1 97
• ·
1 2 C SER IAL
CONTROL
LINE +
f
I
I
+
1 2 C S E RIAL
. .
.
CONTROL
:· . �
LINE UART
PCD8584
AUXILIARY
BUS
JPEG BOARD CONNECTOR
KEY:
+---+
V I D EO BUS
+
• •
+ CONTROL L I N E
�·
• • •
·�
REGISTER READ/WRITE BUS
Figure
10
J 300
Video
1/0 set supports NTSC
( M ),
PAL ( B , G,
H,
D ) , and
SECAM ( B , G, H, D, K, K l ) tcm11<lts.2 I t also suppons sq uare pixels, where the sampling rate is changed to
1 2 .272727 megah ertz ( MHz) t(>r the NTSC f(mnat and to 1 4 . 7 5 M H z t(>r the PAL and SECAM formats.
In addition , rhc )300 uses the SAA7 1 86, a digital video scaler chip that can scale the input to an arbitrarv size and pert(mn horizontal and vertical filtering.
The A/D C011verrers d igitize the i ncoming video signal to 256 levels. A video signal is composed of neg:nive-going svnchronization pulses, a color burst
(to aid in decoding color inf(>rmation ), and positive going ,·ideo."
As an aid to ,·isualizing this, Figur e 1 1 illustrates a simplified version o f the d rawi ng presented in the Color Television Studio Picture Li ne Amplifier
Output Drawi ng. " The l evel bdc>re and after the synchronization pu lses is referred to as bbnk level. Bl<lck level may or may not be the same as blank, depending on the standard . Video signals arc l volt peak to p e ak.
The first stage included in the A/D converters is a three-to-om: analog multiplexer. We used this cir cuit to allow t\YO composite signals to be attached at the same ti me to su pport S Video while allowing the third input to be used as an i nternal loop- back connection.
The TDA8708A chip is used for compos ite video and t(>r the l u m inance portion of S-Video.
Th e TDA8709A chip is used only tor the chrominance portion of S-
Video.
The A/D converters contain an automatic gai n con trol ( AGC) circuit, which l i mits the A/D range. The bottom of the synchronization pulse is set at 0, :md blank level is set at 64 . Given these setti ngs, peak white
Digit<li Tcchnic1\ Journal
Vol . 7 No. 4 1 995
29
PEAK WHITE
- - - - - - - - -
- - - - - - - - - 1
VOLT
VIDEO
VIDEO
COLOR BU RST
BLANK LEVEL
- - - - BLACK LEVEL
- - - - - - - - - - - - - 286
M I LLIVOLTS
- - - - - - _ L__
Figure 1 1
Depicrion ofVideo
Signal
Term inology
_�_ - - - - - - - - - - - - - 0
VOLTS corresponds to a va lue of 224 . If the input video level tends to exceed 2 1 3, a peak white gain control loop is activated, which lowers the internal gain of the video. The SAA7 1 9 l processes the luminance, :111d the resulti n g range i n the
Y value is 1 6 �(x black and approxi mately 220 for white. As recom mended by
CCIR
Report 601 -2 , there is room built into the rwo A/D converters and the DMSD to allow tor addi tive noise that might be present in the distri bution of video signals 3
The J 300 video I/0 design incl udes a video scaler so that the incoming video can be scaled down and til tered prior to compression . There are rwo primary reasons �()r this scaling. First, scaling reduces the amount of data to be processed, which results in a smaller com pressed version of the image . Second , scali ng removes any high-ti-eq uency noise i n the image, which results in higher compression r:ltios.
Unfortunately, i f the user wishes to compress and also to view the incoming video stream, the video will more than likely be scal ed again in the rendering cir cuit in the ASIC.
The J300 output video encoding circuit uses Philips'
SAA71 99B chip as the encoder.
This component is fi>l lowed by a low-pass fi lter and an analog mu ltiplexer
(Philips' TDA8540 chip), which functions as a 4 by 4 analog cross-point switch. The SAA7 l 99R video encoder accepts digital data in a variety of �imnats, including 4:2:2 YlN. The SAA7199B processes the chrominance and luminance according to which stan dard is being encoded, either
NTSC or PA L (B,
G ) .
The input range o f the SAA7 1 99B is compl iant with
CCI R Report 601 -2 for YUV: }' varies fi·om 16 to 235;
U and
\1 vary from 1 6 to 240. The analog multiplexer allows either the composite or S-Video output of the
SAA7199 R to be connected to the output connector.
The switch also allows the video signals to be routed to the input circuit for an i nternal loop- back con nection .
The
J
300 video l/0 design initiaJJy i ncluded a ti·anJC store because the CL5 S O could not guar;lntce that compression of a field of video would be completed before the next field started. Even i f the } 300 sc:ded and �i ltcred the video data prior to compression , some tcm por�1ry storage was needed . We included eight
256K- by-4-bit \TRAMs in the design �or this storage.
In the mode where only the even field is being cap tu red (wh ich cou l d be parr of reducing the size of the final image ti·om 640 by 480 pixels to 320 by 240 pixels), the J300 docs not know when the system wi l l reguest the n e x t field of i ncoming video. VRAMs organized as 768 b y
682 lw 1 6 bits allow room to store rwo �ields of either NTSC: or PAL
,·ideo. The incoming video stream contin u a l ly alternates between these two bufkrs.
The system then has the option of requesti ng the tield that wi ll provide the minimum input latency or the IJst complete field stored.
Requesti ng the tleld with the m inimum input latency creates the possibilit\' that the com pression
:llJd ren dering opcr:ttions will stall wai ting for the finish o f the video field being processed
In another mode of operation, the memorv is configured as a 1 ,024-by- 5 1 2 - by- 1 6-bit b ufl·cr This con tigur;uion is used when compressin g or d ecom pressing still images up to 1 ,024 pixels wide. A nother usc of thc ti·anK store organized i n this wav is tix dein terlacing. In dcintcrlacc mode, the even and odd fields arc recom bined to torm one image. Dcinterlacing allows captu re of a full
NTSC
�i·ame, but of only
5 1 2 lines of a PA L or SECAM ti·amc. This restriction is due to the nature of the s h i ft registe r cycles i mplemented in the V R.At\lls. A side efkct of using this deintcrlace mode when com pressing the in pur is that the temporal effects of combining rhc two �lelds generate what the
CLS 5 0 considers to be a large amount of high-spatial ti·equency components in the image, thus resulting i n poorer compression .
30
Digital
Tn:hniol Journ�l
Vol . 7 No. 4 1 995
Audio Subsystem Design
The designers bel ieved that the J 300 design should include audio capabilities that complemented the video capabilities. Consequently, the design incor porates an analog codec (the CS42 1 5 tl·om Crystal
Sem iconductors) and a digital audio codec ( the
MC5640 l from Motorola Semiconductors ) . These two chips provide all the audio 1/0 specified in the design . They communicate to the rest of the system by means of a serial digital interrace.
To provide audio capabilities such as compression , decompression , and format conversion , the ] 300 incl udes a general-purpose DSP ( DSP56001
Motorola Sem iconductors) with from
8 K by 24 b its of external RAM . This DSP can commun icate to the audio codecs through an integrated port. It a lso han dles the real - time nature of that interface by using a portion of the RAM to bu ffer the digital audio data.
The ] 300 oflers the same type of
DMA support t<x audio data as tor video data. The audio interface con troller ASJ C, along with the DSP, provides support for four independent DMA streams. These streams corre spond to the fou r possible sources or sinks of audio data: analog audio in, analog audio out, digital audio in, and digital audio out. The left channel of the ana log audio connection can also be rou ted to the head phone/microphone jack. Figure 12 shows a block diagram of the audio portion of the
] 300.
Testability of
110 Sections
In the early stages of design, we were aware that built in test features were needed to facilitate de bugging and to reduce the amount of special audio- and video specific test equ ipment required in man ufacturing.
Consequently, one ] 300 design goal was to include internal and external loop-back capability on all major
1/0 circuits. This goal was achieved with the excep tion of the digital audio circuit.
The video encoder can be programmed in test mode to outpu t a flat field of red, green, or blue. This signal was used in internal and external loop- back.
A comparison of the values obtained against known good val ues gives some level of con fidence with regard to the video 1/0 stage . The designers accomplished external loop- back by using a standard S-Video cable.
The analog a ud io codec has internal loop-back capability, and a standard audio cable can be used tor external loop- back rests. External loop- back tests of the h eadp hone/microphone jack required a special adapter.
Even with this degree of internal and external loop back capabi lity, the goal >vas to be able to pertorm much more rigorous testing without the need of spe cial instrumentation . Tests were developed that used two ] 300 systems to ked each other data. One ] 300 system output video data in NTSC or PAL formats of different test patterns, and the other J 300 interpreted the signals. The designers used the same technique for both the digital and the analog audio codecs. This method provided a high degree of system coverage with no additional specialized test instruments.
Hardware Design Environment
The
ASIC was designed completely in a hardware descriptjon language cal led Verilog, usmg no schematic sheets. At first, we simulated pieces of the design, building simple Vcrilog models for all the devices in the ] 300.
We simulated complex chips such as the video scaler and the CL550 as data sources or sinks, readi ng data from or writing data to files in
1/0
-
LINE OUT
-
-
-
DIGITAL
SERIAL
1--
ANALOG
CODEC
1--'--
DMA ARBITRATION
ASIC
I
I
ADDRESS BUS
r$1
33-MHZ DSP56001
AUDIO
CONTROLLER
ARBITRATION
1/0
¢::::::=>
Figure
1 2
) 300
Audio Block
Diagram
Vol .
7
No. 4
1995 3 1
Digital Tcchnic�l journal
memorv. This approach limited the video data that cou ld be compressed or decompressed to samples where both ,·ersions alre:�dy existed.
1n all cases, the
1/0 ports on devices modeled included accurate timing information. Vcrilog includes the capability to incorporate user ddined routines written in t he
C programming language that can be compiled into a
Verilog executable. The
J 300
d e s ign team took advantage of this capability by writing an interrace th�H took
TU RBOchanncl accesses from a portion of shared memory :md used them to dri,·c the Verilog model of the TUR.BOchannel bus. In that wav, the designers could write rest routines in C, c o m p ile them, and run them agai n st the Veri log model of the
ASJ C and o f the rest o f the board design.
The Verilog mod e l proved to be useful i n develop ing manufacturing diagnostics and was used to some extent for d ri,·er a nd library cod e development. It was a \'cry eft-Cctivc tool t-or the hardware designers, bccause much of the test cod e written d uring the design p hase was used to bring up the hardware in the
Llb and later as e x amp l e code tiJr library dC\·el opment.
Usc of the Veri log model f(Jr software developmenr was nor as extensive as was hoped, however. The requirement to have a Vcri log license available e a c h rime a model was invoked limited the n u m ber of users.
There were enough l icenses t-or hardware develop ment, but tew were left t-(Jr software developmcnr.
Another reason the soHwarc development team did nor relv on using the Vcrilog model was that even though the model provided an accurate simul ation of the hardware, t h e mod e l ,,·as a lso
\'crv slow.
Concluding Remarks
With irs Sound
& Motion
J3 00, Fu llVideo Su preme
J PEG, and Full Video Supreme products, Digital hJs achieved irs goa l of dcsigning a hardware option rhar a l lows the integration or· vidco into any workstatio n .
T h e adapter pert-ormancc on difrercnr plattorms depends on many factors, chief among which arc the cft-icicncv of the bus design ( either
T U R B Oc hann c l m
PC:I), the amount of othrr traff-ic on the bus, and the design of the graphics dn·icc. As the per formance of systems, particul arly graphics devices, i ncreases, the bottleneck in the J 300 lksign becomes the pixel t-i · e quency through the J300
ASI C:.
For this reason, any future adapter designs should incorporate a higher pixel tl-equcncy.
The
J 300 f-a m ily of products was the first to ofter
Digital's proprietary AccuVideo rendering technol ogy, aff-ording a high-quality yer low-cost solu tion for low- bit-depth frame butkrs. Rendering ,·ideo to 8 bits per pixel in combination with a high-speed bus a l lowed an architecture tint is i ndependent of thc graphics su bsystem.
Acknowledgments
The hard\vJre engineering team i ncluded Rudv
Stal zer, Petar Antonios, Tim Hel l man, and Tom
Fitzpatrick. Victor Bah! wrote thc audio and video dri vers, and Davis Pan wrote rhc code used by the
DSI'.
Paul Gauthier contributed ro the test routines avail a b l e at power- up. Nagi Sivananjaiah wrote t h e diag nostics, and Lance Bcrc :\lld Jim Gem's made major architectural contributions.
References
I . P. B:�h l , "The ) 30 0 f
�1 m i
Ach p r l y of V i d e o and Au d i o ers
:
S o f t w a r e
A r c h i te c tu r e ,
"
D ig it a l Techni ca!Joumal, v ol. 7, no. 4 ( 1 99 5 , this issue): 34-5 1 .
2 .
Characteristics u/ Telet'isinu
Svstems,
CCI R Report
624-2
( Gene\'<1:
I n te rn a t iona l Radio Co nsu ltati ve
Com m i ttee [ CCI R ] , 1 99 0 ) .
3 .
J;'ucoding dios.
Pmwnete;:o; oj'nt:� ital Te/el'lsioufor
Stu
C C I R Re port 60 1 - 2
( Gene,·a:
!n ternatioml
1\.:ldio Consu ltati ,·e C omm i t tee l CCI R], l 990 ) .
4.
!
nf or
/
1
/ at io n
Tech nolog)'- Oigitu/ Compression and Cadi ng o/
Coni in uous-tune
51 ill lmal.;es:
Requirements and
Clu id e li n es
,
ISO/I EC
1 09 1 8-
1 : 1 994 ( E ) ( G enc,·a : I
C o m m i ss i on
,
1 994 ) . nt c rna r i o n
< l l
O rg a ni z
<J t i o n t-or
Stan dardization/! merna tiona! EkcrrotechnicJI
5 . R. U l ichnev, "Brescnham -q·v l c S c a l i n g
,"
Pmceed in,-.;s of the JSC- T A n n uo/ Con/er< 'IIU' ( Cambri dge,
Mass . , 1 99 3 ) 10 1 - 1 0 3 .
6.
R . Ulich ne\', Digitu/ Ho/jiun ing ( C a m b
Mass . :
T h e
MIT Press, 1 98 7 ) . ridg e,
7. P. B a h l ,
P.
Gauthier, and R. Uliclmcy, "Software onh-
C o mp ressio n , R e n d e r i n g
, and Playback of
Digita l V i d e o
,"
!)igito/ Tecl.m ical
J o u rnal . vol .
7, no.
4 ( 1 99 5 , this iss u e ) : 5 2 -7 5 .
8.
L. Se i le r and R. U l i c h n ey ,
" l megraring Video Re n d e ri n g
76- 8 8 . i n to
G rap h i cs
A c c e l c rJto r
C h i p s
, "
Digital
Technica!Joumal. ,
· o l . 7, no. 4 ( 1 99 5 , t h is issue):
9 . R . Ul i chn ev,
"Video
Rendering,"
D(!i ilal
T echniw l
}ou uwl , vol. 5, no.
2
( S pri ng
1 99 3 ) : 9-1 8 .
1 0 . R . Ulichn
Gen era t ey
, "The V oi dan d-
Cl ust e r
Method for ing Dither Arrays," IS<."'-'1/')PJF Sv111posit1111 un E!l:'ctronic lnw,�iii,C< Science a n d
TechnologJ', San
J ose, Ca lif., vo l .
1 9 1 3 ( Februan· 1 99 3 ) : 332-34 3 .
I I . Industrial p
Electmn ics 1('/l/olit 'e Sto ndord .Vo
/.
Culor Teh:cisirm Studio Picture Line A mplifier
Out u t J)rml'ing
(
A rl i n gto n,
V:� . : l n d us r r i < l l E l ectron ics
Assoc iation, No,·embcr 1 97 7 ) . This pu b l ication is i nt en d ed as a com p:mion d ocu ment ro E l e c t r ica l
Poform a nee
S ta udu rds-.
\ ' lril 10chrullle Telet ·isiolt
Studio htcililics. rrial El
EIA- 1 70
(
A r l i n gt o n, Vd. : l nci u s e c t r o n i cs
Associ:�tion,
N o v embe r
1 9 5 7 ) .
32 Dig:it<ll Technical journal
Vol.
7
No 4 1995
Biographies
Kenneth W. Correll
Ati:cr rece in 1 978, i v i n g � B.S. E.E. ti · o m the Uni,·crsity o fW:� s h i n gro
Ken Co rr ell 11 o r kc d ;tt Spcrrv !-'light S1·stems t(Jr n e i g h t \'Cars, d e s i g n i n g cockpit d i s p b
1
· ,,.,term. He j oin ed
Digi t ;tl in 1 986, focusing on the specitic�tion and d e sig n of:ld1·anu.:d de1·elopmcm projects conce rned ,, · gr;t i r h the i nrc t i o n ofli1·e
,·ideo
�nd c o m p u l n s1·stems; he rccci1-cd
;t p
:� tc n t f(ll' some o f rbis II'OI'k. In
1 990,
Ken rn o 1
· e d ro r\11:\ss:�ch uscrts a n d b e g a n work on the Jvidco :�nd }300 projects. S in c e rhen h e has been i nvolved in the d e s i g n of g ra p hi c s AS!Cs i n rhe l
;r a p h i c s ;tnd M u ltimedia Hardll':�rc
Engi neering Croup within rite W o r k s t a t i o ns Business
Scgmellt.
Robert
A.
Robert U l
Uiiehney ichn c
1
· received a
P h . D . from rhe Massachusetts
Institute of
Tec h n ol o g v in e l ec t ri c a l engi neeri ng and com pu r e r science and a
B.S. in p h y si c s and computer science ti·orn the Uni l' e r si t y o t" Dayro n , Ohio. He joined Dig i t
:� l in
1 978. He is c u r r enr l v
;t senior consulting e n g ine e r wi t h
D i er·; t git a l ' s C a m t h e Video and b r id ge
Resc:lrch
L:�borarorv, where he le:1ds
I mage Processing project.
He has ti kd SCI' l p:�tcms t cJ r con tri burions to
Digital pr o d u c t s in the
,m:as ot'hardll':lre :�nd soti:,,·:�n.:-o n l v morion 1·ideo, graphics conrrollcrs, f-lafjirminp, and sen-cs a s
J n
: k r c c f(,r· :1 number ofrecll llic;tl societies, i and lurd co111'. Bob is rhe :1urhor of
Dii..:ltof n
..: lu di n g
I EEE, ofwhi..:h he is a se n io r mem ber.
Digit:tl Technical jou mal
Vol . 7 No. 4 1 995
33
T he
J300
Family of Video and Audio Adapters:
Software Architecture
I
Pa.ramvi..r
Baht
The
BOO fam ily of video and audio products is a feature-rich set of multimed ia hardware adapters developed by D igital for its Alpha workstations. T his paper describes the desig n and implementation of the
BOO software archi tecture, focusing on the Sound
&
Motion
BOO product. The software approach taken was to consider the hardware as two separate devices: the
BOO audio subsystem and the
BOO video subsystem. Li braries corresponding to the two subsystems provide application programming interfaces that offer flexible control of the hardware while supporti ng a cl ient-server model for multimedia applications. T he design places special emphasis on performance by favoring an asynchronous
1/0 programming model implemented through an innovative use of queues. T he kernel-mode device driver is portable across devices because it requires minimal knowledge of the hardware. The over all design aims at easing application program ming while extracting real-time performance from a non-real-time operating system. The software arch itecture has been successfully implemented over multiple platforms, includ ing those based on the Open VMS, Microsoft
Windows NT, and Dig ital UNIX operating sys tems, and is the foundation on which software for Digital's current video captu re, compression, and rendering hardware adapters exists.
Background
In January
199 1 , an adva nced devel opment project called ]video was jointly initiated by engineeri ng and research organizations across Digita l . Prior to this endeavor, these organizations had proposed and carried out several disjoint research projects pertai ning to video compression and video renderi n g . The
International Organization for Standardization ( ISO)
Joint Photographic Experts Group ( JPEG) was approaching standardi zation of a continuous-tone, still -image compression method, and the ISO Motion
Picture Experts Group's M P EG- 1 eft(m was weJ I on irs \\'ay to ddining an i nternational sta ndard for v ideo compression . '·' ·' Sili con for pert( mning JPEG com pression and decompression at real-rime rates was just becoming avai lable. lr was a recognized and accepted fact rbat the union of audio, video, :md computer systems was inevitable.
The goal of the ]video project was to pool the vari ous resources within Digital to design and develop a hardware and software mu ltimedia adapter tor
Digital's worksr:�rions. ]video would al low researchers to study the impact of video on the desktop. Huge amounts of video data, even after being compressed , stress every u nderlying component including net works, storage, system hardware, system software, and application software. The intent was that hands-on experience with ]video, while providi ng va luable insight toward effective management of video on the desktop, would i n tluence and potemially imp rove the d esign of hardware and software for future com puter systems.
]video was a three-board, single-slot TU RBOchannel adapter capable of supporting
J
PEG compression and decompression, video scaling, video renderi ng, and audio compression and decompression-al l at real time rates. Two J PEG codec chips provided simultane ous compression and decompression of video streams.
A custom application-specific integrated circuit
(AS I C ) incorporated the bus interrace with a direct memory access ( DMA) controller, tilrering, scaling, and Digi tal's proprietary video rendering logic.
]video's software consisted of a device d river, an aud io/video li brary, and appl ications. The underlying
34
Digital Technical jound
Vol .
7 No. 4 1 99 5
ULTRIX operating system ( Digital's native implemen tation of the UNIX operating system ) ran on work stations built around i'vi ! PS
R3000 and R4000 processors. Application How control was synchronous.
The librJry mai ntained mini mal state information, and only one process could access the device at anv one time. Hardware operations were programmed directl y ti·om user space.
The }video project succeeded i n its objectives.
Research institutes both i nternal and external to
Digital em braced }video tor studying compressed video as "just another data type ." Wl1 i le some research institutes used Jvideo t(>r designing network. protocols to allow the establish ment of rea l-time channels over local area networks ( LANs) and \\'ide area networks
(WANs ), others used it to study video as a mechanism to increase user p rod uctivi ty!
· ' J video validated the various design decisions that were d i tkrent from the trend in industry." It proved that d igital video could be successfully managed in a distributed environment.
The success of ;,·ideo, the demand for video on the desktop, and the nonavailability of silicon fc>r MPEG compression and decompression inHuenced Digital's decision to build and market a low-cost multimedia adapter similar in fu nctionality to Jvideo. The Sound
&
Motion J 300 product, rderred to in this paper as simply the } 300, is a direct descendent of the ]video advanced development project. The J300 is a two-board , single slot T RBOchannel option that supports all the rea tures provided by }video and more . Figure l presents the }300 hardware fi.mctional diagram, and Ta ble l contains a list of the features ofkred by the J 300 product. Details and anaJysis of tile J300 hardware can be tc)und in "The ]300 Family of Video and
Audio Adapters: Arch itecture and H ardware Design," a companion paper in this issue ofthe.founwf."
The latest in this series of video/audio ad apters are the single- board, single-slot peripheral component interconnect ( P C J )-based F u i i Video Supreme and
FuliVidco Su preme J PEG prod ucts. These pro d ucts are direct descendants of the }300 and are supported under the Digital U N I X , Microsott \iVindows NT, and
OpenVMS operating systems. FuiiVideo Supreme is a video-capture, video- render, and video-out-only option; whereas, FuiiVideo Supreme ) P EG also incl udes video compression and decom pression . l n keeping with the trend in ind ustry and to make the price attractive, Digital lett out :1lldio support when designing these two adapters.
All the adapters d iscussed <�re col lecti\'ely called the
J 300 t:11nily of video and audio adapters. The sofuvare architecture tor these options has evolved over years from being symmetric in } video to having completely asym metric flow control in the 13 00 and FuiiVideo
Supreme adapters . This paper descri bes the design and implementation of the software arch itecture t()r the
J300 family of mu ltimedia devices.
Software Architecture: Goals and Design
The sofuvare design ream had rwo primary objectives.
The first and most immediate objective was to write software suitable for contro l l i ng the )300 hardwart: .
This software had to provide applic:1tions with an
�lpplication programming imerface ( A P I ) that wou ld hide device-speci fic programming while exposing all lurdware capabilities in Jn intuitive manner. The soft ware had to be robust and bst with minimal overhead .
A second , longer-term objective \\'aS to design a sott ware <lrchitectu re that could be used for successors to the }300. The goal was to define ge neric abstractions that would apply to futu re , similar mu ltimedia devices.
Furthermore, the implementation had to allow porting to other devices with relative ly mini m<ll eftort.
When the project began, no mainstream multi media devices were available on the market, and expe rience with video on the desktop was limited.
Specitlca l l y, rhc l eading mu ltimedia AP!s were sti l l in their intancy, focusing attention on control of video devices l i ke videocassette recorders ( VCRs), laser d isc players, and cameras. Control of com pressed digital video on a workstation had not been considered in anv serious manner.
The core members of the }300 design team had worked on the )video project. Ex periences gained ti·om that project helped in designing an A P I with the t(>llowing attributes :
•
•
•
•
•
•
Separate li braries for the video and audio su bsystems
Fu nctional-level as opposed to component - l evel control ofthe device
Flexibil ity in algorithmic and hardware tuning
Provision for both synchronous and asynchronous tlow control
Su pport t(x a cliem-server model of multimedia comp uting
Support f(>r doing aud io-video synchronization at higher lavers
In addition , the architecture was designed to he independent of the u nderlying operating system and hardware platti.m11 . It incl uded a clean separation between device-independent and device-dependent portions, and, most i mportant, it lett device p rogram ming i n the user space. This last tCaru re made the debugging process tractable and was the key reason behind the design of a generic, portable kernel -mode mul timedia de,·ice driver.
As shown in the sections that f()l l ow, the soti:ware design decisions were intluenced grcnly by the desire to obtain good performance. The goal of extracting re1l time pert(>rmance tl·om a non- real-rime operating system was chal lenging. Toward this end, designers placed special emphasis on providing an asynchronous model for sofuvare flow control, on designing a fast
Vol. 7
No. 4 l 995
35
COMPRESSION PORT
N T SC
PAL
S-viDEO
NTSC
���
AM
S-VIDEO
�
I
I
VIDEO OUT
�
.
UPSCALE
.
I
I l
_ _ _ _ _ _ _ _ _ _ _ _
�
P.2:_U�_E
I
�
L
�
J
I
I
I
I
I
I
I
�r-------���------� 1
I
_ -
- - -
-
-
- - -
R
��
R
_
I
P I X E L
PORT
SYSTEM
I/O BUS
DIGITAL AUDIO
(AES/EBU)
--r-
.___
I
_.
Figure 1
The
Sound
&
Motion
} 3 00
Hacdwacc �liiKti on�ll Di�1gr�1m
Table
1
J300 H a rdware Featu res
Video Subsystem
Audio Su bsystem
Video in (NTSC, PAL, or
SECAM form ats) *
Video out (NTSC or PAL formats)
Com posite or S-Video 1/0
Sti l l -ima ge capture and display
J P E G co mp ress ion and decom pression
Image d ithering
Compact d isc (CD)-q u a l ity ana log 1/0
D i g ital I/O (AES/E B U format su p port) * *
Headphone and m icrophone 1/0
M u ltiple s a m p l i n g rates
(5 to 48 k i l ohertz [kHz))
Motorol a 's DSP5600 1 for audio processing
Progra m m a b l e gain and atte nuation
DMA i nto a nd out of system memory
Sample cou nter
Sca l i n g a n d fi ltering before com press ion
Sca l i n g and filtering befo re dithering
24-bit red, green, and blue (RGB) video out
Two DMA channels s i m u ltaneously opera ble
Video genlocking
G ra p h i cs overlay
1 50-kHz, 1 8-bit cou nter
(t ime-sta m p i ng)
*
National (U.S.) Television System Comm ittee, Phase Alternate Li ne, and Sequentiel Co uleur avec Memo ire
* *
Audio Engine ering Soci ety/Eu ropean Broadcasting Union
36
Digital TedH1iul
)ourml
Vol.
7
1\:o. 4 I \195 ke mcl-tnode d e1
· ic e dri1u, �1111.1 011 pr01·iding an archi rccrurc rhat wou ld req uire rhe J c:tst n u m be r of system calls :.111d minim:tl data co p y i ng .
The kernel-mode de1
· icc d ri1-cr is the [o,,·cst- b·cl soft ,,
·
:�r c mod ule in rh e
J300 software hicr�1rch1·. The d ri1·cr 1·icws the f300 ha r d ll
'
:t r c as t\1'0 d isrincr d cl
' iccs:
J 3 00 audio <1nd the ]300 video . p c nd m g on the the requested service, rhc
D e
]300 kemcl d r i
1
· c r ti lrers comp prop r i atc su hs1·srem dri1·er. This ke1· 111<1 lKls to the :t decision to separ<Jrc the iry i n f l w u ence d and the ove ral l d eve l o
J300 h
: m i ll·are h1· fu nction�ll
rhe Lksign of the u pp er layers of the ll'�lre . It a l l owed d esign e rs to d i1·idc the usk m�m:t geable components, borh in t er ms of engin e er
ing dh>rt and t(>r p r oje c t man:tgement. Separate te:-�ms or k e d on rhe rwo su bsystems t<>r extended p e sofr imo ri ods
, p men t time was r ed uced
.
E�1ch subsvsrem lud irs m1·n kernel d ri,·er, user dri1·er, sottll'�lre l i br
:-� n·
, test applicnions, and di�1gnostics sottw�1rc. The decision to sepat·atc the
�n t d i o and the video software proved to be a good one . Digir a l
' s Jar est m u l rimedi:-� ofk
S u p r e r i ng incl udes PCI-based FullVidco m e adapters that b u i l d on the 1
· id eo su bSI'Stctn sottll'�ll"C of rhc J 300.
U nl i kc rhc J 300, rhc ne11cr
� 1dapters do nor inc l ude :-�n audio subsystem and rhus do not use the � w d f o l l o ll
' i n g the philosoph1
· behind the actu a l des i gn, the ensuing d is c us i o librarv and d river. s ion of rhc I 300 sot i w:tre is orga n ized into two major sections. The tirst describes the sottw:-�re tor the vi d e o subsystem, inc l u di ng rhe design
and implemen tation of the video software li brarv and the kernel-mode video su hsvstem dri\Tr. Pert(xmance data is presented at the end of this section . The second major section describes the softw�1rc written t(Jr the audio subwste m . The p<lper then presents the methodology behind the development and testing proced u res f(x the \·arious software components and some i mpr<l\·cments tiLH arc C U ITCntlv being ill\·esti gated . A section on re lated publ ished \\'Ork concludes the paper.
Video Subsystem
The top of the software hierarchy t(Jr the \'ideo sub svstem is the appl ication layer, and the bottom is the kernel- mode dC\ icc dri\n. The t(JIIowing simplified example i l l ustrates the fu nctions of the various mod u les that compose this hierarchy.
Consider <1 video application th�lt is l i n ked to a mu lti media client library. During the course of execution, the appl ication asks t(Jr a video oper :t tion through a cal l to a client l ibrarv fu nction . The client l ibrary packages the request and passes it though a soc ket to a multimedia server. The server, which is running in the background, picks u p the req uest, determines the sub system t(Jr which it is i nrendcd , and ill\·okes the user mode d river f(x tlut su bS)1Stem . The user- mode driver translates the sen·er's request to an appropri:nc ( non blocking) \'ideo libr�1ry cal l . Based on the operation requested, the video li brarv builds scripts of ILlrdw�liT specific commands and i n t(mns the kernel- mode dC\·ice driver that new comnLmds are J\'<li labk t(>r exe cution on the hardware . At the next possible opportu nity, the kernel d river responds by dmm loading these commands to the u nderlying ktrd warc, which then performs the desired operation . Once the operation is co m pl e t e , results are returned to the <1pplication.
Figure 2 shows a graphical reprcsellt:ttion of the software h i e rarc h y. The modu les abm·c the kcmcl mode device d river, e xcl uding the operating S\'Stcm, are i n user space. The rem<lining modu les :tre in kernel space. The \'ideo l i brary is mod u larized i n to device independent and device-dependen t parts. Most of rhe
J
300-specific code resides in the device-dependent portion of the l ibrary, a n d verv l ittle is i n the kernel mode driver. The fol lo\\'ing sections descri be the vari ous componellts of the video sofuvarc hierarchy, begin ning with the device -independent part of the video l ibrary. T he description of the m ultimedia client l i brary and the m ul timedia server is beyond the scope of this p ap e r.
Video Library Overview
The c o n ce ptua l model adopted t(Jr the software con sists of th ree ded ie:tted functional u nits :
( l ) c1pture or play, ( 2 ) compress or decompress, and ( 3 ) render or bvpass. Figure 3 i l l ustrates this mod e l ; Figure I s hO\\ S th e hard\\·arc components \\'ithin each of the three
APPLICATION
M U L TIMED lA C L I E N T L I B RARY
M U LT I M E D I A SERV E R
OPERATING
SYSTEM
SERVICES
J300/FULLV I D EO S U P R E M E
USE R-MODE V I DEO D R I V E R
I
I
I
I
J300 USE R-MODE A U D I O D R I V E R
I
I
I
VIDEO LIBRARY AUDIO LIBRARY
I
I
DEVICE I N DEPEN DENT
I
FULL V I D EO
J300
S U P R E M E
S P E C I F I C
I
I
II
SPEC I F I C
I
I
I
I
I
D E V I C E I N DEPEN DENT
I
J 3 0 0 S P E C I F I C
I r - - - - - - - - - - - - - - - - - - �
�
- - - - - - - - - - - - - - - - - - - - - ,
I
· -
. . .
J300 K E R N E L-MODE DEVICE D R I V E R
I
V I D EO SUBSYSTEM
I I
AUDIO SUBSYSTEM
I
F U L L V I D EO S U P R E M E
K E R N EL-MODE DEVICE D R I V E R
'
' '
. . .
U S E R SPACE
K E R N E L SPACE
SOUND
&
MOTION J300 HARDWARE
I I
FULL V I D E O S U P R E M E HARDWARE
Figure 2
The ) 300 Video c1 11d ,-\udio Librc1 r\' c1s Componc nrs of DigirJ I 's M u l rimcdic1 Scr\'cr
Digit,li
Tcchnic1l )ourn;1l Vol .
7
�o. 4
1 995
3 7
COM PRESS/
COMPRESSION
DECOM PRESS -- PORT
VIDEO --
PORT
CAPTURE/PLAY
Figure 3
Conceptu<ll Model for the ]300 Video Su bsystem SofTware
L--
RENDER
..J
I
PIXEL
-- PORT units. The units may be combined in various configu rations to perform different logical operations. For example, capture may be combined with compression, or decompression may be combined with render.
Figure
4 shows how these functional units can be combined to form nine difkrent video tlow paths sup ported by the software . Access to the units is through dedicated d igital and analog ports. output colors and the sharpness tor the render unit 1·9
Examples of f/0 tu ning incl ude setting the region of interest tor the compression port and setting the input video format for the ana log port.
Thus, ports are configured to indicate the encod ing of the data, whereas u n its are configured to indicate parameters tor the video processing algorithms. Figure 5 shows the various tunable parameters tor the pons and units. by
All functional units and ports can be cont-lgured the video library through tunable parameters.
Algorithmic tu ning is possible by configuring the three units, and I/0 tu ning is possible by configuring the three ports. Examples of algorithmic tuning
Figure 6 shows valid picture encodi ng for the two
Digital IjO ports. Each functional unit operates i nde pendently on a picture. A picture is defined as a video ti·ame, a video field, or a sti l l image. Figure 7 ill ustrates the difkrcnce between a video frame and a video fiel d . i nclude setting the H u ffman tables or the quantization tables tor the compress unit and setting the number of -f
COMPRESS -
�
NNPAUL
�
G - CAPTURE
BYPASS
PROCESS
RENDER
L
T h e parity setting indicates whether t h e picture is an even field, an odd field, o r an interlaced frame.
CAPTURE AND R E N D E R
CAPTURE AND COMPRESS
CAPTURE, RENDER, AND COMPRESS
- PLAY
( :� ) Amlog
Inpu t .Mode
-E
ECOMPRESS --
��
�
�
RESSED
BYPASS
PROCESS
L
RENDER
DECOMPRESS AND PLAY
DECOMPRESS AND RENDER
DECOMPRESS, PLAY, AND RENDER
( b ) Com pressed I n put Mode
COMPRESS -
- PLAY
---+-i
RENDER
RENDER AND PLAY
RENDER AND COMPRESS
RENDER
(c) Pixel Input Mode
Note that a shaded area represents the render unit.
Figure 4
The Nine Different ]300 Video Flow Paths
38 Digit<11 Technical Joum<11
Vol .
7 No. 4 1 995
CONFIGURABLE
PARAMETERS
PORT
ANALOG
REGION OF
INTEREST
TV STANDARD
PARITY
GENLOCKING
MEDIUM
COMPRESSION
ENCODING
SKIP FACTOR
REGION OF
INTEREST
BUFFER
LOCATION
PARITY
PIXEL
BUFFER
LOCATION
ENCODING
M I R ROR
EFFECT
REG ION OF
INTEREST
UNIT
CAPTURE/
PLAY
I
(SAME AS
ANALOG PORT)
COMPRESS/ t
HUFFMAN
TABLES
OUANTIZA TION
TABLES
RENDER!
BYPASS
BRIGHTNESS
SATURATION
CONTRAST
SHARPNESS
NUMBER OF
OUTPUT COLORS
GAMMA
REVERSE VIDEO
Figure
5
T un�blc Paramcrers
Provided by rhe j 300
Video Libr:�ry
PICTURE ENCODING
PIXEL PORT
�
8-BIT PSEUDOCOLOR
8-BIT MONOCHROME
1 6-BIT
RAW (4:2:2)
24-BIT RGB PACKED
1 6-BIT OVERLAY
Figure 6
V�lid Picture Encoding tor rhe Two Digital 1/0
Pons
�
PROGRESSIVE JPEG. COLOR
INTERLACED JPEG. COLOR
PROGRESSIVE JPEG.
MONOCHROME
INTERLACED JPEG,
MONOCHROME
VIDEO FRAME
(33 MILLI SECONDS)
1 r--�--�--�--�-�--�--�--l-----�
2
ODD FIELD
(16
MILLISECONDS)
3 - - - - - - - - - - - - - - -
EVEN FI ELD
Figure
7
A
Picture,
Which May Be a fr�me Or a Field involve setting up the so f o is o in
Tht: software nonrecurring or the c o nt i g u r a t io n of the capture unit. R e c ur pe vo k e e it he r of recurring o b p r r o
:1 d ly cJassitles operations as eitht:r t: c u r rin g . Nonrecurring operations er a ti t w a r e for su bsequent picture p erati on s . An example of a nonrecurr i n g o pe r ons arc and
DecompressAndRender. a t i o n r i n g r a t i on s are pictu re o pe r a t i o ns that a ppl icati o ns p e rio di ca l l y or aperiodical ly. Examples
CaptureAndCompress,
RenderAndPiay,
All picture o p e r a t io n s
:-�rc p r o vi d e d i n two versions: blocking and non blocking . Blo c ki n g operations torcc the li b r ar y to behave svnchronously with the hard ware, w he r e as n o n b locki n g operations can b e used tc>r asynchronous progrJm tlow. ProgrJmming is simpler with b lo c ki n g operations b u t less e f fi c i e n t
, in terms of overall pertormancc, as compared to ope r a t io n s
.
Al l pictu re o p e nonblocking r a t i o ns rclv on combina tions of input and output buffers a v oid extra data c opie s to
, a pp li ca t ions are required re gi s t er these 1/0 buffers w i tl>J· picture data. To t h the library. The butlers arc locked down b y the librarv and are used tor subseq uent DMA transfers. Results ti·orn every picture
Digir,tl Technical jounLll Vol .
7 No. 4
1 995
39
operation come with a 90-kHz rime stamp, which can be used by applicuions for synchronization. (The
J 300's 1 50-kHz rimer is subsamplcd to 111<1tch the timer rrcqucncy specified i n the
ISO MPEG- 1
System
Specification . )
The video l i brary supports a client-server model of computing through the registration of parameters. I n this model , the video library is part ofrhe server process rhar controls the h:u·dwarc. Depending on i rs needs, each client application mav configure the hardware device d i frerently. To support m u ltiple d ienrs si m u l taneously, the server may have ro dliciently switch between the various hardwJ.rc conrigurarions. The server registers with the \·ideo l ibr;1.rv the r eb·anr set up parameters of the vJrious hmctional units and 1/0 ports r(>r each requested hardware c on figu r a tion .
A to ke n returned by the libr;�ry s er v es to idenril)' the registered parameter SetS r(x a l J
SU bscquent operations associated with the particular con tigurario n
. M u l tiple c l ients requesting the same h ardw:�rc configuration get the same toke n . VVhercvcr appropriate, deflll l t v:� l u cs t(>r p::1rameters not specified d u ring rcgismnion '1re used. Registrations arc classified as either heavvwcight, e .g., setting the nu mber of output colors t(x the render u nit, or l ightweight, e . g . , setting the quanriz:t tion r,1blcs t(n the compress u nit. A hc:wywcight registra tion often req uires rhc library to carry out complex ulcu larions to determine the appropri:tte val ues r(>r the hard\\'arc and consumes more rime than a l ightweight rcgistcnion, \\·hich ma\' be as simple as changing a
\'aluc in a register. Once set, ind i\·id ual parameters un be c!Lmged at a bter rime with edit routines provided by the library. After the c l ient has ti nished using the hardware, the server U llrcgisters the h,lrdware configu r�Hion . The \'ideo l ibrarv deletes all related i nternal stare i n rcmnation associ,ucd with th:tt con figuration only i f n o other client i s using the same configuration.
The l i brary provides routines r()r q uerying the con t-igucnions of t h e ports and u nits ''t a n y given ri me. b:tcnsi\T error checking and reporting arc built into the sofu\.JIT.
Video Library Operation
I n rcrnJI !v, the \'ideo l ibrary relics on q ueues t(>r supporting as\·nchro nous ( non blocki ng) flow control and f(n obt:� i n i ng good perf!.m11ancc. Three types of q ueues are defined within the l i br:try: (
1 ) command queue,
( 2 ) event ( or status) q ueue, and ( 3 ) request queue. The command and event queues arc allocucd bv the kernel-mode dri\·er fi·om the nonpagcd S\'Stern mcmorv pool at kcrnel-dri\·er load time. At device open rime, the two queues are m:1ppcd to the user vir tual memory add ress space and subsequently shared bv the video l i brary and the kernel- mode d river. The req uest queue, on the other hand, is allocated l ibrarv at dn·ice open rime and is P''rt lw the or' the user virtu a l memory space . Detailed descriptions of the three types of q ueues r<>llow. An example shows how the q ucues are used .
Command Queue The command queue, the hc:1rr of the libr,1ry, i s employed for one-w:ty communic:ttion ti-om the library to the kernel d river. Figure 8 shows the composition of the command queue. Essenri:tl lv, the command queue contains commands rhar set up, starr, and stop the hardware for pictu re opcr:ttions.
Pictu re operations correspond to video l i br:try ca l ls i nvoked by the user-mode driver. Even though the architecture docs not i mpose any restrictions, a picture operation usuallv consists of' two scripts: the first scri1)t sets up rhc operation , and rhe second script c l eans u p after t h e hard ware completes t h e operation. Scripts arc made up of packets. The header p:1ckct is c JI
!
cd
J script packer, and the remai ning packets are called command pac kers. The librarv builds packets ,llJd puts them i n ro the com mand q ueue. The kernel d ri\·er rerrie1·cs 'l!H.i i nterprets script packers and downloads the command packers to the hardware. Script packets provide the kernel driver with i n t(mllation about rhc type ofscript, the n u m ber of comm,md pxkets rh:tr constitute the script, :md the hardware i nterrupt to expect once
,,/ 1 comm,Hld packers have been down loaded . C:omm:1nd packers :1rc register l/0 operations. A command packet can contain the type of register access desired , the ker nel l'irtual address of the register, and the value ro usc if it is a write operation . The l i lxarv uses iden ti fiers associ ::ttcd with tlw command pac kers and the script packers to idenril)' the :tssociated operation . The com mand queue is managed :ts a ring bufkr. Two indexes called
PUT a nd
G ET d ictate where nC\1. p:tekcts gcr added �md from where old packets arc to be nrr:tctcd .
A first-i n , first-our ( fl
FO) ser\'icc policv is adhered ro .
The l i brary manages the
PUT index, ,md the kcmcl driver m:mages the
G ET i nd ex .
Event Queue The event queue, :t companion to the command queue, is a l so used t(>r onc-\\''1�' com m u n i cation b u t i n the reverse d i rection, i .e . , ti·om the kernel d ri ver to the library. Figu re 9 shows the composition of the cvcm queue. The kemel d river puts infonnarion i n to rhc q ue u e i n the t(mn ofevcnr P'lckers a hardware i nterrupt ( event) occu rs.
"
E1·enr
· hc n c
\· cr packers contain the type of hardware i nterrupt, the rime
:H which the i n terrupt occurred , an i n teger to identify the completed request, and , when appropriate, a l':t luc from a rcln·anr hard\1\lre register. The li brarv mon i tors the q ueue J n d exami nes t h e event packets to determine which requested pictmc opuation com plete d . As is the case with the command queue, the event queue is managed as a ring bu ffer with a
F I FO sen·icc policv. The l ibrarv manipu l:trcs the CET i ndex, and rhc kernel d river manipulates the PUT i ndex.
40
Digital
Tcchnico1l
)ounul
Vol . 7
No. 4 l ')')5
SCRIPT PACKET
TYPE
- -
CONTI N U E
- • IGNORE
END
REQUEST IDENTIFIER
NUMBER OF
COMMAND PACKETS
EXPECTED INTERRUPT
UNUSED
COMMAND PACKET
COMMAND
- -
DEVICE ADDRESS
Read
- • Write
ReadModifyWrite
SetAl arm
Flush
StopOperation
MASK
VALUE
TAG
END SCRIPT
/
/
/
'
'
/
GET
/
/
/
'
'
/
/
/
PUT
DEBUG
/
/
/
/
STOPPED,
R U N N I NG -+ -
, -�
I
I
STATE
OP.
60
OP.
61
\
'
'
'
'
'
'
/
/
/
/
/
/
/
/
I
OP.
62
1
OP.
64
1
/
/
I
I
I I
I I
I
I l l
. . .
I
I I
I I
I I
I I
OP.
58 l l
OP.
591
-
- �
I
I
COMMAND QUEUE
Figure 8
The CommJnd Queue
EVENT PACKET
TYPE
- -
COMP_DONE
- • REND_DONE
VSYNC
TIME STAMP
ALARM REQUEST
I D ENT I F I E R
RETURN VALUE
START SCRIPT
I
I r- -+-
1
I
I
EVENT QUEUE
GET
I
L - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - �
Figure 9
The Evenr Queue
Req uest Queue The li brary uses tl1e request queue to coordinate user-mode driver req uests with opera tions in the command queue and with completed events in the event q ueue. \Vhen a picture operation is requested, the library builds a request packet and places it in the request queue. The packet contains all information relevant to the operation , such as the location of the sou rce or destination buffer, its size, and scatter/gather maps tor DMA.
Subsequently, the library uses the request packet to program the command queue. Once the operation has completed, the associated request packet provides the i n formation that the li brary needs for returning the results to the user-mode driver. As with the other queues, the service pol icy is
FIFO,
and the queue is managed as a ring buffer.
Digital
Tcclmical
Journ,ll Vol .
7
No. 4 1 995
4 1
Capture and Render Example Figure
10 shows an application displaying live video o n a
UNIX work station that contains a
]300 adapter. The picture oper ation that makes this possible is the video li brary's
CaptureAndRender operation. A description of the asynchronous flow of control when the user- mode driver invokes a
CaptureAn d Render picture operation fc>llows. This example illustrates the typical interaction between the various sofuvare and hardware compo nents . The d iscussion places special emphasis on the usc ofthc queues previously described . l . The user-mode video driver invokes a nonblock ing
CaptureAnd Render picture operation with appropriate argu ments.
2. The library builds a req uest packet, assigns an identitier to it, and adds the packet to the request q ueue. Subsequently, it builds the script and com m:�nd packets needed for se ttin g up and terminat ing the operation and adds them to t h e command q ueue. It then invokes the kerne l d river's start
1/0 routine, to indicate that new hardware scripts have been added to the command queue.
3 . Start 1/0 q ueues u p the kernel routine (which downloads the command scripts to the hardware) in th<.: op<.:rating system's internal call-out q ueue as a deterred procedure call
( DPC) and returns control to the video library. 10
A l phaVCR
I
\I
'
1 J I d )I ll
•
-
-
-
-
--
-
--
--- -
I j''
1
�'
-·-� �
-�-
I I
I '
"-��
-
I .' I I
---
Figure 1 0
Live Video on a U N I X
Workstation Using the Capture and Render Path
J
4 .
The video l i brary r e tu r n s c o n t ro l to th<.: user mode driver, which continues from where it had left off� performing other tasks u ntil it invokes a blocki n g ( i .e., wait) routine. This gives th<.: l ibrar�' an opportunity to check the event q u<.:ue f(>r new events. If there are no n ew events to ser vice, the library asks the kernel driver to "put it to sleep" until a new event arrives.
5 . In the meantime, the
DPC that had previously been queued up starts to execute after being invoked by the operating system's sched uler. The job of the DPC is to read and interpret script pack ets and, based on the interpretation, to download the command packets that constitut<.: the script.
Only the first script that sets u p and starts the operation is downl oaded to the hardware.
6. A hardware i nterrupt signaling the completion of the operation occurs, and control is passed ro rhe kernel d river's hardware interrupt service routine
( ISR). The hardware
ISR. clears the i nterrupt l ine, logs the time, and queues up a software ISR in the system's call-out queue, pass in g it relevant int(>r mation such as the interrupt type and an associ ated time stamp.
7. The operating system's scheduler invokes rh<.: queued sofuvare ISR. The ISR then reads and interprets the current ( e n d ) script packet in the command queue, which provides th<.: type of i nterrupt to expect as a result of downloading the previous (start) script. The software I S R checks to see if the interrupt that was passed to i t is rhe same as one that was predicted by the (end ) script.
For example, a script that starts a render operation may expect to see a
REND_DONE C\'t:nt. When the actual event matches the predicted event, the command pacl,ets associated with the current
( en d ) script are downloaded to the hardw�1rc .
8 . A fter all command packets ti·om t h e ( e n d ) scri pt have been downloaded, the softwart: I S R logs the type of event, the associated time stamp, and an identi fier for the completed operation into the event queue. It then issues a wake - u p cJII ro
:�ny
"sleeping" or blocked operations that might hJve been waiting for hardware events.
9. The system wakes the sleeping l ibrary routine, which checks the event queue for new cvcnrs.
IL1
REND_DONE event is presem, the library uses the request identifier from the event packet to get the associated req uest packet from the req uest queue.
I t then places the results of the operation in the memory locations that are pointed to by add rcsst:s in the req uest packet and that belong to the user mode driver. (The buffer containing the rendered data is not copied because it already belongs to the user- mode driver. ) The library updates the
GET
42
DigirJI Technical )ourn<ll
Vol. 7 No. 4 1995
indexes of the event and n.:quest queues and returns control to the user-mode driver.
10.
The user- mock d river may then continue to q ueue up more operations.
Figure
1 1 shows a graphical representation of the capture and render example. If desired, multiple picture operations can be programmed through the library before
�1 single one is downloaded by the d river and executed by the hardware. Additionally, pertor mance is enhanced by improving the asynchronous tlow t h rough the use of mu ltiple bu ftl:rs f(>r the d i f tl:rent h.mctional units shown in Figure 3 .
Sometimes it i s necessary to bypass the queuing mechanism and program the hardw:�1-c directly. This is especially true �or hardware d iagnostics and operations such as hardware resetting, which require immediate action . In addition, tor slow operations, such as setting the analog port (video-in circuitry), programming the
!Lmiware in the kernel using queues is undesirable.
The kernel d river supports an immediate mode of operation that is accompl ished by mapping the hard ware to the library's memory space, d isabling the com mand queue, and allowing the library to program the hardware directlv.
The Kernel-mode Video Driver
To keep the complexity of the kernel-mode video driver manageable, we made J clear distinction between device programming and device register lo<lding. Device specific programming is done in user space by the video library; device register [/0
( without contextual under standing) is pert(>rmed by the kernel driver. Separating
USER SPACE
DRIVER LIBRARY
KERNEL SPACE
DRIVER!
OPERATING SYSTEM the tasks in this manner resulted in a kernel driver that incorporates little device-specific knowledge and thus is easily portable across mu ltiple devic e s.
The kernel d river al lows only one process to access the device at an�' particular time. (Support tor multiple process access is provided by the mu ltimedia server. )
Components of the video kernel- mode driver include
•
•
•
•
An Initialization Rou tine-The driver's i nitializ:� tion rou tine is e x ecute d by the operating system at d river load time. The primary fu nction of this rou tine is to reserve system resources such as nonpaged kernel memory t<.x the command queue, the event queue
,
:�nd the other internal data structures needed by the driver.
A
Set of Dispatch Routines-Disp:�tch routines constitute the main set of static fi.mctionaliry p r o vided by the d river. The driver provides d ispatch routines tor opening and closing the video su bsvs tem, for mapping and u n mapping hardware regis ters to the kernel and to user virrual m emory address spaces, t(>r locking and u n locking noncontiguous memory t()r scatter/gather
DMA, and tor mapping and unmJpping the various queues to the l ibra r y
.
An As ynchrono u s [/0 Romine-The video librarv invokes this routine to check t(>r pend ing events that have to be processed . If an unserviced event exists, the kernel driver i mmediately returns control to the library; if no event exists, the system puts the library process to sleep.
A S t a r t ljO Routine and a Stop 1/0 Routine
The driver uses the start ljO rou tine to initiate dat<l
HARDWARE
REQUEST
OPERATION ___.._.... BUILD AND QUEUE
PERFORM
OTHER
ACTIVITY
SCRIPT AND
COMMAND
PACKETS
----+-•
QUEUE UP DPC,
-+---'--------�--
RETURN CONTROL
�
INVOKE
BLOCKING
CALL ----+ IF NO EVENTS, GO
TO SLEEP
DPC RUNS, DOWNLOADS
START SCRIPT TO
HARDWARE -----------+
EXECUTE
+
HARDWARE ISR RUNS, -�- INTERRUPT
QUEUES UP SOFTWARE
ISR
:
SOFTWARE ISR RUNS,
DOWNLOADS END
J
SCRIPT, LOGS EVENTS,
TIME
!
RETURN RESULT
Figure 1 1
One Case ofSimpliticd Flow Control When Usi ng the Video Su bsystem
__,;
EXECUTE
Digiral Tcdmiol Journal
Vol .
7
No. 4 1 995
43
•
• transters to and ri·om the J 300 bv downloading reg ister 1/0 co m m a nd s from the command q ueue to the ]300. The stop 1/0 routine is used to terminate the downloJ.ding of future scripts. For pertormance reasons, scripts in rhe process of being downloaded can not be stoppe d .
A Hardware I nterrupt Service Routine-Since the h ardware
J S R runs at :.1 higher prio r i t y than both system and user space routines, it h:.1s purpose ly been kept small, per f ormin g only si mple tasks that J.re absolutely necessary J.nd time critica l .
Speci ricall v
, the hardware I S R records t h e i nterrupt and the time at which it occurred . It then clears the interrupt and queues up a soft\\'arc ISR.
A Software I merrupt Sen·ice Routine-The soti: ware ISR is the he<lrt of the ke rnel driver. It r u ns ar a lower i nterrupt request k vel ( I RQL) than the hardware JSR but has a h igher priority than user space routines. The softw;1rc ISR is invoked
J.S :1
D PC
e i the r by the hardware ISR or l)\' the library through a starr 1/0 req uest. I ts main function is ro process script packers and do\\'nl oad command packets programmed by the video librarv.
Debugging the Video Subsystem
Because of the real-time nature of operations, debug ging the softw;lre was a challenge . The size of the code, the compkx interaction between the ,·arious functional pieces, :1nd the asynchronous n<lturc of operations sug gested that, tc>r debugging pmposcs, it wou ld be help fu l it. h a rdware commands could be scru tinized just bdore the ti na! down l oading took place. Fortu nately, the video li brJry's extensive use of queues made i t pos si ble t(x us to desi g n a custo m tool with knowledge of the hardware and sofTware architectu res that \\'Ould allow us to examine the command scri pts.
In addition to presenting a de b u gg ing chal lenge, the real-time nature of operations limited the scope of
U lJ X tools like dbx, kd bx, and ctrace. Timing was important, and the debugger h<ld the tendency to slow down the overall p r o g ra m to the point where a previ ous rai l ure on a tree svstem would not occur with the d ebugge r cn:�bkd . To catch some of these elusi,·e bugs \\'hile p rese r v i n g the timing i n tegri ry of the oper ations, t h e scr.nch random-access memorv
( 1-v\!\t[ ) on the }300 audio su bsystem (sec
Figure 1 ) was used to store traces. A brief description of the two approaches tollows.
Queue Interpreter
The queue imcrpretcr was speciti cally de v elope d as <111 aid for debugging the \'ideo li brarv. As rhe name suggests, its prim<lry hmction \vas to in � e r p r c t the commands in the com ill�lnd q ueue and the C\'C n ts in the event queue. At random locations in the l i brary, a list of hardware com mands currenrlv in the com m;1nd queue cou l d be \'in, ed bd(m; the kernel d river down loaded them t()r execu tion . For each conHH;111d' the in rormation d isplayed included a seq uence number, the type o f op e r
:t tion, the
ASCI
I name of the register to be accessed , the reg ister's physicJ.I :�ddn:ss, the v�
1 lue to be
\\'ritten,
:�nd, when possible,
J bit-wise interpreution of the \'J. I u e .
This i n k)rmation w:�s used to check if t h e upper Lwcr software had progr:t nHned the device registers in the correct seq uence and with the proper \
' alu c s .
Another i m port;'ln t cap;lbility o f t h e que ue i nter preter was that it coul d step t h rou g h the comma nd packers and down l o
:� d c;1ch command scpar;Hc lv. On m;'lll\' occasions, this rL
I I K t io n h e lped loc a t e <lll<.i isolate the specif-ic register Kcess that was ca u sin g the h.:m.i
\\'arc to stall or to crash the system . B\' using the sequence n u m be r
, the o Ffending hardware command could be traced to the precise location in the library where it had been programmed .
I n addition, the queue i m erpretc r was able to search the command queue for any access to
;1 specific h:trch·are register, cou l d dispLl\' the contents of the e\·ent queue, <1 11d had
<l
"q uiet mode," in \\' hich the interpreter \\'Ould lo g the com m ands on a disk tc>r later analysis.
Audio RAM Pri nter
A l thoug h it was
::1 useful tool f(x de b ugg in g
, the q ueue i m c rpr c ter \\'as nor a good t·eal time tool because it slo\\'eci do,,·n the m
· er
; l l l program execu tion ;111d thus a free ted thc
<1Ctll<11 ri min g.
Si mi larly, kernel driver operations cou l d nor be traced u s ing the system 's printf( ) command hce<HISe it t o o afti.:cted the timing. Furthermore, bcc:tusc of t he ''syn chronous nature of printf( ) and the possibilin' of los ing it, printf( ) was i n e fk cti
,· e i n p in pointin g the precise command th:�t h:�d caused the S\'Stem to t
;
1 i l .
Thus, \\'e had to ri nd J l l J.lternate mech<1ll ism For d ebu gg in g failures rcL1tcd to timing.
The J300 audio su bsystem has an S K- lw-24 -bit
RAM that is never used rc>r :.1ny video ope r
<H i ons . This observation led to the i m p leme n tation oL1 prim h.lllC tion that wrote d i r c c r lv to the J 3 00's au d io
RAM.
This modi tied pri n t fu nction was inte r m i xe d in the s uspect code t!·agment in the kernel dri,·er to bci l i t;He n·ace
�liLllvsis. \Vhen
:1
S\'Stem r:1i lure occu rred or ;ltt er the
<lppl.ication hJ.d s t c
.
> pped , a companion "sn ifter" m u rine
\
\
'ou l d read and d u mp the c on tents ohhe RA M to the screen or to a ti le tc>r anal ysis. The modi tied pri nt fu nction was used primarilv tor debugging dvn:tmic oper<Hions such as the ones in the hard\\'arc ;md soft
,,
.
J r c i n terrupt lLmd lers . Mail \' bugs ,,·ere.: found and tlxcd using this tech niq ue . The one Gl\'CJ.t
,,·as th:H this technique \\'as usefu l onll' in CJses ,,· here the ' ideo su bsvstem was causing a svstem tailure indq)endcm oF the operation of the audio su bsystem .
44
Digiral
Tcchnic.li
)ourn;l l
\.'ol . 7 i'o. 4
1
YY:=i
Video Subsystem Performance
Mea s ur i n g the true perfo r ma nce of any software IS
�enerally a diHi c u l r ta s k . T he com p lex interaction lxtwecn d i t kn .
: n t mod u les and the n u m ber of vari ables that must be t ixc d makes the tas k ard uous. For
\'ideo, the pro b le m is aggra\·atcd lw the bet that th e sp ee d with which the u n de r lvi n g \
· ideo c o mpressio n al g or i thm works is no n line ar lv dependen t on tbe con tent o f the v i deo ti·amcs an d the desi r e d c o m pr essi o n ratio. A user wo r king with a c o mpresse d seq uence that contains ima�cs that :1rc smooth ( i .e., h:t\'C h igh spatial re d u nda nc y) will get a bstcr decompressi o n rare rlun a use r who has a seq uence that co n L1i n s i mag e s that h
; wc r e g ions of high ti·eq u enci e s ( i .e
.
, h:wc low s pat i al red undancy) .
A similar d is c r epan cy will exist when seq uences with d ifkrellt compress i o n ratios are used.
Since rherc JIT no standard video sequences av<lilablc, the :1nalvst has to m:1kc a best guess at cho o s i ng a set of rcprcscnt:ttin: seq u e n ces for exper i men ts . Beuuse the tin:�l r es u l ts
JIT dcpcndcllt on the input data, they arc in tluenced by this decision . Or her poss i ble reasons for t he variability of resu lts are the d i fkring lo a d s on the operat i ng systems, rhe d i fferellt c ontig u r at i o ns of the u n derlv i n g software, and the on: r hcad i mp o s e d bv the difkrent test appl
O u r m i cations . o t i
\
·
:u io n t(x c h ecki n g the perf-ormance of the
) 3 00 and F u i iVideo S u p r e m e
) P EG ad apters was to de t e r mine ll'hcthcr we ha d succeeded i n our goal of d eve l op i n g software that wou l d e xtr;Kt real time per f orm a nee while adding m i nimal overhead. The pl : nt< m11s we used in our cx per i mems were the
AlphaStation 600 5/266 an d the D EC 3000 Model
900. The AlphaStation 600 5/266 w as chosen because it is a I'C : I- base d system �nd could be used to test the Fui iVidco Su preme JP.EG a dap ter. The D EC
3000 Model 900 is a TURBOchanncl sy s t e m and could be used to test t he J 3 00 a d apter. Both systems arc bui lt a ro u nd t h e 64bit Al ph a 2 l 064A processm run n i ng at clock rates of
266 mega hertz ( M H z ) ::111d
275 MHz, r c spe cti \· cly. Each s y stem w as co n fi g u re d with 2 5 6 me ga bvtcs of physical memory, and each wJs r u nn i ng the Digital U N IX Version 3 . 2 opctati ng sys tem and DigiLll's M ult i me di: l
Serv ices Version 2 .0 t(
Jr Digiul U N I X s oftwa r e
. No computc-intensi\·e or
[/0 processes \\·e r e r u nn i n g i n the backgr o u n d , and , hence, the S\'Stems \\'ere l i gh t l y lo�1ded .
Our ex p eriments were designed ro rctlcct rea l ap p l i
eJtions, and speci�1l em phasis w as p lac e d on obtaining r e p ro duc i b l e pe rt(mn a n c c
Lbta. The aim was to understand how the pe r tormance of individual ses sioi1S was affccrcd as the IHi m bcr o h· i dc o sessions was i ncreased. VVc \\'rot c an a pp l icnio n that ca p tured , dithered ,
:111d displ ayed
:.1 l i ve video stream obtained tt·om d i a camera while sim u l t a neou sl y d ec o mp r e s si n g, t h er i ng, and di sp l ay in g mu l t iple video streams read ti·om a local d isk.
This is a c o mm o n function in te k c o nre r enc ing
:lpp l i ca t i o ns ll' hcre t h e multiple c o mpresse d video streams c o m e m · e r tbe network.
We m e asu red the d i splay rate t(Jr th e video s eq uen c e that was being c1pturcd d a nd d ithered and the a\-cragc i sp l ay r:ltc for sequ e n c e s that \\'Crc bei ng decom p re s sed and dithered . The comp ressed seq uences had an a\·eragc pc:1k signal-to-noise r a t i o ( PSNR) of
27
.R decibels
( d B ) and a n a\'CrJge compression ratio or· app r ox i m at el y 0.6 bits per p i xe l . The seq u enc e s h a d bee n c o mpressed and sto r ed on the l o c al d isk prior to t he exper i me n t . Image f r am e size \\'as source input format
(Sl r)
3 5 2 p i xe ls bv 240 lines. Figure 12 and
F i g u r e 13 illustrate t h e pe do rmancc data obt:�incd as a resu lt of the exper i me nts .
I n g e ne results. As r a l , \\'e were satisfied \\'ith rhc p e rform a nc e seen i n Figures 1 2 and 1 3, a total oHive ses sions can be accommod ated at 30 ti·ames per second with the ] 300 on a
DEC
3000 Mo d c l 9 0 0 system and three sessions at 30 f r a mes per second \\'ith the
SOUND
&
MOTION J300
6
� w
5 a: 4
�
3
2
0 5 10 1 5 20 25
D I SPLAY RATE (FRAMES/S ECOND)
KEY:
0
DECOMPRESS, DITHER. AND DISPLAY
•
CAPTURE, DITHER, AND DIS PLAY
Note: The number of sessions n is equal to one capture plus (n - 1 ) decompressions.
Fig u re
1 2
Puform a1K<.: D:H;l C.:nerated bl' a D EC 3000 Model 900
S\·srem \l'ith a Sound
&
Morion ) 3 00 Adapter
FULL VI DEO SUPREME JPEG i)5 6 i)
5
�
3
�
2
0 5
1 0
1 5 20
25
DI SPLAY RATE (FRAMES/SECOND)
KEY:
0
DECOMPRESS. DITHER. AND DISPLAY
•
CAPTURE. DITHER. AND D I SPLAY
Note: The number of sessions n is equal to one capture plus (n - 1 ) decompressions
30
30
Figure
1 3
Perform:1nce DatJ Ce nerared b\' an AlphaSrarion 600
5/266 \\ ith
:1 fui iVideo Su preme ) P EC Ad,lpter
Digit�! Tedmic:1l ) oun1:11 Vol . 7 No. 4
19%
45
FuiiVideo Supreme JPEG on an AlpbaStation 600
5/266 system. The d iscrepancy in performance ofrhe two systems may be attribu ted to the diflerences in
CPU, system bus, and maximum burst length . The
DEC 3000 Model 900 has a 3 2 - bit TURBOchanne! bus whose speed is 40 nanoseconds with a peak tr<lns fer rate of 1 0 0 megabytes per second, whereas the
AlphaStation 600 5/266 has a
PC! bus whose speed is 30 nanoseconds. The D M A controller on the ]300 adapter has a maximum burst length of 2 K pages, whereas the F u l lVideo Supreme J PEG adapter has a maximum burst length of 96 bytes. Since in our experiments data was dithered and sent over the bus
(at 83 Kbytes per frame) to the frame bu ffer, burst length becomes the dominant factor, and it is not unreasonable to expect the ]300 to perf(>rm better than the Fui iVideo Supreme JPEG .
The difference between capture and decompression rate (as shown in Figures 1 2 and 1 3 ) may be explained as tal lows: Decompression operations an; inter mixed between capture operations, which occur <lt a fi·eq uency of one every 33 milliseconds. Overall per formance improves when a larger n umber of decom pression operations are accommodated between successive capture operations. Since the amount of time the hardware takes to decompress a single fi·ame is u n known (the time depends on the picture con tent), the software is unable to determi ne the precise number of decompression operations that can be pro grammed . AJso, in the present architecture, since ::Ill operations have equal priority, i f a scheduled decom pression operation takes longer than expected , it is liable to not relinqu ish the hardware when a new ti·ame arrives, thus reducing the capture rate. \t\Then we ran the decompression, dither, and display operation only (with the capture operation turned off), the peak rate achieved by the Fui !Video Supreme ] PEG adapter was approxi mately 1 65 frames per second, and the rate for the Sound
& Motion
J 300 was about l l 8 fi·:.�mcs per second. Bus speed and hardware enhancemems i n t h e Fui iVideo Supreme
JPEG can be attributed to the difference in the two rates.
The next section describes the architecture tor the
]300 audio subsystem . Relative to the video su bsys tem, the audio software architecture is simpler and took less time to develop.
Audio Subsystem
The ]300 audio subsystem comp lements the
J 300 video su bsystem by providing a rich set of fu nctional routines by way of an audio li br<try. The software hier archy f(H· the audio subsystem is similar to the one for the video subsystem . Figure 2 shows the \'arious com ponems of this h ierarchv as implemented u nder the
Digital UNIX operating system. Briefly, an appl ication makes a req uest to a multimedia server for processing audio. The request is made through invocation of routines provided by a multimedia client library. The mu ltimedia server parses the req uest a nd dispatches the appropriate user- mode driver, which is built on top of the audio l i brary. Depend ing on the request, the audio li brary may perform the operation either on the native
CPU or alternatively on the J 300 digital signal processor ( D S P)
.
Completed results arc retu rned to the appl ication using the described path in the reverse d i rcction.
To provide a comprehensive list of audio processing routines, the software relics on both host-b�1sed and
J300- based processing. The workhorse of the J300 audio suosystem is the general-purpose Motorola
Semiconductor DSP5600 1 ( see Figure 14 ), which provides hardware control t(>r the various :llldio com ponents while performing complex signal processing tasks at real-time rates. Most notable, software running on the DSP initiates DMA to and ti·om system memory, controls digi tal (AES/EB U ) audio I/0 , manages ana log stereo and mono I/0, and supports multiple sam pling rates, including Telephony (8 kHz) and ti·actions of digital audio tape ( DAT) (48 kH z) and compact disc
( C D ) (44 . 1 kHz) rates. The single-i nstruction multi ply, add, and multiply-accumulate operations, the two data moves per instruction operations, and the low overhead k>r specialized data addressing make the DSP
MOTOROLA'S DSP56001 PROCESSOR
RECORD CHANNEL
....._.._
....._.._ PLAYBACK CHANNEL
Figure
1 4
Some Audio Fu nctions Supported b y !Yiotorola ' s DSP5600 l
Processor
46
Digital Technical journal Vol . 7 No. 4 1 995
especially suitable tor comput e intensive audio process ing tasks. Real-time fimctions such as adaptive differen tial pu lse code modulation (ADPCM ) encoding and decod ing, energy calculation, gain control tor analog to-digital ( A/ D ) and digital- to -a1ulog ( D/A) convert ers, and time-stamping are perf(mm:d by software running on the DSP. " Other tasks such as converting between difterent audio tormats ( w law, A-law, and lin ear), mixing and un mixing of multiple audio streams, and correlating the system time with the J300 90-kHz timer and with the sample counter are done on the native CPU bv the l ibrary sofuvare . "
Early in the project, we h a d t o decide whether o r not to expose t h e DSP to the client applications.
Exposing the DSP would have provided additional tlexibiliry tor application writers. Although this was an important reason, the opposing arguments, which were based on the negative consequences of exposing the raw hardware, were more compelling. System security and reliability would have been compromised; an incorrectly programmed DSP could cause the system to bil and could corrupt the kernel data struc tures. Additionally, maintaining, debugging, and sup porting the sofuvare would be difficult. To succeed, the prod uct had to be reliable. Therdore, we decided to retain control of the sofiware but to provide enough tlexibility to sa tis�' as many app lication writers as possible. As customer demand and ked back grew, more DSP programs would be added to the l ist of existing programs in a control led manner to ensure the integrity and robustness of the system .
The fol lowing su bsections describe the basic con cq>ts behind the device- independent portion of the audio li brary and provide an operational overview of the l ibrary internals.
Audio Library Overview
The audio l ibnlr}' defines a single audio sample as the fimdamcntal unit for audio processing. Depending on the type of encoding and whether it is mono or stereo, an audio sample may be any of the t()J iowi ng: a 4- bit
ADPC:M code word , a pair ofkfi/right 4-bit ADPCM code words, a 1 6-bit linear pu lse code modulation
( PC:M) audio level , a pair of lett/right 1 6-bit linear
PCM audio levels, an 8 - bit J..L
Iaw level, or an 8-bit
A- law lewl. The l i brary ddines continually flowing audio samples as an audio stream whose attributes can be set by applications. Attributes provide intormation on the sampling rate, the type of encoding, and how to imerpret each sample.
Audio streams tlow through d istinct directional vir tual channels. Specitically, an audio stream flows into the subsystem for processing through a record ( input) channel, and a processed stream tlows out of the subsystem through a playback (output) channel.
A configurable bypass mode i n which the channels are used t()r a direct path to the hardware 1/0 ports is also provided .
As is the case for audio streams, each chan nel has attributes such as a bufkr tor storing captured data, a buffer tor storing data to be played out, permis sions tor channel access, and a sample coumer. Sample cou nters arc used by the l ibrary to determine the last audio sample processed by the hardware. Channel per missions determine the actions al lowed on the chan nel. Possible actions i nclude read, write, mix, u nmix, and gain control or combinations of these actions.
The buffers associated with the 1/0 channels are ti.x queuing unserviced audio data and are called smoothing bufkrs. A smoothing bufter ensures a con tinuous flow of data by preventing samples from being lost due to the non-real-time sched uling by the under lying operating system . The li brary provides non blocking rou ti nes that can read , write, mix, and unmix audio samples contained i n the channel buffers. A slid ing access window determi nes which samples can be accessed within the buffer. The access window is char acterized in sample- time units, and its size is pro portional to that of the channel bufter that holds the audio data .
Li ke the video l ibrary, the audio library supports multiple device configurations through a set of regis tration routines. Clients may register channel and audio stream parameters with the library ( through the server) at set-up time. Once registered, the parameters can be changed only by u nregistering and then rereg istering. The library provides query routines that return status/progress information, including the samples processed, the times ( both system and J300 specific) at which they were processed, and the chan nel and stream configurations. Overall, the library supports tOur operational ( execution ) modes: tele con ferencing, compression, decompression, and rate conversion. Extensive error checking and reporting are incorporated into the software.
Audio Library Operation
The execution mode and the associated DSP program d ictate the operation of the audio library. Execution modes arc user selectable. All programs support mul tiple sampl ing rates,
1/0 gain control, and start and pause features, and provide location inbrmation for the sample being processed within the channel buffer.
Bu ffers associated with the record and playback chan nels are treated as ring buffers with a FIFO service pol icy. Management of data in the bu ffers is through integer indexes ( G ET and PUT) using an approach similar to the one adopted for the management of the command and event queues in the video su bsystem.
Specifically, the D MA controller moves the audio data from the DSP's external memory to the area in the
Digit1i Tcdmical journal
Vol . 7 No. 4
1995
47
48 channel buffer (host memorv) starting at the
PUT index. Audio dat:l in this same channel buffer is pulled by the host ( li brary) ti·om the location pointed to by the
GET index. Managers of the GET and
PUT indexes are reversed when
DMA is being perti>rmcd trom a channel butler to the DSP external memory. I n all cases, the
FifO service policy ensures that the audio data is processed in the sequence in which i t arri\·es.
The i nterna l operation of the audio l ibLm' is best explai ned with the help of a simple example that cap tures analog audio ti·om the J 300 line-in connector and plays out the data through the J 3 00's line-out con nector. This most basic 1/0 operation is incorpo rated in more elaborate audio processing programs.
The example fol l ows. l . The server opens the audio su bsvstem, a l locates memory tor the 1/0 buffers, and invokes a library routine to lock down the buftcrs. Two buftcrs J.re associated with the record and playback channels.
2. The library sets up the
DSP external memory tor communications between software running on the two processors, i.e., the C PU and the DSP. The set-up proced ure involves writing intc:>rmation at locations known and accessible to both processors.
The intonnation pertains to the phvsical addresses needed bv the
DMA scheduler portion of the
DSP program and t(>r storing progress information.
3. A kernel driver rou tine maps a section of system memory to user space. This shared memory is used tor commu nication between the driver and the library. The type of information passed back and tortb incl udes the samp le nu mber being processed, the associated time stamps, and the location of the
GET and
PUT ind exes within the l/0 bufkrs.
4. Other set-up tasks pertonned by the librarv incl ude choosing the 1/0 connectors, setti ng the gain for the 1/0 channels, and loading the appropriate
DSP program. A start rou tine enables the
DS P.
5 .
Once the DSP is en:� bled, all components in the audio hardware arc u nder its control . The grams the D!YlA
DSP pro controller to take sampled audio data trom tbe line-in connector and move it imo the record channel buffer. It t hen programs the same controller to grab data from the playback channel bu ffer and move it to the external memory fi·om where it is played out on the line-out connector.
6. The library monitors the indexes associated with the l/0 bufkrs to determine the progress, and, based on the index values, the application copies data trom the input chan nel to the output c h:m nel bu ffer. The access window ensures that t.bta copy i ng stays behind the
DSP, in the case of input, :�nd in fi·onr of the
DS P, in the case of output.
Support for M ultiple Adapters
The prinury reason f()r using multiple J 300 adapters is to overcome the inherent limitations of using a single
] 300 First, a single J 300 lim its the application to a single video port and a single :�udio input port.
Some applications process multiple \'ideo input streams simultaneouslv. For example, a television station recei\' ing mu ltiple video feeds may \\'allt to compress and store these t(>r later usage utilizing a single workstation.
Another cx;unple is the monito ring of m u l tiple video teeds ti·om strategically placed video cameras for the purpose of security. Since AlphaStation systems have the necessary horsepower to process sc\·eral streams sim ultancousl�o, supporting mu ltipk J300s on the same system is desirable.
Second , i f a single J300 is used, the video-in and video-out ports cannot be used simu lt:�ncouslv. This limitation exists because the two ports share a common fi·ame store, :JS shown in Figure l ,
:�nd programming the video-in and video-out chip sets is
J heavyweight operation . Multiple ] 300s can allcviJte this problem.
One example of an appl ication tllat req uires the simul taneous usc of the video-in and \'ideo-out ports is a teleconferencing appl ication in \l·hich the \'ideo-in circuitrv is used tor captu ring the camera output, and the \·ideo-out circuitry is used till· sending regu lar snapshots of the workstation screen to an m·erhead projection screen.
A second example is an appl ication that converts video streams ti·om one tormat to another ( e .g . , PAL, SECAM , NTSC) in real time.
As
<1 result of the l i m itations just cited , support for m u l tiple J3 00s on the same workstation was one of the project's design goals. In terms of coding, achie\· ing this goal req uired not rclving on global \'Miables and using indexed structures to maintain state infor mation . Also, because of the multithreaded nature of the server, care had to be taken to ensure that data and operation integrity was mai ntained .
For most Alpha systems, the overa ll performance remains good even with two J 300s on the same sys tem. For high-end svstems, up to three J300s may be used. The dominant limitation in the number ofJ300s that can be h:mdled Lw a s1·stem is the bus bJnd\\·idth .
As the nu mber of J3 00s in the S\'Stcm increases, the data traftic on the system bus increases proportionally.
Having described the software :� rch itectu re, we now shift our attention to the d evelopment environment, testing strategy, and diagnostics software .
Software Development Environment
During the carlv phases of the de\·c lopmcnt process, we depended almost exclusively on }video . Since the
] 300 is primarily a cost-red uced version of ] video, we were abk: to develop, test, and validate the design of
Vol .
7
No. 4
199S
the device-independent portion of the software and most of the kernel de\·ice driver well be fore the actu<ll
J 300 hardware arri\·ed . Our platform consisted of
:1
Jvideo attached to a DECstation workstation, which was based on a M I PS R3000 processor :-tnd was run ning the ULTRI X oper:-tting system. vVhen the new
AJpha workst:Jtions became avail able, we switched our development to these newer and faster m:�chines. We ported the 32- bi t software to Alpha's 64-oit architec tu re . Sections of the kernel device driver were rewrit ten, but the basic design remained intact. The overall porting effort took a little more than a month to com plete. At the end of that time, we had the software running on a Jvideo attached to an AJpha workstation, which was running the DEC OSF /1 operating svsrem
( now called the Digital UNIX operating system). We promptly corrected software timing bugs exposed :1s a result of using the rast AJpha- based workstations.
For the development of the device dependent por tion, we relied on hardware simu lation of the J300.
The difkn:nt components and circuits of the ]300 were modeled with Verilog behavioral constructs.
Accesses ro the TURBOchannd bus were simulated through interprocess communication calls ( I l'Cs) and shared memory (sec Figure 15 ). Because a 64-bit ver sion of Vnilog was un<W:lilable, simulations were run on a 1113chine based on the M I PS R3000 processor running the U LTJU X operating svstern . The process, though accurate, was generally slow.
Testing and Diagnostics
We wrote several applic:1tions to test the software arch itecture. The purpose of these :1pplicJtions was to test the software features in real-world situations and to demonstrate through working sample code how the libr�1ries could be used . Applications were classi fied as video only, :1udio only, and ones that contained both video �111d audio.
In addition, we \\TOte two tvpes of di:1gnostic soft ware to test the u nderlying h�1rdware compone nts:
( l ) read -only memorv ( ROM) based and ( 2 ) operating system based . RO M- based d iagnostics haYe the ad\·an tage that they can be executed ri·om the console level without first booting the s�·stem. The coverage pro vided is limited, however, bcc:�use of the complexity of rhe h:1rdware and the limited size of t he R.OM .
Operating system diagnostics rely on the kernel device driver and on some of the library software. This suite of tests provides comprebcnsi\-e coverage with veritica tions of all the ti.mctional blocks on the
T
300. For the new PCI- based FullVideo Supreme video adapters, only operating-svstem-bascd diagnostics exist.
Related Work
When the }video was concei\·ed in early 1 99 1 , little had been published on hardware :llld software sol u tions for putting video o n t h e desktop. This may have been partly due to the newness of the compression stand ards and to the difficulty in obtaining speci:1lized video compression silicon. S ince then, audio and video compression have become mainstream , and several com puter vendors now have products that :1dd multi media cap:1bi litv to the base workstations.
Lee and Sud harsanan describe a hardware and soft ware design t(>r a ] P
E
G microchan nel ad:1pter card built tor platforms based on I B !vl's PS/2 operati ng syste m . ' -' The adapter is con trol led bv an interrupt driven software running under DOS . In addition, the software is also responsi ble tor color-space conversion and algorithmic tuning of the ] PEG parameters. Audio support is not included in the h:1rdwarc. The paper presents details on how the software programs the var ious components of the board ( e . g., the CL550 chip from C-Cubc M icrosystems and the DMA logic) to achieve compression and decompression. Portabilitv of the sofiw:1re is compromised since the bu l k of the
DEVICE
SOFTWARE PROCESS
WRITE
READ
Figure 1 5
H � rd\\'�rc
S i m u l a t i o n En,·iron menr t(>r Sohw:trc
Dn·cl opmcnt
BOARD
J•----l
MODEL
SIGNALS
TURBOCHANNEL
SIMULATION PROCESS
Digi[�J Tcchnic;ll
]oumal
Vol . 7 No. 4 1 995 49
cod e, which resides inside the i nterrupt service rou tine, is written i n assembly language.
Bol iek and Al len describe the implementation of hardware that, in addition to providing baseline
J PE
G compression, uses a dynamic quanrization circuit to achieve fi xed -rate compression
. 14 The board is based on the RlOH JPEG chip set that includes separate chips tor performing the DCT, Huffman coding, and color-space conversion. The paper's main focus is on describing the Allen Parameterized (orthogonal)
Transform that approximates the D CT while red ucing the cost of the hardware. The paper contains little i n t(xmation about software contro l , architecture, and con trol flow.
Tradi tionally, operating systems have rel ied on data copying between user space and kernel space to pro tect the integrity of the kerne l . Although this method works tor most applications, for mu ltimedia appli cations, which usuaJly involve massive amou nts of data, the overhead of data copyi ng can seriously compromise the system's real-time pertcm1unce . " Fall
:111d Pasquale describe a mechanism of i n - kernel data paths that di rectly connect the source and sin k devices. '<> Peer- to-peer
IjO avoids unnecessary data copying and improves system and appl ication perfor mance. Kitamura et al . describe an operating system architecture, which they rerer to as the zero-copy architecture, that is also aimed at reducing the over head due to data copying. " The architecture uses memory mapping to expose the same physical addresses to both the kernel and the user-space processes and is especially suitable for multimedia operations . The J300 software is also a zero-copy architecture. No data is copied between system and user space.
The Windows
NT 1/0 su bsystem provides flexible su pport tor queue management. " What the
J300 achieved on the
UNIX and OpenVMS platforms through the command and event queues can be accom pl ished on the Windows
NT platform using built in su pport ti·om the l/0 manager.
A queue of pending requests (in the form of
1/0 request packets) may be associated with each device. The use of l/0 packets is similar to the usc of command and event packets in the
J 300 video software.
Summary
This paper describes the design and implementation of the so ftware architecture for the Sou nd
& Motion
]300 product, Digi tal's tlrst commercially available m u l timedia hardware adapter that incorporates audio and video compression. The prese ntation focused on those aspects of the design that place special emphasis on performance, on providing an intu itive API, and on supporti ng a client-server model of computing.
The software architecture has been successfully imple mented on the OpenVMS, Microsoft Wind ows NT, and Digital UNIX platforms. I t is the basis t()r Digital's recent PCI-based video adapter cards: Fui iVideo
Supreme and Fui iVideo Supreme }PEG.
The goals that i nfluenced the J 300 design have largely been realized, and the software is mature.
D igital is expanding upon ideas incorporated in the design. For example, one potential area tor i mprove ment is to replace the
F I FO service policy i n the vari ous queues with a priority-based mechanism . A second possi ble improvement is to i ncrease the usage of the hardware between periodic operations like "ideo cap ture. In terms of portability, the idea ofleaving device specific programming outside the kernel d river can be expanded upon to design device-independent kernel mode drivers, thus lowering overal l development costs. Digital is actively investigating these and other such enhancements made possible by the success of the J 300 project.
Acknowledgments
A number of people are responsible for the success of the J 30 0 fam i ly of prod ucts. The author gratefu l ly acknowledges the contri butions of members of the
]300 software and hardware development teams. In particular, special thanks to Bernard Szabo, the project leader for the
J 300 software; Paul Gauthier, for his invaluable assistance in getting the video l i brary com p leted and debugged ; John Hainsworth , for imple menting the device-i ndependent portion of the audio library; Davis Pan, for writing the DSP programs; and
Robert Ulichney, tor his guidance with the design and i mplementation of the video rendering subsystem .
The J 300 hardware design team was l ead by Ken
Correll and incl uded Tim Hell man, Peter Antonios,
Rudy Stal zer, and Tom Fitzpatrick. Nagi Sivananjaiah wrote the diagnostics that served us well in isolati ng hardware problems. Thanks also to members of the
Mu ltimedia Services Group, including Jim Ludwig,
Ken Chiquoine, Leela Oblicherti, and Chip Dancy, for being i nstru mental in incorporating the ) 300,
F u iiVideo Supreme, and FuiiVideo Supreme J PEG into Digital's multimedia server, and to Susan Yost, our tireless prod uct manager, tor diligently ensuring that the development teams remained on track.
References l . Information Tech nology-Digital Compression and Coding ofContirzllous-rone Still lnwiW'· Pa rt
7.·
Requirements wtd Gu idelines,
1 994
(
1'vlarch 1 994 ) .
ISO/IEC 109 1 8 - l :
2 .
Cod in,(}, q/klouing Pictures and Associated Audio for
Digital Storage Media a/ up to ahout 1 . 5 /Vlhit/s
Pm1 2 Video, ISO/ I EC 1 1 1 72 - 2 : 1 993 ( 1 99 3 ) .
50
Digiral ·r<:chnical Journal
Vol . 7 No. 4
1995
3 .
P.
B<l h l , P. Gauthier, and
R.
Ulich ney, "Software-only
Compression, Re ndering, and Playback of
Digital
Video," niuital Tec.:h n ica! Jou rnal, vol . 7, no. 4
( 1 995, rhis issue): 52-75.
4 .
A. Banerjea et a l . , "The Tener Real -Ti me Protocol
Suire: Design, l m p l cmcnrarion , and Experiences,"
TR-94 -059
( Berkeley, Cal i f. : I nrernational Computer
Science l nsriture, November 1 994 ) , also in
11::"/:::t./A CM
Transactions on Net u• o r ki n p,
( 1 99 5 ) .
5 . A. Bancrjc�1, E. Knightly, r.
Templ i n , a n d H . Zhang,
" Experi ments with rhe Te ner Real -Time Prorocol
Su ite on rhe Sequoia 2000 Wide Area Network," Pro ceeding<: of the
AC\1 Mu l timed ia
'94, San francisco,
Calif. ( 1 994 ) .
6 . W . Fen ner, L. Berc,
R. rrederic k, a n d S. McCanne,
" RTP Encapsulation of J P EG Compressed Video,"
I nternet Engi neering Task force, Audio- Video Trans port Worki ng Group ( M arch 1 99 5 ). ( l nternet draft}
7. S.
Nh:C:a n m: and V. )<Kobson, "\'ic: A Hcxible Frame work t(>r
Packer Video,"
Proceeding-; of the
ACivJ
Mu lli rned i a
95, San h:mcisco, Calif. ( 1 995 ) .
8.
M . Altenhokn et a l . , "TI1e B E RKOM M u ltimedia Co l la boration Ser\'ice," Proceedings media of the'
AC!\tl
Multi
'93.
Anahei m , Cal i f (A ugust 1 99 3 ) : 4 5 7-463.
9. K. Correll and R. U l ich ney, "The ) 3 00 fami l y of
Video and Audio Adapters: Arc hitecture and H ard ware Design," Digilul Technical Journal, vol. 7, no. 4 ( 1 995, this issu e): 20-33 .
1 0 .
S . Leftlcr, M . Mc Kusick, M . Karels, and ) . Quarterman,
The Desig n and Implementation of thf!
4.3
850
UNIX Operating System ( Reading, Mass . : Addison
Wesley, 1 98 9 ) : 5 1-5 3 .
I I. Puf"e
Code Modulation
( PCMJ of
Voice Freque n cies,
CCITT
Recom mendation G . 7 1 1 ( Geneva: I nter national Telecommu nications Union, 1 97 2 ) .
1 2 .
L. Rabiner a n d
Speed? Sig nals
Hall, 1 978 ) .
R.
S..:hatl:r, Digilal Processing of
( Englewood Cliffs,
N . J . :
Prentice
1 3 . D . Lee a n d S . Sud harsanan, " Design of a Motion J PEC
( M / J PEC )
Adapter Card ," i n
D(Q,ilal Vid£'0 Compres sion on Per.;onal Complllers. A lgorithms unci Tech nologv, l'roceedings of
SF/h.
Cal i f. ( rebruary 1 994 ) : 2-1 2 . vol . 2 1 87, San jose,
1 4 . M . Bolick a n d
].
Allen, "J PEG I mage Compression
Hardw:�re Implementation wirh Extensions for
Fi xed-rare and Compressed-image Editing Applica tions," in f)igital Video Compression on Personal
Computer:�·: Algorithms and
Techn olog y,
Proceed ings of
SJ->IJ-:, vol . 2 1 87, San J ose , Calif. ( Februarv
1 99 4 ) 1 3-2 2 .
1 5 .
) . Pasquale, " 1/0 System Design for
Intensive Jv1 ulti med ia l/0," Proceeding-; (1/ tbe Third
1/:'FJ:'
Wod� shop on �'(lorkstation Operation s:vstems. Asilom ar,
Cal i f.
( Ocrober 1 99 1 ): 56-67.
1 6.
K. Fall and
].
Pasq u a l e , " I m proving Continuous media Playback
Performance with In- kernel Data
Paths," Pmceedings a./the
1/'.FF Conference on Mul timedia Co mputing mul
S.vstems, Boston, Mass.
(June 1 994): 100-109.
1 7. H . Kitamura ,
K.
Ta niguchi, H. Saka moto, and
T. N ishida, "A New OS Arc hitecture for H igh Pe rfor mance Communication over ATM Networks,"
Proceedings o/lhe Workshop on l\"etti'Ork and Oper ating S).'stem Suppo11 ./(;r Digi!ctl A udio and Video
( April 1 99 5 ): 87-9 1 .
1 8 .
Jl!li
c ros o f i
Windows
NJ'
Det,ice Driver Kit
(Redmond,
Wash. : Microsoft
Corporation, )an uar\' 1 994 ).
Biography
Paramvi.r Bah!
Paranwir Bah! received B.S. E.
E. and M.S. E.
F.. degrees i n
1 987 and 1 988 from the Srate Universitv o f New York at
B u ffalo. Since joining Digital in 1 98 8 , he has contributed ro several seminal multimedia products i nvolving both hardware and software for digital video. Recently, he led the development of software-only video compression and video re ndering algorithms. A principal engineer in rhe
Systems B u siness Unit, Paranwir received Digital 's
Doctoral Engineering Fel lows hip Award and is completing his P h . D . at the University of
Massachusetts. There, his research has f(>eused on techniques for robust video com munications over mobi le radio IH:tworks. He is the author a nd ..:oaurhor of several scienri tic publ ications and a pe nd ing patenr. He is Jn active member of the IEEE Jnd ACM , serving on program committees of technical conteren..:es and as a reteree for their journals. Paramvir is a member of
Tau l3era Pi and a past president of Era Kappa N u .
Digital Tcc hni,al j o urnal
Vol . 7 No. 4 1 995 5 1
Software-only
Compression,
Rendering, and
Playback of Digital Video
I
Pa.ramvi..r Bah I
Paul S. Gauthier
Robert
A.
Ulichney
Software-only digital video involves the com pression, decompression, rendering, and display of d i gital video on general-pu rpose computers without specialized hardware. Today's faster processors are making software-only video an attractive, low-cost alternative to hardware sol utions that rely on specialized compression boards and g raphics accelerators. T his paper describes the building blocks beh ind popular
ISO, I TU-T, and industry-standard com p ression schemes, along with some novel algorithms for fast video rendering and presentation. A platform-independent software architecture that organi zes the functionality of com pressors and renderers into a unifying software inter face is presented. This architecture has been successfu lly i m plemented on the Digital UNIX, the OpenVMS, and IVI icrosoft's Windows NT operating systems. To maximize the perfor mance of codecs and renderers, issues pertain ing to flow control, optimal use of available resou rces, and optimizations at the algorithm ic, operating-system, and processor levels are con sidered. T he performance of these codecs on
Alpha systems is evaluated, and the ensu ing results validate the potential of software-only solutions. Finally, th is paper provides a brief description of some sample applications built on top of the software architecture, including an in novative video screen saver and a software
VCR capable of playing multi ple, com pressed bit streams.
Fu l l -motion vid eo is rast becoming commonplacc to uscrs of desktop compu ters. The rising expect:ttions [()r l o\\'-cosr, television-qualit\' 1·ideo 11·irb S\'nchronizcd sound ha1·c been pushing manuE1ctu rcrs to create ne11·, i nexpensive, high-quality oftl:ri ngs. The bottlenecks that have been prcvcnring the delive ry ofvidco without specialized hardw:m: arc being GISt aside rapid ly as bster processors, higher-bandwidth computer buses and n e tworks, and larger :�nd f1ster disk d ri1·cs arc being de1·doped . As a consequence, considcr:�ble atrcnrion is c urrently being r(JCuscd on eftlcient imple mentations ofHexible and extensible software solutions to the pro blems of video m:�n:�gcmem and d e l ivcrv.
This paper surveys the methods :�nd architectu res used in soft\\'are-onlv d igital video systems.
Due to the enormous amoums of data inH>h-ed , compression is almost always used in the storage �md transmission of vidw. The higb level of i n tcm1ntion red u ndancy in video lends itsc.l f wel l to compression, and many methods ha1·e been de1·clopcd to rake
J(h·antage of this bet. Vlhile rhc literature is replete with compression methods, 11-c r(>cus on those that arc recogni zed as standards, a rcquiremem ror open Jlld i nteroperable systems. This paper describes the build ing blocks be hind popu lar compression sche mcs of the International Organization r()r Standardization
( IS O ) , the I nternational Telecommun ication Union
Telccommunicnion Standardiz�nion Sector
(
! T L -
T ) , a n d within t h e industry.
Rendering i s �mother enabling technology rc1r 1·ideo on the desktop. It is the process of selling, color adjusting, quantization, and color space conversion of the 1·ideo t()r ri n:d presentation Oil the displa1·. t\s �lll example, Figure 1 shows a simple sequence of 1·ideo decod ing. In the section Video Presentation, we d is cuss renderi ng methods, along with some novcJ algo rithms for tast 1·ideo rend ering and presentation, and descri be an impleme ntation tiLH parallels rhe tech niques used in Digital's hardii'J.rc 1·ideo offeri ngs.
We rol loll' that d iscussion ll'ith the section
The
Software Video Li brary, in which we present a com mon arch itecture rc>r video compression, decom pression, and p l avback that al loii'S i ntegration into
Digital's multimcdi<l products . VVc then describe tl\'0 sJmple applications, the Video Odyssel' scrccn s�11 cr
52 Digit,ll Tcclmic;ll )ou mal
Vol . 7 No. 4
J99S
COMPRESSED
BIT
STREAM
Figure 1
Components in a
Video Decoder Pipe l ine
DECOMPRESS
PRESENTATION
� - - - - - - - - - - - - -
:
RENDER
I
DISPLAY
I :
1
._ _ _ _ _ _ _ _ _ _ _ _ _ _ j
:� nd
::1 software-only ,·ideo player. 'vVe conc l ude our paper by surveying re lated work in this rapidly e\'olv ing �1rca of software digital video.
Video Compression Methods
A svstem that compresses and decompresses video, whether i m plememed in h:�rdw;�re or software, is cal led �1 video codec ( ti>r compressor/ decompressor ) .
1\tlost video codecs consist o f a seq uence o f compo nems usually con nected i n pipeline t::tshion. The codec designer chooses speci tic components b::1sed on the design goa ls. By choosi ng the appropriate set of build ing blocks , a codec can be opti mi zed tor speed of decompression , rcl iabilitv of transmission, better color reproduction, better edge retention, or to pertim11 at a specitic rarger bit r:ne . t:or example, a codec cou ld be designed to trade off color q uality for transmission bit rate by removing most of the color int(mnation in the d:.1c1 ( color subsamplin g ) . Similarly a codec may incl ude a simple decompression model (Jess process ing per pixe l ) and a complex compression process to boost the playback r:l te at the expense of longer com pression times . ( Com pression algorithms that rake longer to compress th:111 to decompress arc said to be
�lS\'mmetri c . ) Once the components and trade-ofts h�we been chosen, the designer then ti ne tllnes the codec to perform wel l in a spccitic appl ication space such as teleconkrencing or video browsing.
Video Codec Building Blocks
I n this section, we pn:sent the various building blocks behind some popu lar and industry-standard video codecs. Knowledge of the [(>!lowing video codec eomponellts is essential t()r u nderstanding t he com pression process :md ro �l ppreciate the complexity of the algorithms.
Chrominance Subsampling
Video is usual l y described as being composed of :1 sequence of i mages. E<lch image is a matrix of pixels, and each pixel is repre sen ted bv three 8-bit values: a single lum inance \'alue
(Y) that signitles brighmess, and two chrominance \'al ues ( U <llld V, or sometimes Cb and Cr) whic h , taken together, specit·\, a unique col or. By red ucing the amount of color i n timn.nion in relation to lum inance
( s u bsampling the chrominancc ), we can red uce the size of an image with little or no perceptual efkct. The most common chrom inance subsa mpling tec lmique decimates the color signal by 2 : 1 in the h orizontal direction. This is done either by si mply throwi ng out the color inri:m1ution of alternate pixels or by averag ing the colors of two adjacent pixels and using the average tor the color of the pixel pair. This technique is commonly re krred to as 4 : 2 : 2 su bsampling. When compared to a ra,,· 24-bit image, this results in a com pression of two-thirds. Decimating the color sign�1l
Lw
2 : 1 in both the horizontal and the verric1l direction
( by ignoring color i n ri:m1ution tor al ternate lines in the image) stJrts to result in some perceptible loss of col or, but the com pression increases to one-half. This is referred to JS 4 : 2 : 0 s u bsampling: [()r every 4 l u m i nance samples, there is a single color specitied b v a pair of chrominance values. The u ltirnJte chrominance subsampl ing is to throw a\\'aV all color int(mn;�tion a nd keep only the l u mi nance datJ ( monochrome video ). This not onlv red uces the size of the input dat�l but also greatly simplifies processing tor borh the com pressor and the dccompressor, resulting i n bsrcr codec pedormance. Some telccontcrencing systems allow the user to switch to monochrome mode to increase t!-ame rate .
Tra nsform Cod ing
Com·ening J sign�ll, ,·ideo or otherwise , ti-om one representation to another is the task of a transtimn coder. Transtonns em be usefu l tor video compression i f they can convert the pixel datJ into a form i n which redu ndant and insigniticmt i n fix marion i n the video's image can be isobtcd and removed. Many transtimns convert the spatial (pixe l ) data into tl-equency coefticienrs that can then b e selec tively elimi nated or qu antized. Tr:mstimn coders address tbrce central issues in i m age coding:
( I ) decor relation ( converting statistically dc pmdcm image clements i n to independent spectral coefficients ) ,
( 2 ) e nergy c o m p:�ction ( redistrib u tion a n d loc:llization of energy i n to :1 sm:� l l n u mber of coeft!cienrs), <111d
( 3 ) computational complexity. It is well documented that hu man vision is biased toward low ti-cquencies.
B y transforming :�n image to the tl-cquency domai n , a codec can capitalize on this knowledge and remo,·e or reduce the high - tl-equencv components in the quantization step, dkctivelv compressi ng the image .
I n addition, isolating a n d elimi nating high-tl-el)uency components i n an image results i n noise reduc tion since most noise i n video, introd u ced during
Vol . 7 :-.:o. 4 I '195
53
the digitization step or from transmiSSion interfer ence, appears as high-frequency coefficients. Thus transf(>rming helps compression by dccorrclating (or whin:ning) signal samples and then discarding nonessential information ti-om the image.
Unitary ( or orthonorma l ) transforms fal l int o either of two classes: fixed or adaptive. Fixed transforms are independent of the input signal; adaptive transforms adapt to the input signa l . ' Examples of fixed trans forms include the d iscrete Fourier transform ( OFT) , the d iscrete cosine transform ( DCT), the discrete sine transform ( DST), the Harr transform, and the Walsh
Hadamard transform ( WHT) . An example of an adaptive transform is the Karhunen-Loeve transf()rm
( KLT ) .
Thus far, no transform has been found tor pictorial information that completely removes statisti cal dependence between the transform coordinates.
The KL T is optimum in the mean square error sense, and it achieves the best energy compaction; however, it is com putationally very expensive. The WHT is the best i n terms of computation cost since it requires only additions and subtractions; however, it performs poorly in decorrelation and energy compaction.
A good compromise is the OCT, which is by tar the most widely used transform in image coding. The
OCT is closest to the KL T i n the energy-packing sense, and , l i ke the DFT, it has fast computation algorithms available f(x its implementation 2 The DCT is usually applied in a sliding window on the image with a com
mon window size of 8 pixels by 8 lines ( or simply, 8 by
8 ) . The window size (or block size) is important: if it is too small , the correlation between neighboring pixels is not exploited ; if i t is too large, block bou nd aries tend to become very visible. Transti>rm coding is usually the most time-consuming step in the compression/decompression process.
Scalar Quantization A companion to transform cod ing in most video compression schemes is a scalar quantizer that maps a large number ofinput levels into a smal ler number of output levels. Video is com pressed by reducing the number of symbols that need to be encoded at the expense of reconstruction error.
A quantizer acts as a control knob that trades off image quality tor bit rate. A careti.dly designed q uan tizcr provides high compression tor a given q uality.
The simplest form of a scalar q uantizer is a uniform quantizer in which the q uantizer decision levels are of equal length or step size. Other important quantizers i nclude Lloyd Max's minimum mean sq uare error
( M M S E ) quantizer and an entropy constraint quan tizer. ' ' Pulse code modulation ( PCM ) and adaptive differential pulse code modulation ( AD PCM ) arc examples of two compression schemes that rely on pure q uantization without regard to spatial and tem poral redu ndancies and without exploiting the non linearity in the human visual system.
54 Digital 'Tc,hni,al Journal
Vol . 7 No. 4
1995
Predictive Coding Unless the image is changing rapidly, a video sequence wi l l normal ly contain sequences of frames that are ver�' similar. Predictive coding uses this fact to red uce the data vol ume by comparing pixels i n the current frame with pixels i n t h e same location in t h e previous frame a n d encoding the difference. A simple form of predictive coding uses the value of a pixel in one frame to predict the value of the pixel in the same location in the next trame. The pred iction error, which is the difference bet\vcen the predicted value and the actual value of the pixel, is usual ly sma l l . Smaller numbers can be encoded using tewer quantization levels and fewer coding bits. Often the d i fference is zero, wh ich can be encoded very compactly. Predictive coding can also be used within an image frame where the predicted value of a pixel may be the value of its neigh bor or a weighted average of the pixels in the region. Pred ictive coding works best if the corre lation bct\veen adjacent pixels that are spatially as well as temporally close to each other is strong. Differential PCM and delta modu lation ( DM ) arc examples o f two compression schemes i n which the predicted error is q uantized and coded . The decomprcssor recovers the signal by applyi ng this error to i ts predicted val ue tor the sample. Losslcss image compression is possible i f the prediction error is coded without being quantized .
Vector Quantization An alternative to transform based cod ing, vector quantization attempts to repre sent c lusters of pixel data ( vectors ) in the spatial domain by predetermined codes.5 At the encoder, each data vector is matched or approximated with a code word in the codcbook, and the add ress or i ndex of that code word is transmitted instead of the data vector itself. At the decoder, the index is mapped back to the code word , which is then used to represent the original data vector. Identical codebooks are needed at the compressor ( transmitter) and the decompressor
( receiver) . The main complexity l ies in the design of good representative codcbooks and algorithms f(>r finding best matches efficiently when e xact matches are not available. Typical ly, vector quanti zation is applied to data that has already undergone pred ictive coding. The prediction error is mapped to a su bset of values that arc expected to occur most t!·equently.
The process is called vector q uantization because the values to be matched in the tables are usual ly vectors of t\vo or more values. More elaborate vector quantiza tion schemes arc possible in which the difference data is searched tor larger groups of commonly occurring values, and these gro u ps arc also mapped to single index values.
The amou nt of compression that resu l ts from vec tor quantization depends on how the val ues i n the codebooks arc calculated . Compression may be adjusted smoothly by designing a set of codebooks
and picking the appropriate one for a given desired compression ratio.
Motion Estimation and Com pensation Most codecs that use i ntertrame compression use a more elaborate form of predictive coding than described above . Most videos contain scenes in which one or more objects move across the image against a fixed background or in which an object is stationary against a moving back ground. I n both cases, many regions in a frame appear in the next frame but at different positions. Motion estimation tries to find similar regions in two frames and encodes the region in the second frame with a dis placement vector ( motion vector) that shows how the region has moved . The technique relies on the hypothesis that a change i n pixel intensity from one frame to a nother is due on ly to tra nslation.
For each region ( or block) in the current frame, a d isplacement vector is evaluated by matching the information content of the measurement window with a corresponding measurement window \V within a search area
.\ placed in the previous frame, and by searching for the spatial location that minimizes the matching criterion r.i.
Let L/ x,y) represent the pixel intensity at location ( x.y) in fi·ame i; and i f (d,,d,.) rep resents the region displacement vector f(x the interval n
(
=
( i+ n
)
i), then the matching criterion is defined as r.l
=
1r1�11}t:t
1
[
,�, J I
L,(x,y)
( 1 )
The most widely used d istance measures are the abso lute value l lx
I I = lx I and the q uadratic norm l l x l l =x2. Since finding the absol ute minimum is guar anteed only by perf(xming an exhaustive search of a series of d iscrete candidate displacements within a maximum d isplacement range, this process is com putationally very expensive. A single d isplacement vector is assigned to all pixels within the region.
Motion compensation is the inverse process of using a motion vector to determine a region of the i mage to be used as a predictor.
Although the amount of compression resulting from motion estimation is large, the coding process is time-consu ming. Fortunately, this time is needed only in the compression step. Decompression using motion estimation is relatively fast since no searching has to be done. For data replenishment, the decompressor sim ply uses the transmitted vector and accesses a region in the previous frame pointed to by the vector tor data replenishment. Region size can vary among the codecs using motion estimation but is typically
1 6 by
1 6 .
Frame/Block Skipping O n e technique for red ucing data is to eliminate it entirely. In a teleconfe rencing sit uation, for example, if the scene docs not change
(above some threshold criteria), it may be acceptable to not send the new frame ( d rop or skip the frame).
Alternatively, i f bandwi d th is limited and image quality is important, it may be n ecessary to d rop frames to stay within a bit-rate budget. Most codecs used in telecon ferencing applications have the ability of temporal sub sampling and are able to gracefu l l y degrade under limited bandwidth situations by d ropping frames.
A second f(>rm of data elimination is spatial subsam pling. The idea is similar to chrominance subsampling d iscussed previously. In most transtcxm-based codecs, a block (8 by 8 or 1 6 by 1 6) is usually skipped if the d i fference between it and the previous block is below a predetermined threshold. The d ecompressor may reconstruct the missing pixels by using the previous block to pred ict the current block.
Entropy Encod ing Entropy encoding is a form of sta tistical coding that provides lossless compression by coding input samples according to their frequency of occurrence. The rwo methods used most frequently include H u ffman coding and run- length e ncoding.''
H u ffi-nan coding assigns tewer bits to most frequently occurring symbols and more bits to the symbols that appear less often . Optimal H u ffman tables can be gen erated if the source statistics arc known . Calcu l ating these statistics, however, slows down the compression process. Consequently, predevelopcd tables that have been tested over a wide range of source images arc used . A second and simpler method of entropy encod ing is run-length encoding in which sequences of identical d igits arc rep laced with the digit and the number in the sequence. Like motion estimation, entropy encod ing puts a heavier burden on the com pressor than the decompressor.
Before ending this section, we wou ld like to mention that a number of other techniques, including object based coding, model-based codi ng, segmentation based coding, contour-texture oriented coding, fi·actal cod i ng, and wavelet coding arc also available to the codec designer. Thus far, our coverage has concen trated on explaining only those techniques that have been used in the video compression schemes currently supported by Digital . In the next section, we describe some hybrid schemes that employ a number of the techniques described above; these schemes are the basis of several international video coding standards.
Overview of Popular Video Compression Schemes
The compression schemes presented in this section can be col lectively classified as first-generation video coding schemes.7 The common assumption in all these methods is that there is statistical correlation between
Digital
Technical journal
Vol .
7
No. 4 1995 5 5
pixels. Each or· rhcse methods attempts to exploit this correlation by e mployi ng redu ndancy reduction tech niques to achieve compression.
Motion-J PEG Algorithm compresses each
Motion- ) PEG (or
M - J PEG ) f l
·
a
me of
a
video set] Ucnce using the
ISO's
J
o
i n t Photographic Experts Group (JPEG )
continuous-tone, sti l l - image compression
standard ."
As such ,
it is
an
i nt rafl·ame compression scheme. I t is not wed
to
�my particular subsampling fcm11at, i mage color space, or i
m
a
g
e dimensions, but most typica lly
4 : 2 : 2 subsampled YCbCr, sou rce input fcm11at ( S i f,
352 by 240) data is used . The ) PEG standard specifies both lossy
�md
lossless compression schemes. For video, only the lossv baseline fJCT coding scheme has gained acceptance. The scheme relics
on
selective quantization of the ti·equency coefficients f()J iowed by
H u ffman and run-lengt11 e ncoding f()r its compres sion. The stanLhrd ddi ncs a bit-stream format that contains both rhe compressed data stream
and
coding
parameters such as the number of components, quan tization t:Jblcs, Huftlnan tables, and sampling facrors.
Popu lar
M
-
J
P
EG
file
formats usually build on top of the ) PEG-specified formats with little or no modi fica tion . For example, Microsoft's audio-video interleaved
( AV I )
tixm
at encapsu lates
each J
PEG
frame with its associated audio :md adds an index to the starr of each fi·arne at the end of the ti le. Video editing on a framc b\'-ti·ame basis is possible with this fcmnat. Another advantage is tl·:tme-limited error prop:�gation in net worked, distributed appli cations. Many video d igitizer boards incorporate
] P EG
compression in
hardware to
compress and decompress video i n real time. Di
g
i tal
's
Sound
&
Motion J300 and Fui!Video Su preme ] PEG are two such boards.'' '"
The baseline J
PEG codec is a
S\'tnmctric �1lgorirhm as may b e seen i n F i g u r e
2a and
Figure 3 .
ITU-T's Recommendation H.261 T he
ITU
-
T's
Recom ·
mendation
H
. 2 6 1 is a motion-compensated, DCT based video co
d
i n
g
standard . " Designed f()r the teleconferencing market and developed primarily for low- bit-rate I ntegrated Services Digital Network
( IS D N ) services, H .2 6 l shares simi larities wirh ISO's
) PEG
still-image compression standard.
The
target bit rate is p
X
64 kilobits per second with
jJ
rangi ng between
1
and 30 ( H .2 6 l is also known as p
X
64 ) .
Only two t[·an1C resol u tions, common intermediate format ( C ! F, 352 bv 2 8 8 )
and
q uarter- C I F ( QC I F ,
1 76 bv 1 44 ) , arc allowed .
AJI st:�ndard -compliant
codecs must
be
:t
bl
e
to operate \\"ith QCI F; CIF
is
optional. The
input
color space
is
fixed
b
y
the
I nternational Radio Consul tative Committee ( CCIR)
60 1 YCbCr standard 's with 4:2:0 subsa
m
plin
g
(sub· sampling of chrominance componenrs by 2 : 1 i n both the horizontal and rhe vertical directio n ) .
T
wo types of frames arc ddi ned : key ti·ames th:-tt are coded
56 Digiul Tcchnicll
]ournJ!
Vol . 7 No. 4 1995
i ndependently and non-kev frames that arc coded with respect
to
a previous frame. Key frames are coded i n a man ner similar to JPEG . For
n
o
n
-key fi·ames, block- based motion compensation is per fc>rmed to compute interfl·ame difkrences, which are then DCT coded and quantized .
T
h e block size is
1 6
bv
16, and each block
can have a
difkrcnr q uanti
zat
io
n table. FinJIIy, a variable word-length
encoder
(usual ly employing
H u ftl11an and ru n
l e n
g
t
h methods) is used for coding the q uanti zed cocfticicnrs. Rate control is done by d ropping frames, skippi ng blocks,
and
increasing qu:�ntization . Error correction codes arc em bedded in the bit
s
tre
a m
to help detect and possiblv correct transmission errors. Figu re 2b shows a block diagram of an H .2 6 l decomprcssor.
ISO's M P EG-1 Video Standard The M PEG- 1 video standard was develop e d by ISO
's
Motion Picture
Experts Group ( M PEG ) . Li ke the
H .26 1 algorithm,
M PE
G- 1 is also an i medrame video codec that removes spatial rcd undancv bv compressing kev ti-;lmes u s i n g techniques s i m i l a r to J PEG and removes te m po r a l redundancy rh r o u g h motion estimation and compensation ' l.1
� The standard defines three d i ffer e n t types of frames or pictures: intra or I
- ti·ames that are compressed independently; predictive or P-tl·ames that use motion compensation from the previous I or
P - tl·ame; and bidirectional or B ti-ames rhat contain blocks predicted fi·om either a preceding or t<
>
J iowing
Por
1 - frame
( or i nterpolated from bo t h
)
.
Com pres sion is greatest for B - f[·ames and Jcasr fc>r ! - frames.
(A kmrth type of ti·anlC, called rbc D-ti·amc or the
DC. inrracoded trame, is a lso defined tor i mproving
Elst- forward -type access, but it is hardly ever used . )
There is n o restriction on the input ti·arnc dimensions, though the target bit rare ot"l .5 m egabit s per second is t{lr
\'ideo c o n t ain i ng
S i r h· a m e s.
Subsampling is fixed
;H
4 : 2 : 0 .
M PEG - 1 employs ad a ptiv e qu::unization of
DCT coefficients tc>r co mp r e s s in g
I -fi-:um:s a n d for compressing the difkrcnce bet\veen actu:�l and pre d icted blocks in
Pand
B ti·am es .
A
1 6- by-
I
6 sliding window, called a macroblock, is u sed in motion esti · marion; a nd a variable word - length encoder is used i n r h e final step to fu rther lower the output b i t rate . The fu l l
M PE G - 1 stambrd speci fies a system stream that includes a video ;lnd an audio subsrream, :�long with ti m in g information needed for svnchronization bet\vcen the t\vo. The video substrcam contains the compressed video dar:� and coding parameters such a s p i ct ur e rate, bit rate, and image size.
M PEG- l has become increasinglv popular primari l y because it ofkrs better compression than
JPEG without compro· mising o n q u al i t y
Se v e r a l v e n d o r s an d c h i p manu · facturers ofkr specialized hardware tc
> r M PEG compression and d eco m p r ession . Figu re 2 c shows
:t block diagram of a n
M
PEG- l video dccom pressor.
COMPRESSED
BIT STREAM
( a ) Baseline ] P EG Dccom pressor ( ISO Srandard , 1 99 2 )
CODE TABLES Q TABLES
COMPRESSED
BIT STREAM
L
ON/OFF CONTROL
MOTION VECTORS
( b ) Recommendation
H
.26 1 Decom prcssor
(
I T U -T Srandard, 1 990 )
4 : 1 · 1
YCBCR
CODE TABLES
Q TABLES
COMPRESSED
BIT STREAM
L
ON/OFF
CONTROL
FUTURE
PICTU R E STORE
MOTION
VECTORS
i
i
·
����
TYPE
· - · r
-
· - ·
�;
r
·
·
�
1
_.... PREVIOUS
PICTURE STORE
( c ) J\!! PEG- 1 Video D c c o m p r ess o r ( ISO Sra ndard, 1 994)
Figure 2
Playback Contigurarions for Compressed Video Srrcarns
Figure 3
!SO's Base line j
PEG C:ompresso1-
AC COEFFI C I E NTS
Q TABLES
Dig;ir;ll Technica l journal
COMPRESSED
BIT STREAM
Vol. 7
No. 4 1 995 57
Intel's INDEO Video Com pression Algorithm
Intel 's proprietary INDEO video compression algorithm is used primarily for video presentations on personal computer ( PC) desktops. I t employs color subsa m pling, pixel differencing, run-length encoding, vector quantization, and variable word -length encoding. The chrominance components are heavi ly subsampled . For every block of 4-by-4 luminance samples, there is a single sample of Cb and Cr. Furthermore, samples are shifted one bit to convert them to 7 -bit values. The resulting precompression t(xmat is cal led because on average there are
YVU9,
9 bits per pixel. This subsampling alone yields a reduction of 9 /24. Run length encoding is employed to encode any run of zero pixel ditkrences.
PCWG's IND EO-C Video Compression Algorithm
I N D EO-C is the video compression component of a telecon ferencing system derived fro m the Personal
Conferencing Specification developed by the Personal
Conferencing Work Group ( PCWG ) , an ind ustry group led by Intel Corporation. Like the M PEG stan dard, the PCWG specification ddines the compressed bit stream and the decoder but not the encoder.
INDEO-C is optimized for low-bit-rate, !SON-based connections and, u n like its desktop compression cousin, is transform- based. It is an interframe ;�] go rithm that uses motion estimation and a 4:1 chromi nance subsampling in both d irections. Spatial and temporal loop filters are used to remove high frequency a rtifacts. The transform used for converting spatial chta to frequency coefficients is the slant trans form, which has the advantage of requiring only shifts and adds with no m u l tiplies. Like the DCT, the fast slant transform (FST) is applied on image subblocks for coding both intrafi·ames and diftcrence frames. As was the case in other codecs, run- length coding and
Huffman coding are employed in the final step.
Compression and decompression of video in software is faster than other interframe schemes like M PEG- 1 and H . 26 l .
Compression Schemes u nder Development
I n add i tion to the five compression schemes described in this section , tour other video compression standards, which are currently in various stages of development within ISO and ITU-T, are worth mentioning: !SO's
M PEG-2, ITU-T's Recommendation H .262, ITU-T's
Recom mendation H .263, and !SO's MPEG-4. u 1'
Although the techniques employed in MPEG-2,
H .262, and H .263 compression schemes are similar to the ones discussed above, the target applications are different. H.263 focuses on providing low-bit-rate video ( below 64 kilo bits per second) that can be trans mitted over narrowband channels and used for real time conversational services. T h e codec wou ld be employed over the plain old telephone system ( PO
T
S
) with modems that have the V.32 and the V.34 modem technologies. M PEG-2, on the other hand, is :limed at bit rates above 2 megabits per second, which support a wide variety of formats for multimedia applications that req uire better quality than M PEG- 1 can achieve .
One of the more popular target applications for
M PEG-2 is coding for high-definition television
( H DTV) . It is expected that ITU-T will adapt M P EG-2 so that Recommendation H .262 w i l l be very similar, if not identical , to it. Finally, like Recommendation
H . 263, ISO's M PEG-4's charter is to develop a gene1ic video coding algorithm for low-bit-rate multi media applications over a public switched telephone ncr.vork.
( PSTN ) . A wide variety of applications, including those operating over error-prone rad io channels, arc being targeted . The standard is expected to embrace cod ing methods that are very different from its precur sors and will include the so-called second-generation coding techniques.7 M PEG-4 is expected to reach draft stage by Novem ber
1997.
This ends our d iscussion on video compression tech niques and standards. I n the next section, we turn our attention to the other component of the video play� back sol ution, namely video rendering. We describe the general process of video renderi ng and present a novel algorithm tor efficient mapping ofout·of.range colors to teasiblc red , green , and blue ( RG B ) val ues that can be represented on the target display device. Out-of range colors can occur when the d isplay quality is adjusted duri ng video playback.
Video Presentation
Video presentation or rendering is the second impor tant component in the video playback pipeline (see
Figure l
). The job of this su bsystem is to accept decompressed video data and present it in a window of specified size on the display d evice using a specified number of colors. The basic components are sketched in Figure 4 and described in more detail in a p revious issue ofrhis]ournal.15 Today, most desktop systems do not include hardware options to perform these steps, but some interesting cases are avai lable as d escribed in this issue 9·1" When such accelerators are not available, sofuvare-only impleme ntation is necessarv. Software
DECOMPRESSED
YUV
COLOR ADJUST SCALE DITHER
COLOR SPACE
CONVERT
RGB COLOR
INDEX
Figure 4
Componenrs ofVideo Rendering
58
Digital Technical Journal Vol . 7 No.
4 1995
rendering algorithms, although very efficient, can still consume as many computation cycles as are used to decompress the data.
All major video standard s represent im age data in a l u minance-chrominance color space. In this scheme, each pixel is composed of a single l u minance compo nent, denoted as Y, and two chrominance components that are sometimes rete rred to as color difference sig nals Cb and Cr, or signals
U and V. The relationship between the familiar RGB color space and YUV can be d escribed by a 3 - by-3 linear transf(xmation: r
8 h
= M u
( 2 ) where the transf(Jrmation matrix ,
1
0 a
M
=
1 h c
1 d
0
( 3 )
The matrix i s somewhat simple with only four values that are not 0 or l . These constants Jre a =
1 .402 , h
= -
. 344, c
= -
. 7 1 4 , and d
=
1 .722.
The RG B color space cube becomes a para l le lepiped in YUV space. This is pictured in Figure 5, where the black corner is at the bottom, and the white corner is at the top; the red , green, and blue corners are as labeled . The chrom inance signals U and V are usually subsamplcd , so the rendering subsystem must first restore these components and then transfcmn the
YUV triplets to RGB values.
Typical frame buffers are c o n fi g u re d with 8 bits of color depth . This hardware colormap must, in general, be shared by m ultiple applications, which puts a pre mium on each of the 256 color slots in the map. Each application, therefore, must be able to req uest render ing to a limited number of colors. This can be accom plished most dkctivcly with a m u l tilevel d ithering scheme, as represented by the dither block in Figure 4. y
B
The color adjustment block controls brightness, con· trast and saturation by means of simple look-up tables.
Along with up-sampling the chrominance, the scale block i n Figure 4 can also change the size of the i mage. Although arbitrary scaling is best performed in combination with filtering, it is found to be too expen sive to do in a softwar e only implementation. For the case of enlargement, a trade-off can be made bet\¥een image qual ity and speed ; contrary to what is shown in
Figure 4, image enlargement can occur after dithering and color space converting. Of course, this wou ld result i n scaled d ithered pi xel s
, w h ic h a r e c e r tai nl y less desirable, but it wo u ld also result in taster processing.
To optimize computational efficiency, color space conversion from Y UV to RGB takes place after YUV d ithering. Dithering greatly reduces the n u m ber of
YUV triplets, thus al lowi ng a single look-up table to perform the color space conversion to RGB as well as map to the final 8 - bit color index required by the graphics d isplay system . Digital pioneered this idea and has used it i n a number of hardware and software only products . '7
Mapping Out-of-Range Colors
Besides the obvious advantages of speed and simplic ity, using a look-up table to convert d ithered YUV val ues t o R G B values has t h e added feature of allowing carefu l mapping of out-of: range Y UV values. Refer ring again to Figu re 5, the RGB solid describes those r, g, and h values that arc feasible, that is, have the nor malized range 0
<;:: r, g, h <;;
1 .
The range ofpossible val ues i n
YUV space are those for 0
<;:: y
<;:: l and - . 5
<;:: u , u
<;:: . 5 . It turns out that the RGB
2 3 . 3 percent of this possi ble solid occupies only
YUV space; thus there is ample possi bility for so-called infeasible or out-of range colors to occur. Truncating the r, g, and h val ues of these colors has the efkct of mapping back to the
RGB parallelepiped along lines perpend icular to its nearest surface; this is undesirable since it will result in changing both the hue angle or polar orientation in the chrominance p l a n e and t h e luminance value. By storing the mapping in a look- up table, decisions can be made a priori as to exactly what values the out-of. range values should map to.
There is a mapping where both the luminance or y value and the h ue angle are held constant at the expense of a change in saturation . This section details how a closed-form solution can be found for such a mapping. Figure 6 is a cross section of the volume in
Figure 5 through a plane at y =Yo ·
The object is to find the point on the surtace of the RGB parallelepiped that maps the out-of:range point (y0,
U0, q,) in the plane of constant y.,
( constant l u minance) and along a straight
I ine to the u-u origin ( constant hue angle) . The sol u tion is the i ntersection o f the closest RGB surface and the line bet\veen (y0,
1 10,
1!0) and (y,. 0, 0). This l ine can
Figure 5
The RGB "Cube" in
YUV
Space
Digital
Technical journJI
Vol .
7 No. 4
1 995 59
FEASIBLE RGB REGION
DESIRED SURFACE POINT
.5
r-v
��
----+-----�
-.5
Figure
6
Mappi ng Our-of-Range Y U V Points to rile Surfxe ofrhe
RCI3
P;lr;l l k lcpiped in a
PJJnc of Consranr .H. be pa r ametrica l ly represented as rhc locus (.H, a 11,, a1\,) t(>r
'
' singl e parameter a. The RG B values t(>r these points are where
'---------------1-------+------__j r
,
..,
= M ex 11" h o. u o
.5 ex(C/1' 11 ) + _)' " ex( I?
/10 + ct •11) + ) � , cx.(dll , ) +y,
( 4 )
the matrix M
is as given in equ
J tion ( 2 ) . To t-ind where this parametri c line wi ll intersect the RGB paral lelepiped, we can t-irst solve for the ex at rhc intercept val ues ar each of the six bounding surbce p!Jncs :�s tc>llows:
Surface
Plane r = l
,r;
= 1 b = l r = O g = O h = O
In tercept
Val ue ex ,
=( 1 -y,)/cn\, ex, = (
1 -y,)/( b 11, +u;,) ex., = ( 1 -y11 )/dll0 ex, = (ex, - 1 ) ex,
= ( ex , 1 ) ex6 = (ex., - 1 )
Exactly three ex ; will be negative , with each desc r i bi ng rhe in t er cept with extended RGB surr:1ce planes oppo si te the
Ii-i' origin .
Of the r e main in g th ree ex ; , the two l arg e st values will describe imercep r s with extended
RG B stu·bce plan e s in infeasible RGB space . This is because rhe RG B volume, a pa r al lelepiped
, is a convex polyhed ron. Thus the solu tion must simply be the smallest p ositive a ; .
Plugging r his vJiue of ex into equa tion (4) prod uc e s the des i red RGB value.
The Software Video Library
When we started this p r oject, we had two objectives in mind: to showcase the processing po\\'er of Digita l's newly de,·clopcd Alpha processor and to usc rhis power ro mJkc dig:iral video easily ava ila bl e to devel
opers and end users by providing extremely l ow-cost solutions. vVe knew that because of the compute i ntensive mturc of video p ro cessi n g, Digita l 's Alpha processo r would outpc r t-<m11 any competitive proces sor in a head ro head match. By providing the Jbilit v to manipulate good
q u: tli t v desktop video without the need for addition:�! hardwJrc, we wanted to m:�kc
Alpha-based S\'Stcms the computers of choice t(>r end users who \\'Jntcd to incorporate mul timedia into their applications.
Our objecri,·cs translated ro r he creation of
:1 soft ware video li bc1ry th a r beca me a realirv bccJuse of three key observJtions. The first one is embedded in our motivation: processors had become p owerfu l enough to pertcm11 complex signa l-processing opcr<l tions at real -rime r:�tcs. With the p o te nt ia l of even greater speeds in the n c ar future, low-cost m u l timed i;l solutions wou ld be possible since a ud i o and video decompression co u ld be done on the nJtive processor without anv ;ldd irionJI harchl'<lre .
A second obscn·ation \\'as that m u l tiple e m ergi ng aud io/video compression standards, both tormal am! industrv de bcro, \\ e re g;1ining populari tY with ::�ppl i cation vendors :-tnd henc e needed to be su pported on Digi ta
I 's p l ;ur<>rms. On carefu l exami n J tion of the c ompression ;: dgorirhms, we observed that most of the pro minent schemes used com m on bui lding blocks ( sec F i gure 2 ) . For example , a l l rive interna tion a l st:�mbrds-) PEC, i'vl PEG- 1 , M PEG-2, H . 26 l , and H .2 63-have DCT- based transform coders t(>l lowed by a q u ;lnti zcr. Simi larl y, Jll rive usc Hufti11an co di n g i n their ti n:� ! step. This meant thar ,,·ork done on one codcc cou ld be reused t(>r others .
A t h i rd observ;ltion w:�s that the most com mon component of
,
i d eo- base d appl ications
11·as
,
i d eo plavback ( t(>r example, ,-ideoconfcrcncing, ,·idco-otl dcmancl , ,·ideo pi aver, :md desktop teb ision ) . The output decom p ressed streams from the various decoders h
: w e to be sofrware-rendercd for d i sp l a l
' on systems that do not have support for color space con version and dithering in their graphics ad apt ers. An efficient so f twJ tc r e ndering scheme cou ld rhus be shared by ;\ I I video pbycrs.
VVith these observations in mind, we deve loped a software video l ibrJrl' contai ning qua lit v imp lem en tations of
ISO, ITU-T, :1nd industrv de tacto ''ideo coding stand ards. In rhc sections ro tol lo,,·, \\'e presc llt the architecture, implementation, opti m iz a tion, :�nd perf o rman ce of the soft\\'are \'ideo library. We c om plete o u r presenrJtion by describing examples of video based applicnions 1\'rirrcn on top or- this l ibr an · ,
60 Digiral Tcchnic.ll Journal
Vol .
7 o.
4 1 99 5
including a no\'l:l video screen saver we e:�l l Video
Odyssey and
:1 soth v a r e
o nl y video pbver.
Architecture
Keeping in mind the obsen·ations outlined :�bove, we designed a sort\\':IIT video li brary ( S U B ) that would
•
•
•
•
•
Provide a common architecture under which m u l tiple <lltdio and ,·idco codecs and rcnderers could be accesscd
Be the lo\\'eSt, fu ncrion a l l v complete l:wer in the software ,·ideo codec hierardw
Be bst, cxtensiblc, and thrcad-sak, providing reen trant code \\'ith minimal overhead
Providc an intuitive.:, simple, tlcxiblc, and extens ible :�ppliotion programming intcrt;Ke ( A P I ) that su pports
:t dient s ervcr model o f mu ltimedia computing
Provide an API that would accommodate m u lt i ple u pper laycrs, Jl!owing t(>r easy and seJmless i ntcgra tion into Digital's mu ltimcdi<l products
Our imention was not to create a library that wou ld be cxposcd to end- user :�pplications but to crcate one that wou ld provide a common architecture tor video codecs h>r casy integration i mo Digital's m u ltimedia products. SLI B's A l'l was purposely c ks i g n ed to be a su perset of Digit<l i 's M ultimedia Services' API tor greater tlcxibilitv in terms of algorith m ic tuning and control. Thc librarv w o u ld tit we l l under the actual programming interface provided to cnd users
Digital's Multimedia by
Services. Digital's M u l timedia
AP I is the same as Microsott's Video For Windows
API, which tacil itates the porting of m u ltimcdia appli cations from Windows a n d Windows N T t o Digital
UNIX and OpenV1VIS platforms. Figure in relation
7 shows S U B to
Digital 's mu lti media sofu,'are hierarchy.
The shaded regions indicate the topics discussed in this paper.
As mentioned , the library comains routines tor audio and video codecs and Digita l 's propricty video rendering algorithms. The routines are both algorithmicallv and optimized tor the parricuiJr p!Jttorm on which they are ofkred . The sothvare has been success fully i mplemented on multiple plattonns, including the Digital
Windows
U N I X , the OpenVMS, and Microsoft's
NT operating systems.
Three classes of routines are provided tor the three subsystems:
( l ) video compression and decompres sion, ( 2 ) video rendering, and ( 3 ) audio processing.
For each su bsystem, routines em be ti.trtbcr classified as ( a ) setup routines, ( b ) action routi nes, ( c ) q uery rou tines, and ( d ) teardmvn routines. Setup routines create and initialize all relevant internal d::tta structures. They a.lso compute val ues tor the various look-up tables such as the ones used by the rendering subsystem .
Action routines perform the actual coding, decoding, and ren dering operations. Q uery routines may be used before setup or between action routines. These provide the programmer with information about the c::tpabiliry
APPLICATION 1
�
APPLICATION N
�
DIGITAL'S MULTIMEDIA
C L I E NT L I B RARY
� t
DIGITAL'S MULTIMEDIA
SERVER (DIG ITAL UNIX.
OPENVMS)
APPLICATION 1
�
. . .
APPLICATION M
�
MICROSOFT'S VIDEO
FOR
WI NDOWS
(WIN DOWS NT)
I
•
I NST ALLABLE
COMPR ESSION
MANAGER D R I V E R
- -
-
-
-
- -
- - - -
-
1
-
- -
- r-
- - -
M ED I A CONTROL
I NT E R FACE D R I V E R
-
- - - - -
SOFTWARE VIDEO
CODING LIBRARY
(SUB)
-
+
I
VIDEO CODECS
I
t t
t
JPEG MPEG H . 2 6 1
. .
•
•
I
VIDEO RENDERERS f f t
DITHER SCALE COLOR
SPACE
CONVERT
I t
COLOR
ADJUST t
SAMPLE
I
1
AUDIO PROCESSORS
RATE
CONVERSION
t
ADPCM
I
f
M P E G - 1
Figure
7
Digital Technic1l founul
Vol . 7
No. 4
1995 6 1
of the codec such as whether or not it can handle a particular input format and provide information about the bit stream being processed. These routines can also be used for gathering statistics. Teardown routines, as the name suggests, are used t()r closi ng the codec and destroying all internal memory (state information) associated with it. For all video codecs, SUB provides convenience functions to construct a table of contents containing the otlsets to the start of frames in the input bit stream. These convenience functions are useful tor short clips: once a ta ble of contents is built, random access and other
VCR functions can be implemented easily. (These routines arc discussed further in the sec tion on sample applications.)
Implementation of Video Codecs
I n this section, we present the program How for multi med ia applications that incorporate the various video codecs. These applications arc built on top of SLIB.
We also discuss specific calls ti·om the l i brary's A
P
I to explain concepts.
Motion JPEG
Motion
) P EG is the de facto name of the compression scheme that uses the
J P EG compres sion algori thm developed tor s till images to code video sequences. The motion
J P EG
(or M-
JPEG) player was the tlrst decompressor we developed. We had recently completed the Sound
& l'vlotion J 300 adapter that could perform
J PEG compression, decompression, and dithering in hardware .''·1" 'Ve now wanted to develop a software decoder that wou ld be able to decode video sequences produced by the J300 and its successor, the FuliVideo Supreme
]PEG adapter, which uses the peripheral component interconnect
(
P
CI)
.
1
"
Only baseline
J PEG compression and decom pression have been implemented in SUB. This is suffi cient for greater than 90 percent of today's existing appl ications. F igure 2a and Figure 3 show the block diagrams for the baseline
J PEG codcc, and Figure 8 shows the flow control for compressing raw video using the video li brary routines. Due to the symmetric structure of the algorithm, the flow diagram for the
J PEG decompressor looks very similar to the one for the
] PEG compressor.
The amount of compression is controlled by the amount of quanti zation in the i ndividual image ti·ames constituting the video sequence. The coefficients for every 8-by-8 block within the image
F(.x,y) are quan tized and dequantized as
F
'1
'·
=
= l
·
F(x , y )
I
F
(x
'· y) r:, (x,y)
X
QTable (x,y) .
( 5 )
In equation ( 5 ) , QTable represents the quantization matrices, also called visibility matrices, associated with the frame
F(x,y).
( Each component constituting
Vol . 7
No. 4 1 995
62 Dig;iral
Technical Jounl<ll
SvOpenCodec
SvOueryCompressor
SvRegisterCallback
SvCompressBegin
SvCioseCodec
Figure
8
Flow Control for M-]
P EG
Compression the ti·ame can have i ts own QTable . ) SUB provides rou tines to download QTabJes to the encoder explic itly; tables provided in the
ISO specification can be used as defaults. The li brary provides a qual ity factor that can scale the base quantiza tion tables, thus pro viding a control knob mechanism tor varying the amount of compression from frame to ri·ame. The quality factor may be dynamica l ly varied between
0 and 10,000, with a value of 10,000 causing no quan tization (all quantization ta ble clements are equal to l
)
, and a value ofO resulting in maximum quantiza tion (all quantization table elements arc equal to 255 ).
For intermediate values:
C'lip
.
(
QTablc (x,y)
=
Visihi/ityTabla(x.y) X( I O' -Quu!Fuuor)X255
I O' Xmin
(
vtsihilityTah/e(x.y)
)
( 6 )
)
.
The Clip() function forces the out-of bounds v<�lues to be either 2 5 5 or l .
At the low end of the quality set ting (small values of the quality factor), the above tonnula produces quantization tables that cause noticeable artifacts.
Although H u ffman tables do nor affect the quality of the video, they do intluence the achievable bit rate for a given video quality. As with q uantization tables,
SUB provides routines tor load ing and using custom
Huftinan tables for compression . Huffman coding works best when the source statistics are known; in
practice, statistically optimized H u ftinan tables are rarely used due to the computational overhead involved in their generation . In the case where these tables are not explicitly provided, the library uses as dctault the baseline tables suggested in the ISO specification. Jn the case of decompression, the tables may be present in the compressed bit stream and can be examined by invok ing appropriate query calls. In the AVI format, Huffman tables are not present in the compressed bit stream, and the default ISO tables are always used.
Query routines t(x determining the supported input and output formats for a particu lar compressor are also provided . For M - J PEG compression, some of the su pported input formats include interleaved 4 : 2 : 2
Y U V, noninterleaved 4:2:2
Y UV, interleaved and non interleaved RG B, 32-bit RGB, and single component
(monochrome ) . The supported output formats include }PEG-compressed YUV and }PEG-compressed single component.
ISO's M PEG-1 Video
Once we had implemented the
M
-
J
P
EG codec, we turned our attention to the MPEG- l decoder. MPEG- 1 is a highly asymmetric algorithm.
The committee developing this standard purposely kept the decompressor simple: it was expected that there would be many cases of compress once and decompress multiple times. In general, the task of com pression is much more complex than that of decom pression . As of this writing, achieving real-rime performance tor MPEG- 1 compression in software is not possible. Thus we concentrated our energies on implementing and opti mizing an M PEG - 1 decom prcssor while leaving M PEG- 1 compression for batch mode. Someday we hope to achieve real-rime com pression all in soti:wan.: with the Alpha processor.
Figure
9
ill usu·ares the high-level scheme of how SUB tits i nto an MPEG player. The MPEG- l system stream is spl i t i nto i ts audio and video substreams, and each is hand led separately by the different components of
ISO
1 1 1 72-2
" the video l ibrary. Synchronization between audio and video is achieved at the application layer by using the presentation time-stamp information embedded in the system stream. A timing controller mod ule within the application can adjust the rate at which video packets are presented to the SUB video decoder and renderer. It can indicate to the decoder whether to skip the decoding of B - and P-frames.
Figure
10 illustrates the tlow control for an M PEG- 1 video player written on top of' SUB. The scheme relics on a cal lback function that is registered with the codec during initial setup, and a
SvAddButfers function, writ ten by the client, which provides the codec with the bit stream data to be processed. The codec is primed by adding multiple buffers, each typically containing a single video packet from the demultiplexed system stream. These buffers are added to the codec's internal bufter queue. After enough data has been provided , the decoder is told to parse the bit stream in its buffer queue until it tinds the next (first) picture. The client applica tion can spec if), which type of pictu re to locate ( I , P, or
B ) by setting a mask bit. After the picture is found and its information returned to the client, the client may choose to either decompress this picture or to skip it by invoking the routine to tind the next picture. This pro vides an e ffective mechanism for rate control and tor
VCR controls such as step forward, fast forward, step back, and fast reverse.
If the client requests that a non-key picture
(
P or B ) be decompressed and the codec docs not have the required reference
(1 or P) pic tures needed to perform this operation, an error is returned . The client can then choose to abort or pro ceed until the codec tinds a picture it can decompress.
During steady state, the codec may periodically invoke the call back function to exchange messages witl1 the client application as it compresses or decompresses the bit stream. Most messages sent by the codec expect some action from the client. For example, one of the messages sent by the codec to the application is
SLIB SYSTEM
PARSER
NETWORK
LEFT
RIGHT
Fig ure
9
SUB as Parr of a Full M PEG Player
Digit�!
Tc(hnictl Journal
Vol.
7 No. 4
1 995
63
SvOpenCodec/
SvOpenRender
SvDecompressQuery
SvRegislerCallback
SvDecompressBegin/
SvRenderBegin
SvGeiDecompressSize
SvSeiBrighlness
SvSeiSaluralion
SvSeiConlrasl
SvAddBuHer
ADD ADDITIONAL DATA
BU FFERS
SvAddBuHers
NO MORE
DATA
SvCioseCodec/
SvCioseRender
ADD ADDITIONAL DATA
BU FFERS
NO MORE
DATA
SvRenderFrame
Figure 1 0
F!ow Control for
MPEG- 1
Video Playback a C B
_
E
N
D
_
B
U
F
FE RS message, which indicates the codec has run out of data and the client needs to either add more data buffers or abort the operation. Another message, CB
_ REL E
A S
E _
B
UFF
E
RS
, ind icates the codec is done processing the bit-stream data in a data bufter, and the bufter is available for client reuse. One possible action for the clie11t is to till this newly available bufter with more data and pass it back to the codcc. fn the other direction, the client may send messages to the codee through a ClientAction field. Table
1 gives some of the messages that can be sent to the codec by the application.
Another use for the callback mechanism is to accom modate client operations that need to be intermixed between video encoding/decoding operations.
For example, the application may want to process audio samples while it is decompressing video. The codec em then be configured such that the ca llback fu nction is
64
Digital Technical Journal
Vol . 7 No. 4
1995
Ta ble
1
List of Cl ient Messages
Message
CLI E NT _ABORT
CLI E NT _CONTI N U E
DROP
CLI ENT _ PRO C ESS
I nterpretation
Abort processing of the fra me
Continue p rocessing the fra me
Do n ot decompress
Sta rt p rocessi n g invoked at a ( near) period ic rate. A CB_PROCESS I NG message is sent to the application bv the codec at reg u lar intervals to give it an opportunity for rate control of video and/or to perform other operations.
Tvpical ly the order in \\'hich coded pictu res arc pre sented to the decoder docs not correspond to the order in which they are to bt displayed. Consider the t()llowing example:
Dispbv Order I l
Decoder [ nput I 1
B2 R3
P4
BS B6
P7 B 8
P 4
B 2 B 3 P 7 B5 B 6
IlO
The order mismatch is an ::�rtibct o f the com algorithm-::\ B
-p i c t p r e ssion ur e
C:Jilnot be decoded until both its past and finure reference ti·amcs h:J\'C been de.coded
·
Similarly a
P-picture cannot be d eco d e d until J ts past rdcrcncc ti·amc has been ckcodcd . To get around th1s problem, S U B defi nes an output multibufkr. The si ze of this mu ltibufrer is approxim:nclv equal to three times the size of example, a single Ll llco mpn.:sscd frame. For t(>r a 4:2:0 subsa mplcd
C!F image, the si z e of t h e multihufti:r \\ 'Ou ld be 352 lw 288 by 1 .5 b\' 3 bvtcs
( the exact size is retu rned by the libr:�rv du ring initi:�l c()(kc setu p ) . After steady st:ltc hJs been rea invocat i on c h e d
, each to the decompress c ::� l l yields the correct next ri·a mc to be displayed as shown in
F ig u re ll . To avoid expensive copy opcr<nions, the m u l ti b u ffer is a l lo c ated :�nd owned by the software above S U B .
ITU-T's Recommendation
H . 261
(a.k.a. p
X 64)
At the li br:�n· level , decompressing an H . 2 6 1 simiL stre:un is ver�·
; r to
M P EG- 1 de c o d ing \\'ith one e x c ep tion: instead of three types of pic t u res
, the H . 2 6 1 rccom membtion defines onl�· t\\'O , key ti·a mcs a n d no n
key h·anlCs ( no bid irectional prcdinion ) . The implication t(>r implementation is that the size of the multibuffer is approximately twice the size of
::1 singlc de co mp r e ss e d h·;unc .
Furthermore, the order in which com p ressed h·amcs :
1
1 c p r e s e nted to the de c o mp re s s or is the same as the order in \\'hich they arc to be displaved.
To satist\• the H . 2 6 1 1-ccomnH:nd::�tion, S U B i m p l e ments a
;
trc::�ming i ntertace f(>r com pr es s i o n and decompression. In this m o d e l , the app l i c ation teeds input bu fti:rs to the codec, \\'hich processes the data 111 the bu fti:rs and returns the processed datJ to the app li ution through a call back rou tine. During dcco
� l1pression, the application l ayer passes input bufters cont::�ining sections of ::�n H . 2 6 1 bit stream . Th
� b1t stream un be divided arbitrarily, or, in the case of live tcleconkrencing, each bufkr can contai n data from a transmission packet. Empty output bu ffe rs are also passed to the codec to fil l \\'ith reconstructed imag_es.
PictuiT ti·;uncs do not have to be a l igned on buffer boundaries. The codec parses the bit stream and,
"'hen enough data is a\·ai lablc, reconstructs ::�n image.
Input b u fkr s arc ti
·e e d by c al l i ng the c11lback rou tine.
When an image is reconstructed, it is placed Ill an out put butter and the buffer is re tur ned to the applica tion th m u gh the cal lback routi ne. The comprcss1on process is s imil ar
, but input bufkrs cont::�in images and output bufkrs contai n bit-stream data.
One advantage to this s t re aming interface is that the ::�pplication layer docs not need to know the svnrax of the H .26 I b it stream. The cod e c is r e sponsi ble t(>r
:1!1 bit-stream parsing. Another advantage is th:Jt the c::�ll b<lck mecha nism f(>r retu rning c o mp l e t e d images or bit-stream bu f k rs a l l o\\'S th e app l i ca tion to do other tasks \\'ith out implementing multithreading.
S U B's architecture and API can easi ly accommo date !SO's pression
M P EG-2 and ITU -T's H.263 video com a lgo r i thm s because of their s i mi b r i t v to tbe
M PEG- 1 :1nd H .2 6 1 algor i thms.
Implementation of Video Rendering
O u r s o ft w a r e implementation of ,·ideo renderi ng cssc ntiallv parallels t he hard\\·are realization detailed c lsc\\'hcn: in this issue -" As \\'ith the hard\\':Jre Imple mentation, the solt\\'are renderer is bst and si mple because the co m pl ica t e d compu tations arc p e r fo r med off line in building hardware and so ft dithering in YUV s the various look- up t:1blcs. I n both w a r e cases, a shortcu t is achieved by p a c e and then c o n verti n g to some snull number of RGB i ndex values in :1 look-u p table
. "'
Although in most cases the m�1pping ,·a lu cs i n the look-up t::�blcs remain tixed for the duration of the r u n , the video li brary provides rou ti nes to dvnamicallv adjust im:1gc brightness, contrast, satu ration, am
�
_ the nu mber of c ol o r s. Image scaling is possible but affects pert(mnance .
When qual itv is i mport�lllt, the so perf(mns scaling before dit the prim::�rv concern, it is done a ncr d i theri ng. ft w a re h ering :1 1Hi when speed 1s
Optimizations
We app r oach ed the p r o blem of optimization fi·om two directions: Plattorm -indcpendcnt opti mi zations, or algorithmic kno\\ enhancements,
\\ ' ere done b,· e x p l oi t i n g
' l edgc of the c om pr essio n a lgo r i th m and the
Figure 1 1
M u lributkring i n
S U B
I
)it-'ir.ll
Tcdmi,,d )numal Vol .
7 No. 4
1 995 65
i n p u t data stream. Plattorm-dcpcndcnr optim izations were done b y examining the sen·iccs <Wailablc ti·on1 the und e r l yin g operating system and by c\·a l u <lting the
Jttri bu res of the system's
As can be seen t pr o ce ssor. i
· o m Ta ble
2 ,
th e DCT' is o n e of the most comput<ltional lv intcnsi,·c components in t h e co mpre ssion p i p el i n e
.
It is a lso com mon to Jll five i nt ern ationa l stand ards. Therd(xe, a s p e cia l e ftort was made in c hoosing and ri optimizing the DCT. Since all ve stand ards c1l l tc>r t he i nverse DC
T
( I DCT) to be p os tpr occssc d with inverse q uantization, sign ificant al g o ri th mi c s a v in g s were o b ta ined b y compu t i ng a scalar mu l ti ple of the DCT and m ergin g the appropri ate scJiing into the q uant i z e r . The DCT implemented in the l ibra r y is
<1 moditied version of the one d i mensional su le d
DCT pr o p os e d by Ar<�ri cr a l . " The two-d i mcnsion<ll DCT is obtained b y p er to r m ing a one-dimensional ncr on the columns a OIH:-dimcnsional ncr on the rows. t (
) l l o w
A totJI ed b of y
80 mu ltiplies and
464
adds arc nee de d for a fu l l y popu lated 8-lw-8 bl ock . I n h ig h lv co mp ress ed ,-ideo, the cocfticicnt mJtrix ro be tr:mstormed is genera l ly sparse because a large number of e lements are "zeroed" out due to hcavv qu a ntiz a ti o n
.
We e x pl oi t this bet to speed up the
DCT comput<ltions. In the d ec odin g process, the H uftin,m d e co d e r compu res and passes to the I DCT a list of rows
,md col umns that arc all zeros.
The I DC:T then simply skips t he s e columns.'" Another o p t im izati o n uses a diff-erent
I DCT, depending on the number of nonzero c o e ffi c i e n ts
.
The o v eral l speedup due to these t ec h niq u e s is dependent on the amount of compression . For l ig ht l y c o m p r e ss e d vid e o
, we observed th:tt t he overhead due to these techniques slowed down the de co m p r e ssor
.
We overcame this dif� ficu l t y bv building i nto
SUB
t h e ada ptive selection of the :t p pr o p r i at e o p timiza tion b<1Sed on c om i nuous sta tistics gathering. Run-time statistics of the number of blocks per t i· ame that are all zeros are maintained, and the number of ti·,Hnes ov er which these statistics arc c \
·a lu
:� tcd is p ro ,
· idc d as a p:trameter t(>r the client applicnions. Statistic gathering is mini m a l
: :1 counter u pda te < llld an occasional com p a r e .
The second c o m po nent of the video decod ers we looked a t was the Huffman decoder. Ana lvsis of the co m p r es s e d data indicated that short-cod e - length s ym b o l s were a l ar g e part of the compressed bit stream. The decoder was written to hand le trcq uently occu rring very short c od e s ( < 4 bits) as s pec ia l cases, thus avoidi ng l oads ti·om memorv. for short codes
( <
8 bits), l oo k
u p tables were used to avoid bit-bv-bit decod ing. T ogeth e r, these two classes of codes
:1ccount f
(
J r well over 90 perce nt ofthe tool col lection of the v aria ble
l ength codes.
A third compute-intensive ope r a tion is raster-to block conversion in p r e p a r a ti o n tor compression. This operation had the potential of sl ow i ng down the com pressor on
Alp ha
bas ed systems on which b y t e and short accesses are done i ndi r e c t l y . 'vVe implemented an assembly language routine that u n would read the c om p rc s sed i n p ut color image and com·crt it to th ree one-dimensional arravs containing 8 - bv-8 blocks i n seq uence. Special care was t a k e n to ke e p memon' n.:fcrcnccs aligned . ReiC\·ant l)\'tcs \\'ere obtained th r oug h shifting and masking operations.
Level shifting \\'aS a lso inc o r p o rat e d withi n the routine to ::�void tou c hi ng the same data again.
Other enhancements included rep l acing m u l tipl ies and d i,·ides with shifts and adds, avoiding i n t eg er to tloating- poillt conversions, and using tlo:1ti ng-point oper ations wherever po ssi bl e . This optimization is particu brly suited to the Alpha architecture, where floatin g po i nt o p e ra t ion s are significantly b st cr than integer o p e ra t i o ns . 'vVe also worke d to r e d u ce memory b a n d w i d t h
.
I l l -p lac e d memory accesses can stall the processor and slow down the computJtions. I nstruc tions generated by the comp i l er were analyzed :t nd someti mes reschedu led to void data haza rd s, to keep the on-chip p i p el i n e ful l , a n d to �woid un necessarv loads and stores. Critical and small loops \\'ere umoJicd to m:1kc better use of floating-point pipelines.
Reordering the c o m p utati o ns to reuse dJtJ a lre a d \
· in registers
,mel caches hel pe d minimize thrashing in the cache and the translation lookasidc bufk r. Mcmorv was accessed through offsets rather thJn pointer
Ta ble 2
Ty pical Contri butions of the Major Com ponents in the Pl ayback of Co m p ressed Video (S I F)
Cod i ng
Bit-stream H uffman
Scheme Parser and
Run-length
Decoder
Inverse
Quantizer
IDCT Motion Vector
Tone Adjust.
Compression, Quantization
Dither, Quantize
Block to
Raster
(INDEO only) and Color Space
Convert
Display
1 2 .4%
1 0. 5 % 35.2%
33.7% 7.4%
M-J P E G 0 . 8 % decode
M P E G - 1
0.9% decode
I N D EO decode
1 .0 %
1 3 .0% 9.7% 1 9 .7% 20.2%
57 . 5 %
3 1 .4 %
36.0%
5 . 1 %
5 . 5 %
66
Dig:i t;ll Tcchnicli )ounul
Vnl.
7 l'o. 4
1995
iiH.:rcments. More lo c.1
l variables tlnn globJI \'Jria bles were used. Wherever possible, fixed values were hard coded i nstead of usi ng variables that would need to be computed . Rckrenccs were m<lde to be 32-bit or
64-bit aligned accesses instead of byte or short.
Consistent wi th one of the design goals, SUB was made thread-sate and fu lly reentrant. The Digital
U N I X, the Open VMS, and Microsofi:'s Wi n d ows NT o perati n g systems all ofkr support tor multithreaded appl ications. Applications such as video pl.lyback can improve their perf(mnance by having separate threads t(Jr readi ng, decom pressing, rendering, and disp l a y i ng. Also, a multithreaded application scales up well on a m u l tiprocessor system . Global multith read ing is possible if the librarv code is reentrant or thread-sate.
When we were tryi ng to multith read the l ibr
J ry inter nals, we found that the overheJd caused by the birth and death ofthreJds, the increase i n memory accesses, and the t i
· a gmen tJtion of the codec pipeline caused operations to slow dm .
vn. For these reasons, rou tines within SLJ
B were kept si ngle-threaded . Other operating-system optimizations such as memon· l ock ing, priority s ched ul ing, nonpreemptio n
,
�md taster ti mers that are generally good for real-time applica tions were experimented with but not i nc l u ded in our present implementation .
Performance o n Digital's Alpha Machines
Measuri ng the perhmnancc of v ideo codccs is gencr
JIIy a dif fi c u l t problem. In addition to the usua l depen· de ncic s such �1s sysn:m load , efficiency of the u n derlying operati n g syste m , and appl ication over head, the speed of the video codees is depen de nt on the con tent of the video seq uence being processed .
RJpid movement and action scenes can delay both c omp r ession and decompression
, while slow motion and high - ti·equencv comcnt i n a video seq u en ce can g cn n al l y result in bstcr decompression . When com pari ng the pcrform<lncc of one codec against another, the analyst must make certain that a l l codccs process the same set of video s e q u ences under si milar oper ating cond i tions. Si nce no seq uences have been
�lcceptcd as stalllbrd, the ana l y st must de c id e which seq uences are most typic:J I . Choosing a seq u ence that tavors the decompn:ssion process and presenting those results is no t uncom mon , but i t c111 lc
J d to t:1lse expectations. Seq uences with similar peak signal-to noise ratio
( l'S
R) may not be good enough, beca u se more ofi:en than not l'S 1R (or equivalently the mean s q uar e error) does not accuratel y measure signal qual ity. With the s e though t s i n mind, \\'l' chose some sequences that we thought were ty pical and used these to measure the pedC.mnance of our sofi:warc e<Klccs.
'vVc do not present comparative results to codccs imp l emented elsewhere since we did not ha,·c access to these codecs and hence cou ld not rest these with the same sequences.
Table and
3 p r e sen ts the characteristics of the three v i deo
" s equenc e s used in our experi ments. Lct
L;(x.y)
L ;(x,y) repr esent the l u m inance component of thc original and the reconstructed ti·anle i: l et
11 and m rep r esent the horizontal and vertical dimensions of a t i
· am c
; and let N be the number of franKs in the video sequence. Then the Compression
Ra tio, the aver<lgc output BitsPerPixcl, :1nd the averag e
PSN R arc calcul:ned as
Compression RJ tio
=
L bits in frame [i ] of origi nal ,·ideo
� I
L bits in frame [i ] of com pressed v ideo i.o:l
( 7)
Avg. B its P er
P i x cl
=
� -
N X n
X m
,
.\'
L
1 bits in frame [i J of
( 8 )
Avg. PS NR
= (9)
2 5 5
2 0l og
111
(
n\nL L
I
[
L; (x,y)-L,(x.y) . .
'
Figure 1 2 sho\\'S the PSN R f(>r individual ti·:unes in
) the video seq uences Jlong with the d istri bution of ti·anK size tor each of th ree test seq uences. Frame d imensions within a sequence always remain constant.
Table 4 provides speci fications of the workstations and PCs used in o u r experiments for generati ng the various perf
(.J
r m a ncc n u m bers. The 2 1 064 chip i s Digit a l
' s first c o m m ercia l ly avai lable AJpha pmccs· sor. [t has a load-store arch itecture
,
0 .
is based on a
7
5 - m icrometer com plementary metal ·oxide semi con d uctor (Civ!OS) techno]Ob'J', contains 1.68 million transistors, has a
7- and 10-stagc i n teger and fl oating point pipel in e , has sepJratc 8
ki l o byte i nstruction :md data caches, and is designed f( >r dual is s u e . The
2 1 064A microprocessor Ius the same architecture as the 2 1064 but is based on
:1
0.5
m i c r o mcte r CMOS technology :md su pports faster clock rates.
W e p ro v i de pcr tC.m
n a n c c num bers f(x the video seq uences char<Kterizcd in T:1blc
3 .
Figure 1 3 provides measured data on CPU usage when compressed video
( from Table 3) is p layed back at 30 fra m es per second
0
11 rhc various test pbrt(ml1S shown in Table 4. We chose "percentage of CPU used" as a measure of pcr ti:>rmancc because we \\'anted to know whether the
CPU could handle any other tasks when it was doing video processing. Fortunately, it tu rn e d out the
Vol.
7 No. 4 l'19S
67
Ta ble
3
Cha racteristics of the Video Sequences Used to Generate the Performa nce N u m bers Shown i n Figure 1 2
Name
Seq uence 1
Com pression
Algorithm
M-JPEG
Sequence 2 M P EG-1
Video
M-JPEG
Seque nce 3 I N D E O
Spatial
Resolution
(width X height)
352 X 240
3 5 2 X 288
352
352
X 240
X 240
Tem poral
Resol ution
(No. of Frames)
200
200
200
200
Avg.
BitsPerPixel
0.32
0. 1 7
0 . 5 6
0. 1 6
Compression
Ratio
50: 1
69 : 1
28: 1
47: 1
Avg. PSNR
(dB)
3 1 .56
3 2 .28
3 1 .56
28.73
40 uj
35
30
25
�
20
;:("
1 5
65
1 0
0..
5
0
SEQUENCE
1
(MOTION JPEG)
40
�
35
30
25
�
20
;:("
1 5
65
1 0
0..
5
0
34
�
33 w r:Q
e.
32
31
30 g:
29
28o 25
0.08
0.07
0.06
�
0.05
�
0.04 w
0
0.03
0.02
0.01
MEAN =
RANGE
26.65
=
4 1 .03
4.97
2.07
50 75 1 00 1 25 1 50 1 75 200
FRAME NUMBER
1 6 1 1 16 21 26 31 36 41 46 51 56 6 1
FRAME SIZE (KBITS)
SEQUENCE
2
(MPEG-1 VIDEO)
0.06
0 05
>f-
0
0.04
�
0.03 w
0.02
0.01
MEAN =
RANGE
=
1 7. 1 1
1 4 .47
PEAK-TO-AVG. RATIO =
25 50 75 1 00 1 25 1 50 1 75 200
FRAME NUMBER
o ��������====------�
1 1 1 21 31 41 51 61 71 81 91 1 0 1
FRAME SIZE (KBITS)
SEQUENCE
3
(INDEO)
5.72
0.25
25 50 75 1 00 1 25 1 50 1 75 200
FRAME NUMBER
0.20
�
0. 1 5
�
0. 1 0
0.05
MEAN =
RANGE
=
1 6.06
6.96
PEAK-TO-AVG RATIO =
9 . 80
0 ���--��������--��--_L
1 8
FRAME SIZE (KBITS)
71 78
Fig ure 1 2
Characteristics ofrhc
Three Tesr Sequences
Dig;irJI T..:dmiul journal
Vol.
7 i'\o. 4 1995
Ta ble 4
Specifications of Systems Used i n Experime ntation
System Name
Al phaStation
600 5/266 workstation
Alpha Station
200 4/266 workstation
D E C 3000/M900 workstation
DEC 3000/M 500 workstation
CPU
A l p h a
2 1 1 64A
A l p h a
2 1 064A
A l ph a
2 1 064A
A l p h a
2 1 064
Bus
P C I
Clock Rate
2 6 6 M Hz
(3.7 ns)
Cache
2 M B
2 M B
PCI
266 M H z
(3 .7 ns)
TURBOcha n n e l 2 7 5 M H z
(3 6 ns)
2 M B
TURBOch a n n e l 1 33 M H z
(7.5 ns)
5 1 2 K B
Memory
64 M B
Operati ng
System
D i g ital U N I X
V3.2
Disk
RZ28B
3 2 M B D i g ital U N IX
V3.0
RZ58
64 M B
32 M B
D i g ital U N IX
V3.2
RZ58
D igital U N I X
V3.0
RZ57 ans\\'er \\'as a resounding "Yes" in the ctse of Alpha processors.
The video plavback r:.nc
\\'<lS measured with sohwJr c
,
· i de o re nd e r i ng cn:�bkd . When hard w
:� r c rende r i n g is available, estimated values for video pl<lyback arc provided .
From Figure 1 3 , it is dear that rodc1v's workstations arc capable ofpl:!ving
SIF video :�t ful l rrcllllC rates with no hardware acceleration.
H i gh
q u al irv M-
]PEG and
M PEG - 1 compressed ,·ideo dips can be pl:wcd a t fu l l speed w i t h 20 percent to 6 0 percent o f the CPU avai l able for other tasks. INDEO decompression is bstcr than
M- JPEG and MPEG due to the absence of DCT processing. ( l N D EO uses a vector quantization method based on pixel difterencing. ) On three out of
M-
JP
E
G
�
3ooo/M5oo
�
250-4/266
�
3ooo/M9oo
�
600-5/266
0
••••••c
�����
20 40
� c:=�
�---L.-�.,....-�----'-
% OF CPU USED
1 40
MPEG-1 VIDEO
�
3000/M500
�
250-4/266 a3
3ooo/M9oo f-
�
600-5/266
I
0
•·
·····-=
•••••c
20 40
==:::J
��!. ����2.._. -
60
---'-
80
�·-�----'-
1
DO
1 20 1 40
% OF CPU
U S E D
I N DEO
�
3000/M500
�
250-4/266
�
3000/M900 in
600-5/266
·····-===:J
•••c::::::J
•·-==::J
��;=��
-
---'-
----'--�
-�
----'-
0 20 40 60 80 1 00 1 20 1 4 0
% OF
C P U U
S
ED
KEY:
•
WITH HARDWARE VIDEO RENDERING
0 WITH SOFTWARE VIDEO RENDERING
Figure 1 3
PcrccntclgL' of C I' U RL·quin:d fcJI' RcJ I - rimc l'bdJack ar
30 r[)S on four Difkn.:IH A l phcl hc�sed S\'S[Cim
Digiral
Tc-:lm i-:al
Joumal
Vol . 7
�o 4 [ <)')�
6'!
the tou r machines rested , two
S
I
F INDEO clips cou ld be p l a y ed back at full speed with CPU capaci t y kft over tor or her tasks.
The data also shows th e advan t age of placing the color conversion a nd r e ndering of the video in the gr aphics hardware (sec Table
S o ftw are rendering accoun rs
2 a nd Figure
13
) . r(:>r one third of th e total playbac k time. Since re ndering is esse nt ial l y a table look-up function, it is a good candidate tor moving i nto hardware . If hardware \'ideo rendering is avai l able, m u ltiple M - ) PEG and M PEG - l clips can be pla y ed back on three of the tour m;Khincs on which the soh w are w a s tested .
Software video com pre s si o n is more time-cons u m ing than deco m pression . All algorithms d i s c ussed i n this paper are as y mmetric in the amou nt o f processing needed tc>r compression and decompression . Even t hough the J PE G a l g or i thm is theoretically s y m metric, the performance of the J PEC de coder is better than that of the encoder. The d i fference in performance is d ue to the sparse nature of the q uantized c oe ffi ci ent matrices , which is exploited by the appropri a te I DC:T o p t imi z a t ions .
For video encoders, we me;lsu red the rare of co m
pression tor both
SIF and quarter
S!F ( QSIF) to rmars.
Since the overhead d ue to l/0 a ftccts the rate at which the compressor works, we present measured rates c o l lected when th e raw video seq uence is read from d isk
:md when it is captured in real time. The capture cards used in our e x p eri ment s were the Sou nd
&
Mot ion
)300
( t( > r systems with the TU RBOchannel bus) and the F u i iVideo
S up re m e ( t(>r PCI - b<lsed s y s te m s ) .
The compressed bit streams \Vere st or ed as
AVI ti le s on local disks. The sequences used in this experi ment were the same ones used tcx obtaining measurement f(>r the various decompressors; th e ir ou tpu t characteristics arc gi ven in Table bers
3. T:1blc 5 pro\'ides pcrt(>rmance num rc>r the M-) PEG ;lnd an u noptimized I N DEO compressor. For M-) PEG, rat e s t<x both monochrome and color video seq uences
:11-c provided .
The data i n Table
5
indicates tlut the MJ PEG com p r ess io n ou rpertorms I N ]) EO (al though one h:1s to keep i n mind t h a t
INDEO w as not optimized ) . This di ffe r ence occu rs because MJ
PEG co mpressi on, u n l i ke ! N D EO, does 110t rely on intedi·ame pred iction or motion estimation J{>r compression. Fu rt hermore, w h en ra w video is co mp ressed ti·om disk, the encoder p e r t ( mn s be tter rhatl when it is captu red and com pressed i n real rime . This can be e xp l ained on the basis of t h e overhead res u l ting trom context s wi tching in the operating system and the sched u l i ng of sequential capture oper a t ion by the applications . Rea l -ri me cap ture and c ompression of i mage sizes larger than
QS!f sti ll require hardware �1ssisrancc. It shou ld be noted that in Table 5 , the maximum compression rate t(x re a l
-tim e capture a nd compression docs not e x c eed
30 ti·anKs per second , which is t he l imit of the capture hardware . Since there arc no such reads, co l i m i tatio ns for d isk m pression r:1tes of greater than
30 fi·ames per second tor
QSir sequences arc recorded .
With the newer Alpha chip we expect to see improved pe rf or m ance .
A b c ro r we ne g l ected in our calcu lations was prdi l tering. Some capture boards arc c1pable of c ap tu ring o nl y in C C I
R
60 1 form at and do not include d eci m a tion til rers :1s p<l rt of their hard ware. I n such cases, the software hJs to fi lter eac h fr ame down ro C!F or
QCIF, w h i c h adds substanti<lllv to the ove ra l l compression ti m e . For applications that do not re q u ire real- ti me co m p ression , soft w are d igita l -video compressio n mav be a viable sol u tion since video can be captured 011 E1st disk arrays and co mpressed later.
Table 5
Typical N u m be r of Frames Compressed per Second
M-JPEG (Color)
System Com press
(fps)
SIF QSIF
Capture and
Com press (fps)
SIF QSlF
M-JPEG (Monochrome)
Compress Capture and
(fps) Com press (fps)
SIF QSIF SIF
QSIF
Al phaStation
600 5/266 workstation
AlphaStation
200 4/266 workstation
DEC 3000/M900 workstation
DEC 3000/M500 workstat ion
2 1 . 0 79.4
1 0 .8
1 3 . 2
6.7
45. 1
56.6
26.6
20.0
7.9
7.3
30.0
1 2 . 0 30.0
28.0
8. 1
3 2 . 8
1 5 .8
2 1 .9
1 0.4
1 30
72.9
87.8
40.4
29.0
20.0
1 4.0
7 . 4
30.0
30.0
2 9.0
8 . 2
I N D EO (Color)
Com press
Capture and
(fps)
Compress (fps)
SIF
QSIF SIF QSIF
8 . 7
5.6
6.0
3 5 .4
22.0
2 5 .4
5 . 8
4.2
4 . 5
2 . 8 1 1 .8
2 . 2
2 3 . 0
1 3 .0
7 . 6
8 . 7
70 Digiral Tcchnic.ll journ;ll
Vol . 7 No. 4 1 995
Sample Applications
We implemented sever a l app l i ca t io n s to test o ur ar c h i
tec t ure ( co d e c s and renderer) and to cr ea t e a test bed f()r perfim11ance mosuremcnts. T hes e programs a lso served as s a m ple code f()r software developers incorpo r at i n g
SUB i nto other multimedia software layers.
The Video Odyssey Screen Saver
The Video Odyssey screen saver uses software video decompression and 24-bit YCbCr to 8 - bit pseudo color rendering to deliver video images to the screen
, a control panel, shown in Figure 14 .
The user
Gln select ri·om several methods of display ing the decompressed video or let the computer cycle through a l l methods. The floaters mode, shown in
Figu re 1 5 , Hoats one to r()Ur copies of the video around the screen with the n umber of Hoaring win dows controlled by a slider in the control panel . The snapshot mode floats one window of the video around the screen, b u t cvcrv second takes a snapshot of a frame and pastes it to the background behind the floating window.
All settings in the control panel arc saved in a con figuration fil e in the user's home d irectorv. The user selects a video fi le with the Ti le button . In
�
he c ur rent i mplementation, a1w AVI tile containing Motion } PEG or rJ\\' YUV video is acceptable. The user can set the rime interval for the screen saver to tJke over. Controls f(H setting brighrness, contrast, and saturation are also tion or with X2 s u l in g
. at normal resolu
Scaling is i ntegrated with
)!ideo ()dyssey
On
Sleep in
1 1 o
! min.
1
Floaters
--'-'-
3
�
4
1
Float Speed
10
Screen saver:
Combo
lltm\@4
Snap Shot
1
·�
1
Video Rate
10
-'--'
0
Brightness
10
Striper
Pileup
6.0
�
0
Contrast
10
!
Video File l clock.avi
--'-'-
0
Saturation
10
_j
Large Window
I
•
Password protect
!
I
Start
I I
Preview
I
[!]
Figure
1 4
Video Odyssev Control Panel
Figure
1 5
Video Odyssev
Screen s�ver i n t:lo�rcrs Mode the color conversion and d ithering t(>r opti mization. A pause kJture allows the user to leave his or her scree n in a locked state ll'ith an acri,·e s cr een saver. The screen is un locked only if the correct password is provided .
The Software Video Player
The softwJre video player is an application ft)r viewi ng video th:n is similar to a VCR
.
. Like Vid e o Odvssev, the software video plaver exercises the decompt-c�sio;l and r en d e ri n g po r tio ns of
SUB.
Unli ke Video Odyssev, the software video plaver allows random access t
;
>
�11�, po rt ion of the video and permits single-step, revers
�
, and bsr- forward fu nctions. figure 1 6 shows the dis play window of the software video player.
F i le
Options
Figure
1 6
The
Software Video Plaver Dispbv Window
Vol . 7 :--Jo 4 1 99 5
7 1
72
The user mm·cs through the tile ll'ith a scro l l bar and a set o f VCR-Iikc buttons. The b utton on the far
Jefr of the d isplay window a l l ows the video to be dis played a t normal size or at a magnification of
X
2 . The br- right button :t I l ows adjustment of brigh mess, con tr:tst, satur Jtion , and n u mber of displayed colors . The qual ity of the ditheri ng a lgorithm used i n rendering is such that v:�lucs as low as 25 colors lead to acccptJblc image qual itY. Allml'<lble ti le formats t(Jr the soft\I'Jrc video pi aver <llT M- J PEG (AVI t(xmat and the J PEG fi l e interchange tcmnat or J FIF), MPEG- 1 ( both video and system streams), and raw YUV.
Random access i n to the tile is done i n one of t\vo ways, depending on the tile tcm11at. For t(mnats that conta i n an index of the ti·ame positions i n the ti le ( l i ke
AVI tiles ) , the i ndex is simplv used to seck the desired ti·:�me . For tormJts thJt do not contain an i ndex, such as MPEG - 1 and J
H r , the software video plavcr csti mates the l oc1rion of a ti·an1C based on the toul length of the video clip and a running average of tramc size .
This technique is adequate tor most video clips and has the advant age of avoiding the ti me needed to t!rst build an index by sCJnning through the file.
I n terti"ame compression schemes like M P EG - 1 and
I ND EO pose special problems when trving to access a random ti-anlC i n a
,·ideo c l i p . M PEG- 1 's B- and
P - ti·ames arc dcpcndcm on preceding frames and can not be decompressed alone. One technique ti.>r h a n d ling random access i mo flies with non-kcv tr;�mes and no frame index is to use the tile position spec i fied by the user ( with a scrol l bar or bv other means) as a starting point and then to search the bit stream t(x the next key ti-amc ( a n I-ti·amc in M PEG- 1 ) . At that point, d ispLw can proceed normal ly. Rn'ersc plav is a lso a problem ll'ith these f(mnats. The sof1:ware 1·ideo player deals with rnTrse by displaving only the kcv ti·ames.
It could d isplay a l l tramcs in reverse by prcdccom pressing all frames i n a group and then displaying them i n reverse order, but this would req u i re l a rge ;unounts of memory and would pose problems with processing delays . Rate control fu nctions, i nclu d i ng bst- t(Jnvard and fast-reverse fu nctions, can be done bv sclcctivcl v th roll'ing o u t non-kcv ti·;�mcs a n d processing kcv or
I- rr:uncs on h-.
Other Applications
Several other applications using d i tkrcnt components of S U B were a lso writte n . Some of these arc
( 1 ) E ncode-a video encoding application that uses
S U B 's compression component to compress Llll' video to M -J PEC ti.mnat, ( 2 ) Rcndit-a viewer tor true color i m ages th�lt uses S U B's rendering compo nent to scale, tone-Jdj ust, dither, quant ize, color space convert, and disp l �1y 24- bit RG B or 1 6-bit Y UV images on ri·amc bufftrs with l i m ited p l anes, and
( 3 ) rou ti nes r(Jr viewing compressed on-line video documcntJtion tlut was incorporated i n to Digital 's vidcocon ti.:rcncing product.
Related Work
\Nh i lc considcrJblc cft(Jrt has been dc\'Otcd to opti m izing video decoders, little has been done t(x video encoders. E ncoding is gen eral ll' computationallv more complex and rime-consuming than decodi ng. As a result, obtaining real - ti me pcrt()rmJ ncc ti·om encoders has not been feasible. Another ration a l i zation t(Jr interest in decoders has been that many applications requ i re video p l ;�yback and only a kw arc based on video encod ing. As a result, "code once, play many times" has been the dominant p h i l osoplw. I n most papers, resc;�rchers h a1'e focused on techniques t(Jr opti mizing the various codecs; l'cr\' l i ttle has been p ublished on provid i ng a u n itcm11 architecture and Jn intu i tive API f()r the video codccs.
In this section, we present resu l ts from other papers p u bl ished on software video codccs. Of the three i nternational standards, M PEC- 1 has J ttracted the most attention, and our present�nion is bi<lsed sJ ightlv toward this stJndard . vVe concentrate on work that implcmcllts at least one of the three recogn ized i nter national st;mcbrds.
The J I'EC software was nude popul;� r bv the
I ndependent Soti:warc J PEG Group tcm11ed by Tom
L1ne."" He and his colleagues implemented and made available tree software that could pcrt(m11 b;�seline J PEC compression and decompression. Considerable atten tion w:�s given to software mod u larity :md ponabilit1·.
The m;�in objccti1·c of this codcc ll'<lS sri l l -i m;�gc com pression although its modified version h�1s been used t(x decompression of motion J PEG seq ucnccs as wel l .
The
M I'EC software \·ideo decoder \\"JS made popu lar by the m ultimedia researc h group Jt the U niversitl• ofCali torni<l, Berkeley. The availability of this tl-ee soti: ware sparked the i nterest of many who now had the opportunity to play with and experi ment with com pressed video. PJtel et al. descri be the implementation of this softll"arc MPEG decoder -" "The hxus in their paper i s on an M P EG - 1 video plJ1·cr that wou ld be portable and bst. The authors dcsuibe 1 arious optimizations, i ncl ud ing in -I inc proccd urcs, custom coding frequent bit-t\viddling operations, and render i ng in the YUV space with color conversion through look-up tJblcs. They observed that the key bottleneck toward real-ti m e performance was not the compu t<ltion im·ohcd but the mcmon· bandwidth. Thev a lso concluded that data structure organ i zation and bit-lcl·cl manipulations were critical t(Jr good pcrt(x mancc. The au thors propose J nm·cl metric t()r com paring the pcrteJrmance of the decoder on svstems marketed by d i fferent systems l'cndors. Their metric, tbc percentage of requ ired bit r:ltc per second per
Vol .
7
No. 4 l 'l95
thousand dol l ars ( PBS D ) , takes i n to <Kcount the price of the system on which the decoder is being evaluated .
Bheda and Sri nivasan describe the implementa tion of an M P EG- l decoder that is portable across platrixms bccwsc the software is written cntirelv i n a high-le\·cl Lmguagc . " T h e paper describes t h e \'ari ous optimizations done to improve the decoder's speed and provides pertormance n u m bers in terms of nu mber of ri-ames d isplayed per second . The authors compare the speed of their decoder on various piJtr(mm, including Digital's rirst Alpha- based PC: run ning Microsoft's Windows N T svstem . They conclude that their decoder perrC.m11ed best on the Alpha svstem .
I t was able to decompress, dither, and d isplay a 320pixel by 240-line video sequence at a rate of 1 2 . 5 frames per second. A very brief description of the API sup ported by the decoder is also provided . The A
PI is able to support operations such as random access, rast [()r w:mi , and bst reverse. Optional skipping of B - frames is possible tor rate control. The authors conclude that the size ofthe cache and the perr(xmancc of the display sub system are critical r(lr real -time pert(mnance.
B haskaran <1lld Konstantinides describe a real time
M
P
E
G -
1 software decoder that can p lay both audio and video data on a HewlettPackard PA- RJSC processor-based \\'orkstation 2 ' The paper provides step-by-step details on how opti mization was carried out at both the algorithmic and the architectu ra l levels. The b<lsic processor was enhanced by including in the i nstruction set several m u l timedia instructions capable of ped(lrming para l l e l arithmetic operations that are critic:d in video codecs. The display subsvstem is able to hand l e color conversion of YC:bCr data and up-sampling of inL1ge d ata. The pedC.m11ance of tht.: decoder is comp<lrt.:d to software dt.:coders running
011 different platrcmm rrom d i fferent manuf:1crurers. The comparison is not tru lv rair because tht.: authors com pare their ckcodcr, which has h <lrdwart.: assist:t11cc
:tvailable to it ( i .e . , <lll enhanced graphic su bsystem and new processor instructions), to orhu- decoders that arc tru ly softwart.: based . Furthermort.:, sin ct.: a l l the codccs were not running on tht.: samt.: machint.: u nder similar operating cond itions a nd sinct.: the scqut.:nce tested on tht.:ir decockr is not the same as tlw
0 1 11.: used by tht.: othns, the comparison is not tru lv accurate. The papn dot.:s not provide any i n formation on the program ming i n terbcc, the control tlow, and the overa l l soft
W<UT arch i rectu rc
There a rt.: n u rnu-ous otl1t.:r d t.:scriptions of tht.:
M
PEG- 1 soft\\' art.: codecs. Eckart dcscribt.:s a software lv! PEG video pL l
\ n that is cap�1 blc of ckcoding both
�Hi d io and \·idt.:o in rt.:al time on a
PC with a
90-mcgJ hntz Penti u m proccssor.2' Software available tix this decoder is frt.:dy over the lntcrnt.:t. Gong and Row�.: c"kscribe a par.1lkl implementation of the M P EG - 1 t.:ncoder that runs o n a network ofworkstations 2' The pntormance improvements of greater than 650 percent are reported when the encoding process is performed on
9 nt.:tworked H
P 9000/720 systems as compared to a single svstem. vVu et a l . describe the impkmt.:ntation and per formance of a software-on ly H . 2 6 l video codec on the Power
PC 60 l red u ced i nstruction set computer
( RISC ) proct.:ssor."' This paper is i nteresting i n that it deals with optimizing both the t.:ncoder and the decoder to facilitate rea l-time, fu l l -d uplex network connections. The codec plugs u ndn the QuickTimt.: architectu re ckvcloped by Appl e Computer, I n c . Jnd can be invoked by applications that have programmt.:d to the
Quick.Timc interface. The highest d isplay rate is slightly u nd n
1 8 f ra m es per second fix a
QS!f vicko s�.:quence coded at 64 kilobits per st.:cond with d isk access. With rt.:al - time \'ideo capture i ncluded, the frame rate rt.:d uces to between 5 and 10 frames per second . The paper provides an i n tnt.:sting insight by giving a brea kdown of the amou n t of time spent i n t.:ach stage of coding and decoding on a complex instruction st.:t computer
( CISC) vnsus a
IUSC system .
Although tht.: paper does a good j o b of descri bing the optimizations, verv l i ttle is mentionni about the soft ware architecturt.:, the programming i ntnface, and tht.: control flow.
We end this section by recommending some sourct.:s tor obtaining add itional i n formation on the start.: of the art i n sothvare-only video in particular and i n m u ltimedi<l i n genera l . First, the Society of Photo
Optical I nstru mentation Engi nens ( S PI E ) and the
Association of Computing Machinery
( ACM ) sponsor annual multimt.:dia conferences. The proceed ings trom t hese conferences provid e a comprclwnsive record of the advanct.:s made on a year-to-year basis. I n addition, both the I nstitutt.: of Electrical and E lectronics
Engineers
( I E E E ) and A C
M regularly publ ish issut.:s devoted to multimedia. These spt.:cial issues contain review papers with sufficient technical details.
" ,
Final ly, an excel lent book on tht.: subject of video com pression is the recently published Dip)! a/ Piclures ( sec ond edition) by A run Netraval i and Barry Haske I ti·om
Plenum Press.
Conclusions
We have shown how popular video comprt.:ssion schemes art.: composed of an i n tnconnection of d is tinct functional blocks put togt.:thcr to meet specified design objecti\·cs. The objectives art.: almost alwavs set bv the target applications. We have demonstrated that the video rcmkring su bsystem is an i mportant compo nent of a compkte p layback sol u tion and prcst.:ntt.:d a n ovel a l gorithm ti:x mapping out-of range colors.
DigirJI Tcchnil"al journ,1 l
Vol . 7 No. 4
I 995
73
We described the design of our software architectu re for video compression, decompression , and playback.
This architecture has been successfu l ly implemented over m ultiple platforms, includi ng the Digital
U N I X, the Open VMS, and Microsoft's Windows
N"f operat ing systems. Performance results corroborate our claim that current processors can adeq uately handle playback of compressed video in real time with little or no hardware assistance. Video compression, on the other hand , sti l l requires some hardware assistance ri:x real-time performance. vVe believe the widespread use of video on the desktop is possible if high-quality video can be delivered economically. By providing software-only video playback, we have taken a step in this direction.
References
I .
A. Aka nsu and R. H addad , "Signal D e co m po s it i o n
Tech n i q ues: Transf(Jrms, Subba nds, a nd Wavel ets,"
O p t c on
'92,
Short Course 28 ( Novem bcr 1 99 2 ) .
2 . E . Feig a n d S. W i n o g r a d , " F as t Al g o r i t hms for Discrete
Cosine T r a n s f o r m
,"
!f'F:r: T ra n s actions
011
S
(� nal
Proce s sing vol. 40, no. 9 ( 1 992 ): 2 1 74-2 1 93 .
3 .
S . Lloyd , " Le<lSt Squares Quantization in
PC:lv!,"
IF;'F;E
7h m sactions on I nfor m ation nx'Ot!'. vol.
28 ( 1 982 ) :
1 29-1 37.
4 . ] . Max, " Quantizing for Mini m u m Distortion ," !Nh'
T ra n s actions on Information Theory, vol . 6, no. 1
( 1 960 ) : 7-1 2 .
5 . N . Nasrabadi a n d R . Ki ng,
"
I ma g;e
Cod i n g using
Vector Quantization : A Rev i e w,
" fh'E't' Tra n s actions
Communications. vol. 36, no. 8 ( 1 988 ) : 957-97 1 .
6 . D. H u fti11 a n , "A Method for the Construction of Min i mu m R e d u nda n cy Codes," Proceeding\· of the
!NE, vol . 40 ( 1 95 2 ) : 1 098-1 1 0 1 .
7 . H . Harashima,
K .
Ai zawa, and T. Saito, "Model based Analysis-Synthesis Cod i n g of Video T e l e p h o ne
I m ag e s
-
C o n c e pt i o n and B a s i c Study of I n t e l l i g e nt
I m a g e C o d i n g , " Transactions of
Jh'JCt·. vol .
E72, no. 5 ( 1 98 1 ) : 452-458.
8. I nfor m atio n 'Fechnologv -D1gital Compression and Codin,r.; of Continuo u s-tone Still lnwges.
Part
1: R e q u i re n z ents and Gu idelines, ISO/ ! EC !S
109 1 8- 1 : 1 994 ( G e n ev a : International Org<mizarion tor Standard ization/1 nternational Electrorec h nical
Commission , 1 994 )
.
9 . K . Corrd l and R. U l i c h n e y, "The J300 ram i l y of Video a n d Audio Adapters: Architecture and Hard ware D esi gn ,
"
Dl/!,ita! Techn ical Journal no. 4 ( 1 99 5 , this issue): 20-33.
. vol . 7,
10. P. Bahl, "The ) 300 F a m i ly of
Video and Audio
Adapters: Software Archi tecture," DipJtal Tech n ical jou r n a l
, vol . 7, no. 4 ( 1 99 5 , this issue ) : 34-5 1 .
1 1 . Video C o dec for Audiol 'isual Serl 'ices at khits/s. ITU-T p
X 64
Recomme ndarion H . 26 1 , CDM XV- R
37- E
(
G e ne v a : I nternational Te le g t aph and Tele ph o ne
Cons u l rative Com mittee, 1 990 ) .
1 2 . Co d ing of iv!ouing Pictures and Associated A udio for D(r.;itctl Stora,r.;e Jllledia at [ijJ to about
1 . 5
;J;Jbits/s.
!SO/! EC Standat-cl 1 1 1 72 - 2 : 1 993 ( Genn a: I nterna tional Org a n i z ati on tor
Stand,l rdi/_,nion/1 nternational
Elecrrotec l111ical C o m m is si o n , 1 993 ) .
1 3 . Gerzenc Coding ofJIIouing Pi c t u r es o n d Associated
Audio, Recomrnendation f!. 2(J2. ISO/I EC C D
1 38 1 8 - 2 : 1 994 ( Ge n ev a : T ntcrnation<l l Organization f(JtS ta n dardi z ati o n /
! n ternarional E l e c r rote c h nica1
Commission, 1 994 ) .
1 4 . Special Issue o n Ad v an c es i n I t n a ge a n d Video Com pression, Pro cee dings of the ff:'FF ( Februarv
1 99 5 ) .
1 5 . R . U l i chnev, "Video Rendering," !Ji,r.;ito! Tecbrziwl
Jou mal. vol . 5, no. 2 ( Spr i n g 1 99 3 ) : 9-1 8 .
1 6 .
L . Sei ler and R .
U l ich ney, " l nn:gr<lting Video Render ing i n to Graphics Accelerator Ch ips," Digital Tech ni cal jou nwl. vol . 7, no. 4 ( 1 99 5 , this issue ) : 76-88 .
1 7 .
R . U l i c h ney, "Method and A a D igital Color Image from a p p a r atu rim s for
Mappi ng
Color Spa ce to a
Second Colm Space," U .S. Patent 5 , 2 3 3 ,684 ( 1 99 3 ) .
1 8 . Y . A r a t · i , T. Agui, and t'vl . N ab j i m< l
Scheme
,
" A bst DCT-SQ f(lr I m age s , "
Jh'h't·
J'i-a nsactions
Jf:'JCF.
E - 7 1
( 1 98 8 ) : 1 095-1097.
1 9 . K. Froi tzheim a n d K.
Approach to
Wol f� " Kno w ledge
b as e d
JPEG
A c c eler at i o n ," D i g i t a l Video Com pression : Algorithms a nd Te c h nol o g ie s 1 9 9 5 , Pm ceedings oft he SPTF:; vol . 24 1 9 ( 1 99 5 ) : 2-1 3 .
2 0 . T. L111c,
")PEG
S o ft wa re," I ndependent ) PEG Grou p, u n p u b lish ed paper available on the I nternet.
2 1 . K. Patel, B. S m i th, and L R o w e
, " Per formance of a
Sotrware M PEG Video D e co der
,"
Proceedings ol
A C!vl M ulti med ia
'93,
Anaheim, Ca l i f ( 1 99 3 ) : 75-8 2 .
2 2 . H . R heda a n d P. Srini vasa n , " A H igh-performance
Cross-platt<mn M l'EG Decoder," Digir � 1 l Video Com pressio n : Algmithms and Technologies 1 99 5 , Pro ceedings o lt he
SPIF. v o l
. 2 1 8 7 ( 1 994 ): 24 1 -248.
23.
K.
B ha s k a t -
,111 <llld K. K o ns t
< l n t i nid cs, " Rea l-Time
M PEG - 1 Software D e c o d i ng on H P Workstations,"
Digit<ll Video Compression: Algmith llls and Tec h n o l o g i e s 1 99 5 , Proceedings oj' thc
Sl'l!:', vo l . 24 1 9
( 1 99 5 ) : 466-473.
24. S. Ecbrt,
Pbyb,Jck
" H igh Performance Softw�li'C M P EG Video f(lr !'C :s," D i g i t a l V ideo C:om pt-cssion: Algo rithms :11 1 d T ec h n ol o g i e s 1 99 5 , Proceedings of the
SP!l:. vol . 24 1 9 ( 1 99 5 ) : 446-454.
25.
K.
Gong and
L Rowe,
" P a t
'
< l lkl
,v!
P J-:
G
Encoding," P m cee d in
S)'mposium, Sacramento, C:� l i f . ( 1 994 ) .
-
1 Video
, r.; s oft he
!C)CJ4
Pictu re Codinr;
74 Digital Technical journal
Vol . 7 No. 4
1 995
26. H . Wu ,
<lnd
K.
W;1ng, ) . Normi l e , D . l
' o nc
-: kon
,
K .
Chu,
K. S u n g , " l'ertimnann: of
Real-rime Software on ly H . 2 6 1 C :od ec on rhe Power Macintosh , " D ig ital
Vidn
>
1 99 5 ,
C o mp r e s sion : A lg o ri t h ms and Tec flroceedit(�S of' !he
.'1/'//:'. vol . h n o lo gie s
24 1 9 ( 1 995 ) :
492-49 8 .
2 7. Sp ec i a l Issue on D i g;i r a l
M u ltimedia Svsrems, Com
/llllll ic(l/irms oj'lhe A C:.\'1. vol . 34, n o
.
I ( Apri l 1 99 1 ).
Biographies
Robert A . Ulichney
R.obert U l i c h n e v received '' P h . D . purer science and a B.S. in ti·om the M.assa
c hu se
I nstitute ofTech nologv in electrical eng:im
· cri n g
�nd com p h y s ics and computer science t rs ti·om the University of Davton, Oh i o
.
He joine d
D i g it
; ll in 1 978. He is currentlv
Digital's a s eni or consulting engineer \\'ith
C ambrid g e Research Laboratory, \
\
' h e r e he leads rhe Video Jnd Image Proc eral patents tix contrib
�1re e ssin g pr oj ec t
.
He h :1s ti led sev u t i o n s to D i g it;ll p r odu c ts in the a s of h ard w are and sofhvare-only motion video, gr;1phics
<.:on trollers, and hard copy. Bob is the author of
Dt�ilal
Ha(Jioninp, and serves as a rcte r ec tor '' n u mber oftedmical societies, including I EEE, of which he is J senior m e m b e r.
Paramvir Bah!
ParJ nwir !hhl n:ceived
B .S.E.E. and M .S . E . E . degrees i n
1 987 ;md 1 988 from t h e State U n i v ersir y of Ne w York a r
R u thlo. Si nce joi ning D i git a l i n 1 98 8 , he h;ls contri b u t ed ro s e v e r a l seminal m u l rimediJ products inv ol vin g both hardware ;�nd sol-tw;H·e t(>r digiL1I video. Rc c c nr l y
, he led the d c ve lop n
1 e nr ofsofrw;lre -only video c o m p ressi o n and v i d e o r ender i n g
<1 l gori rhm s
. f\ f>ri ncip�d e n gi ne e r in the
Svsten1s B usi ness Unit,
P aran w ir n:u.:ived Digital's Doctoral
En g i n e e
1· i n a r rhc Uni,·ersirv focused on g t r:e llow s h i p Aw;mi and is c o m p l et in g his P h . D . o f M;lssach mcrrs.
There, h i s research has e ch
n i q ue s fi>r robust ,·ideo c omm u n ic at i o ns over mobile radio networks.
H e is rhe author and coa u tho r o fs e v cr;ll s c icnri t ic p ublic1ri on s and a pend in g patent. H e is an acri,·e member o f t h e I EEE a n d ACM, serving o n pro gra m com mittees of rechnic;ll co n tl: r cn c e s and ;lS a rderee ti>r their journals. ParJ m\·ir is a m e m ber o f
T l ll J3cra
Pi and a pas r pr esi d ent of
Era
K
;lp pa
N u .
Paul S. Gauthier
1\ud
Gau thier is
Sofrworks, a so the t r w p r e siden t ;llld tin111der of
I n u g e a re dcvelopmenr c o m p a n y in Wesrtord ,
N!Jssachuserrs, sp e cial i zi n g i n in u g e
; l n d v ideo proc es s i n g
.
He rcc.
: e i v ed J Ph.
I). in p h y s i c s ti·om Syr;Kuse U n i v ersity in 1 975 ;lnd
Ius w o rke d with ;1 V;lricry of c o m p ani es on medical illl;lging, ma c h i n e vi s i o n
, prcpress, and d ig i ta l video. I n ! 99 ! l'�ul col b borarc.:d with the Associated Press on d e v e lo p i n g ;In ekcr ro n i c d;lrkroom.
D u r in g the G u l f vV;Jr, ncwspJpc rs used h i s sohw;m: to l
> tJken r oc es s photogr;� phs of t h e nighttime �\track on B;1ghd;1d by th e United
Stares. Working w it h Digic1 l , he h;ls c o mri b ured to adding video to the Alph;l deskto p .
Digiral
Tc..:hnicd )ourn.ll
Vol .
7
No. 4
I
995
7 5
Integrating Video
Rendering into Graphics
Accelerator Chips
I
Larry
D.
Seiler
Robert
A.
Ulichney
The fusion of multimedia and traditional com puter graphics has long been predicted but has been slow to happen. The delay is due to many factors, includ ing their d ramatically different data type and bandwidth requirements. Digital has designed a pair of related graphics accel erator chips that integrate video rendering primitives with two-dimensional and three dimensional synthetic graphics primitives. The chips perform one-dimensional filtering and scaling on either YUV or RGB source data. One implementation dithers YUV source data down to 256 colors. The other converts YUV to 24-bit
RGB, which is then optionally dithered. Both chips leave image decompression to the CPU.
The result is sign ificantly faster frame rates at higher video quality, especially for display ing enlarged i mages. The paper compares the implementation cost of various design alter natives and presents performance comparisons with software image rendering.
Fo
r years, r
h
c
co
mpu ter i nd us
tr
�' c
on
fid
e
n tly predicted th
a
t u biq
u
i
to
us, i ntcg
r;n
cd
m u l timedia compming
\\':lS just
around tile corner.
A lte r a n u m be r
of de!a,·s, this c
om pu
t
i
n
g env
i
ron ment is tln
a
lly a
r
eal
i
t y
.
It is no,,· p
o
s
s
ib
l
e to buv
p
e
r
son
a
l co
mp
uters (
P C: s ) and \\'mk stations that com bi n
e
:� ud io
p
rocessi
n
g
with
r e
a
l-ti me d is
p
l
a
v and manipu lation of v
i
de
o or other
s
a
m
p l ed data, though
usu�1lh-
with s
igni
ti
ca
n
t limi
t
a
t
i
o
ns
.
For the most part, the in d us try has fol lowed one of two paths to ac h i e v e real-time video p rocessi n g. On one path, video tc�nurcs �H·c implemented almost entire l y in soltwarc . When
�1pplicd to the d isplay of moving images, this
:1 p p r o a c h r y pic 1 l l v results in a co m b i nation of low n:solution, slow upd ate times, and s m all im�1gcs.
The alternative has been to ac hic ,
· e good , -id e o i mage displa v pe r for ma nce lw adding a se p ar a t e \·ideo hard\ \· a r e option to <l
PC. I m a g e d i spla v i s i nteg;rated in the box �111d on the screen but is d istinct rl·om the ha r dwa r e that i mp le m e nts t r
:� d i tiona l sv n t hc t ic grap h ics. Frequ e mly , this design f o r c es pertonnance com p r o mises, f()r ex a m pl e, by l i m i t ing t h e n um ber of vide o imag e s that can :1ppc:1r at t h e same time or by l i m i t i ng the interaction of images with the w i n d ow sys t em.
Reccntlv, two kev c n
J bl i n g te ch nolo g i es h:�vc com bined ro m:�kc
:1 better so l u tio n p o ss i b l e . Advances in silicon tech nologY eo�1blc l o\\ '
· cost graphics controller chips to be d e signed \\'ith a signi fica nt num ber of gates ded ti ic :Jted to su pport i n g multimedia features.
I n addi o n, the pc r iphcr�1 l component i nterconnect ( PCI ) bus pro\'ides high·b�lnd\\'idth, p e er- to- p e er com m u n ica tion between the CPU, t he main memory, an d option cards. Peak band\\'idth on the standard 32 -bit
PC! bus is 133 meg<lbytcs per second ( M B/s), �md highcr pedorm:mcc versions �1rc also available. Good PC:I implementations can transfer sequ e ntial data ar
80 to
100 MB/s. Eq ually important, the PCJ bus allows m u l timedia solutions to b e incrementa l l v bui l t u p fl·om a s o fuvJ r c-only implcmclltation t hrough \'::Jrious lC\'cls of hardw::Jt'C support. The I'Cf .\1ultimcdia
Oes(�ll
Guide describes this incremental approach and also prm
· i des standards f(Jr LnctK\. and ,·ideo data f(J rmats '
This p�1per describes a Digital engi n eer i ng p r oje c t whose goal \\'�lS to combine \ ideo rendering tC a turcs and tradition�! [ S\'Ilthctic g r a ph i cs into a u n i fied grJph ic s chip, viel d i n g h ig h -q u .1lin
·
, rea l - time i m age d i s p l �l\'
76
Digiral Tcchnic.11 Journal
Vol . 7 No. 4
.1 995
as part of the base gr:tphics option at minimal extra cost. This project resulted in two c hip implement<l tions, uch with i ts own V<lriation of the same basic design . The TGA2 chip w:ts designed in the Work systems Group tc)r usc i n Digital's PowerStorm 3 D 3 0
:t n d PowerStorm 4D20 graphics options. T h e Dagger chip
( D
ECchip 2 1 1 30 ) was designed in the Silicon
Engineering Group to match the needs of the
PC m:tr kct. The TGA2 and D:tggcr chips arc PC! bus masters and can accept ,·ideo chta ri·om either the host CPU or other video hardware on the
PC:
I bus.
The basic block diagram of the two chips is i l l us tt·atcd in Figure 1 . l'Cl commands are interpreted as either direct memorv access
( DMA) requests or draw ing commands, which the pixel engine block converts to h·Jmc buffer read �md write operations. Alternately,
PC:
I commands can direct�\' access the ti·ame bufkr or the ' ideo graphics arLl\'
1n the Dagger chip, the
(VGA) and
RA.tv!DAC: logic.
VGA and
RA.t\1DAC: logic is on-chip; in the TGA2 chip, this logic is implemented oH�chip. Most ofthc video rendering l ogic is contained in the pixel engine block; the command i nterpreter and
DMA engine blocks requ ire some additional logic to su pport video rendering.
The [()[[owing sections describe the capabilities, costs,
<md trade-orh of the ,·ideo rendering feature set as implemented in the Dagger <llld TGA2 graphics chips.
Defining a Low-level Video Rendering Feature Set
The key q uestion when integrating multimedia into a trad itional synthetic graphics chip is which featu res shou ld be implemented in hardware and which should be ldi: in soft\\·a1-c . A cost-cffe:.:ti,·e design cannot
<
PCI BUS
lc=>
PCI INTERFACE include enough gates to i mplement e\·crv kature of interest. In addition, time-to- market concerns do not allow all features ro be designed i nto the hardware.
Therefore, it is essenti•1l ror designers to ddine the pri mary trade-off between katures that can be easily and effectively implemented in hardware and those that can be more easily implemented in software without compromising perr<:mnance.
For the Dagger and TGA2 graphics chips, our basic decision was to lca\'C i mage compression :md decom pression in software and put all pixel processing opera tions into hardware. This approach lets sori:ware do what it does best, which is perrorm complex control of relatively smal l amou nts of data. It also lets hardware do what it docs best, which is process large amounts of data where the control is rel atively simple and is inde pendent of the data. Speciflcallv, in these two graphics chips, image sca ling, fil tering, and pixel fcxmat com·er sions are all performed in hardware.
Performing the scaling in hardware greatly reduces the amount of data that the software must process and that must be transmitted over the PC! bus. for exam ple, a 320-by-240-pixel image represented with 1 6- bit pixels requires j ust 1 50 K bytes. Even at 30 ri·anKs per second ( �)s ) , transmitting an image of this size con sumes about 5 percent of the available band\\·idth of a good PCI bus implementation . This data cou ld be displaved as a 1 , 2 8 0 by 960 array of 3 2 - bit pixels for display, which would use more than 80 percent of the rcr bus bandwid th, if the scaling and pixel r(>rmat conversion occu rs in software.
One data-intensive operation that we chose not to implement in hardware is video i n put. Designers will need to revisit this decision with each new generation
[ r - - - -
,
G E N ERAL-
PU RPOSE
[
PORT AND [ RAMDAC
VI DEO
[ ..._. CONTROL
LOGIC
I
(TGA2)
._
_ _ _ _
I j r - - - -
,
I
I
I
VGA.
VIDEO LOGIC, [
AND l
RAMDAC
VI D EO
]--
OUTPUT
(DAGGER)
I
1----I
._
_ _ _ _ j
FRAME
BUFFER
MEMORY
Figure 1
D<1ggcr cllld ·rGA2 Chip StnJcturc
DigitJI Tcchniul ]oum.1l
Vol . 7
No. 4 l 995
77
of graphics chips. For the current generation , we decided to require the usc of a separate video input card for the su bsct of systems that require video cap ture. We decided not to include video captu n: support in the Dagger and TGA2 chips for two basic reasons.
First, current application-specific integrated circuit
( ASIC) technology would have allowed only a p<lrti<ll sol ution . vVe cou ld have put a video i np u t port in hardware but could not have supported the complex operations needed tix image compression .
The second reason stems fi·om a market issue. Video display is rapidly becoming ubiquitous, j ust Js mice and multiwindow d isplays have become commonplace for interacting with PCs and workstations. It is now practical to support high-quality, real-time video dis play i n the base graphics chip. However, the market tor video i nput stations is stil l much smaller than the market for video display stations. When the size of the video input station market is large enough, and the cost of integrating video input is smal l enough, sup port for video input should be added to the base graphics chip.
Video Rendering Pipeline
This section describes the stages of video render ing that arc implemented in the Dagger and
TGA2 graphics chips. These stages are pixel preprocessing, scaling and filtering, dithering, and color conversion .
In some cases, such as scaling and filtering, the two implementations arc practically identica l . In others, such as color conversion, d ramatical ly difkrcnt imple mentations are used to address the difkrcnccs in requirements for the two chips.
Pixel Preprocessing
The first stage in the pipeline inputs pixel d :1 ta and converts it i n to a standard form to be used by the rest of the pipeline. This involves both converting input pixels to a standard ti>rmat and pretransbting pixel values or color component values. The Dagger and
TGA2 chips use DMA over the PCI bus to read packed arrays of pixels ti·om memory.
Pixel Format Conversion Multimedia images are typi cal l y represented in Y U V format, where the Y channel specitles l u m i nance and the U �ll1d V channels repre sent chromi nance. After the C:P l' ILlS decompressed the source image i nt o arravs of } : l . �ll1d \ ' pixel \ al ucs, this data is transmitted to the graphics chip in one o f a n u m ber of be speci ticd JS red/green/blue
( 1\G
B) triples i nstead of Y U V triples, or as a single index v01luc that speci fies a color ti·om a color map random -:�cccss memory
( RAM ) in the video logic. The PC! Mul!imcdia
Desl/.;n
Guide specities many standard pixel t(mn:�ts 1
Figure
2 shows some of the input pixel hm11ats that are supported in the Dagger and TGA2 graphics chips.
The
Y U V tim11ats on the left allocate 8 bits tor each channel . The upper t(>rmat of the h>ur uses 32 bits per
YUV pixel and is called Y U V - 4 :4:4 + a . '
The alpha field is optional .:111d is not used in the D:�ggcr and TGA2 chips. Alpha values are used tor blend i ng operations with parti a l ly t:Llllsparent pixels. An alpha ,·alue of zero represents a fi.dl)' transparent pixe l , and the maximum value represents a fu lly opaque pixel .
The remaining three YUV t(mnats spccit-\· a separate
Y va lue per pixel but su bsamplc the { i and I" ,·a l ues so that a pair of pixels shares the same l I and V values. Most
YUV compression schemes subs�1111ple the chrominancc channels, so rhis approach does not represent any loss of data ti·om the decompressed image. Since the h uman visual system is more sensitive to changes in luminance than to changes in chrominancc, r(>r natural images, C and V can be subsampled with little loss ofimage q ual.itv.
The three 1 6- hit Y UV formats rcprcscm the most common orderings for chrominancc-su bsarnpled
YUV vaJ ues. The littlc-endian and gib-cndia11 orderings are called Y UV - 4 : 2 : 2 . ' The l ittle-end ian ordering is the orde r th�lt is typical l y prod uced on the PC! bus
3 1 24 23
16 I S 8 7
32-81T
YUV,
LITILE- ENDIAN ORDER
0
I
ALPHA
I v
I y
I u
I
3 1 2 4 23 16 1 S
8 7
1 6-81T
YUV,
LITILE-E N DIAN ORDER
0
I
Y1
I
V01
I
YO
I
U01
I
3 1
2 4 23 1 6 1 5 8 7
1 6-81T
YUV.
G I B-ENDIAN ORDER
0
I
V01
I
Y1
I
U01
I
YO
I
3 1 2 4 23 16 I S 8 7 0
I
U01
I
YO
I
V01
I
Y1
I
1 6-81T
YUV,
BIG-ENDIAN ORDER
78
Figure 2
YUV and RGB Pixel hmmrs in the Dagger and TGA2 Chips
Digital Tcchniol journal
Vol .
7 No. 4 1 995
3 \
2 4 23 1 6 1 5
8 7
0
I
ALPHA
I
R
I
G
I
8
I
32-81T RG8 (8/8/8)
1 5 1 1 1 0 5 4 0
I
R
I
G
I
B
I
1 6-BIT RG8 (5/6/5)
7 5 4 2 1 0
I
R
I
G
I
B
I
8·81T RG8 (3/3/2)
15 14 10
9
5 4
0
H
R
I
G
I
8
I
1 6-BIT RG8 (5/5/5)
7
I
0 s-BIT INDEx
I
8-BIT INDEXED
by a l ittle-end ian machine. The gib-endian ordering is produced on the PC! bus by a big-endian machine that converts its data to little-end ian order, as req uired h>r transfer across the PCI bus. That operation pre serves byte order �i.>r 8 - bit and 32-bit data types but not for 16-bit data types l i k�.: this one . Final ly, the big cndian byte orderi ng is uscd bv some vidco rcndering software and hardware options.
The
RGB formats on thc righ t sidc of Figure 2 allo clte varying numbers of bits to the rcd , green, and blue color channels to prod uce 8- bit to 32-bit pixels.
To achieve accepoble appearance, 8-bi t RG � req uires high-quality dithering, such as that provided by the AccuVideo dithering technology contained in the
Dagger and TGA2 chips and described later in this sec tion . Thirty-two-bit RG B has an optional alpha chan nel that is not used in the Dagger and TGA2 chips.
Some hardware uses the �ield for control bits or over lay planes instead of ti.>r the alpha value. Two different
1 6 - bi t RG B formats arc common. One format pro vidcs 5 bits per color channel and a single alpha bit that ind icates transparent or opaque. The other format provides an extra bit fix the green channel, since the eye is more sensitive to green than to red or blue.
Finally, 8 -bit indexed t(mnat is shown at the bottom of Figure 2. This tim11at is simply an 8 - bit value that represents an i ndex into a color map. Dagger has an i ntegral color map and d igital-to-analog converter, whereas TGA2 n:quires an external RAtvlDAC chip to provide its color map. The 8-bit i ndexed �i.mnat can represent an indexed range of values or simply a collec tion of independent values, depend ing on the needs of the application. In the Dagger and TGA2 chips, the
� - bit indexed f(mnat is p rocessed by being passed through the Y channel .
Once in the pipeline, the pixels arc converted to a standard torm:�t consisting of three 8 - bit values per pixel. The three val ues represent
RG B or YUV compo nents, depending on the original pixel f(>rmat. I f the original field contains tewer than 8 bits , ti.>r exam ple, in the 8-bit
RG B ti.mnat, then the available bits are repl icated . Figure 3 shows the expansion of R.G B pixels to 8/8/8
RG B f(mnat. Replicati ng the available bits to till low-order bit positions is preferable to fi ll ing the l ow-order bits with zeros, since repl ication stretches out the origi nal range of values to i nclude both the lowest and highest values i n the 8 - bit range, with roughly equal steps benveen them.
Adjust Look-up Ta ble
In the TGA2 chip, a 256-emry look- up table ( LUT) may be used during pixel prepro cessing. Figu re 7 ( discussed in the section Color
Conversion Algorithms) shows this table, called the adjust
LUT, in the TGA2 pipeline. This table supports two different data conversions: lumi nance adjustment and color index conversion. The adjust LUT is not available in the Dagger chip because it requires too many gates to meet the chip cost goal tor Dagger.
Luminance adjustment is used with YUV pixel tor mats. When this feature is selected, the 8- bit Y value from the input pixel is used as an index into the adjust
LUT.
The 8 -bit value read ti·om the table is used as Y i n thc next pipeline stage. Proper programming of the table allows arbitrary lum inance adjustment fi.mcrions to be performed on the input Y value; brightness and contrast control are typically provided through this mechan ism . Stand ards for d igitally encodi ng video specifY limited ranges tor the
Y, U, and
V values, largely to prevent analog noise from creating out-of-range values.2 A particularly importallt use of this luminance adjust feature is correcting the poor contrast that would otherwise result from this range limitation. In this case, the adjust LUT may be used to remap the Y val ues to cover the full range of values fi-orn 0 to 2 5 5 .
Another desirable feature is chrominance adjust ment, under which the
U and
V values are also arbitrar i l y remapped . The }300 provides this feature; however,
TGA2 does not, for two reasons.' First, chrominance adjustment is required less often than luminance adj ustment and can be emu lated i n software when the teature is req uired . Second , chrominance adjustment consumes a significant amount of chip area-either 2 K o r 4 K bits o f memory, depending on whether
U and
V usc the same table or different tables. I n this genera tion of graphics chips, the kature could not be j usti fied in the
TGA2 chip. The Dagger chip, which was
Figure
3
Expanding RGB
Pixels to
8/8/8 RG B Fonmt
Digital Technical JournJI
Vol . 7 No. 4 1 995 79
intended tc>r lower-cost systems, incl udes neither chrominancc nor luminance adjust LUTs .
T h e other usc for the adjust
LUT in the TGA2 chip is for color index conversio n . This operation can be performed when the input pixel tc·m11at is 8 bits wide.
In this case, the 8 - bit i np u t pixel is used as an index i nto the t:�blc. The resu lting \'a luc is used as the
Ychanncl value in the rest of the pipeline, and the U and V channels are ignored . Later in the pipel ine, the color conversion stage is skipped , and the r�channcl value is used directly as the resulting 8-bi r pixel va lue.
Color index conversion is an operation that is particularl y desirable when using the Wind ows
NT operating system . Typically, 8- bir screen pixels are converted to d isplaved col ors by lllt<lllS of a color LUT in the back-end video logic. Under the X ·wi ndow
System graphical windowing environment , the map ping between an index and its color can be changed only by the application. Under the Windows
NT oper ating system, however, the mappi ngs may cha nge dynamically. Therefore, an application that has stored an i mage as 8-bit index ,·alues will ntcd index values to remap those bd(xe copving it to the screen . This con version can be done in software, hut it is flster and simpler ro usc the adjust LUT in the TGA2 chip to per form the rcnupping.
Scaling and Filtering
I n the next stage in the rendering pipeline, the chip pertorms scaling and tlltering. The Dagger and TGA2 chips support one-d imensional ( 1 - D) scaling and filter ing in hardware . Limiting the chips to 1 -D filteri ng sig nificantly simplifies the chip logic, since no line bufkrs are needtd. Somewhat higher-quality images can be achieved using two-dimensional ( 2 - D ) fi ltering, but the difference is not significant. This d i fterence is fu r ther reduced by the AccuVid eo dithering algorithm that is implemented by the Dagger and TGA2 chips.
Two-dimensiona l smoothing tilters can be su pporttd with added software processing, if required.
Bresenham-style Scaling
Image scaling in the Dagger and
TGA2 chips uses pixel replication but is not lim ited to integer mu ltiples. Instead, images can be scaled hom anv i ntegral source width to any integral desti nation width. Scal ing is implemented through an adaptation of the Bresenham line-drawing algorithm .
A compl ete description o f this Bresen ham-sryle sca ling algorithm appears in "Bresenham -sryle Scaling"; the tol lowing paragraphs provide an outline of the algo rithm, which is the same scaling algorithm used in the
J 300 fam i ly of adapters 3'
The B rescnham scaling algorithm works l i ke the
B resenham line-drawing algorithm. Suppose we are drawing a line ti-om ( 0, 0) to ( 10,
6
)
, so that dx
=
1 0 and c�v
=
6 . This is a n X- major line; that is, the line is longer i n the X di mension than in the V dimension.
80
Digit.1l Tcchniol )ou rn<li
Vol.
7 No. 4 1 995
The Bresenbam algorithm draws this \'ector b\' in itial izing an error term and then i ncrementing it c/.\ rimes, in this example, 10 rimes. Each rime the algorithm i ncrements the term, a pixel is d rawn . The sign of t he error term determines whether to tind the next pixel position bv stepping to the right ( incrementing the X positio n ) or b\' steppi ng d iagonall\' ( increment ing both X and }"). The error term is incremented in such a way t h at as the X position is incremented 10 rimes, the Y position is incremented 6 rimes, rhus d rawing the desired vector.
For Bresenham scali ng, dx rcpresems the width of the source image, and c/1• represents the \\'idth of the destination image on the screen . vVhen reducing the size of the sou rce image, dx is greater th:m d)' and the error terms and increments are set u p in rhe same way as the X-major Brescnham line drawing, :ts described in the previous paragraph. One source pixel is processed each rime the error term is incrcmcmed .
When Bresenham 's line algorithm i ndicates a step in the X dimension on I)·, the source pixel is skipped . vVhen the algorithm i ndicates :1 step in both the .\· and the V dimensions, the source pixel is written to the destina tion . As a result, cx:tcrly dx so urce pixels arc processed , and exactly d1• ofrhem
JIT drawn to the screen .
Enlarging an im:�gc works in a similar bshion . For cx:1mplc, consider
�1 source i mage rbar is nJITO\\'er than the destination image, that is, dx is less th;m d)' This is cqui,·alent to d rawing a ) : ma j or Brcscnh:tm line in which the error term is incremented d)' tin1es :�nd rhe
X dimension is incrcmcmcd dx ti mes. The scaling algo rith m draws a sou rce pixel to the desti n;Hion at each step. If the l ine-drawing algorithm i ncrements only in rhc Ydimension, it repeats the cu rrent pixel. If the lin e dra\\'ing algorithm increments in both t he X : llld the Y dimensions, it steps to and d isplavs rhc next source pixe l. Consequent!\', the dx source pixe ls arc rcplicJted to yield c/J! destination pixels, thus enlarging the image.
The Bresenham line-drawing algorithm has t\\'O nice properties that arc shared by the Brescnham scal ing algorithm. First, it requ i res no divisions to com pur e the error increments. Second, it produces lines that are as smooth as possible, gi,·en the pixel grid .
ThJt is, tor an X-major line, each of the ch pixels has a l' posi tion that is the closest pixel to the i ntersection of i rs
X position wirh the real
\'ector.
Simi larlv, the
B rescnham scaling Jlgorithm selects pixels ti1Jt h:l\'e the most even spacing possible, given the pixel gri d .
Just a s l ines c a n h e d rawn ti-om lett to right o r fi·om right to left, images can be dra\\'11 in either direction .
An image dram1 in one d irection is tht mi rror image of the image drJ\\'n in rhc other direction . J'vlirror i magi ng is sometimes used in telecon krcncing, so that users can look at themselves rhe way they norm:dl\' see themselves. Similarly, images can be ru rncd upside down by simply drawing to the d isplay fi·om bottom to top instead of fi·om top to borrom.
to
Scaling in the
V dimension is pertcmned simi larly
X-d imension sc:1.li ng. On the TGA2 chip, scaling is performed i n software instead of in hardware: the software increments an error term to decide whether to skip lines ( tor reducing) or repeat lines ( for enlarg ing). This is accepc1.blc because the CPU has plenty of spare cycles to pertc>rm the scaling compu tations while the al gorithm draws the preceding line. The Dagger chip su pports Y-d imension scaling in hardware to red uce the number of commands that are needed to scale an im;1ge .
Smooth ing and Sharpening Fi lters
Like the J300, the
Dagger and TGA2 chips provide both smoothing and s harpening tilrers. Table 1 shows the available filters.
All are three-tap fi lters that are inexpensive to imple ment in hardware . The smoothing tilters are used to improve the quality of scaled i m:�ges. The sharpening ti lters provide edge e n hancement. The two filters marked with asterisks (
* )
arc ava i l a ble only on the
TGA2 chip. The others are avaibblc on both the
Dagger and the TGA2 chips.
The th ree rows of Table l show th ree levels of smoothing and s h:trpcning tilters that can be applied .
The degree of smoothing and sharpening may be selected separate ly. The first row shows the ident i t)· ti lter. This is selected to disable smoothing or sharpen ing. The second and third rows show three-tap filters that perform J moderate and Jn Jggrcssive degree of smoothing or sharpening.
Note that when using the aggressive smoothing tilter, the center clement does not contribute to the result. This filter is intended tC.>r postenlarge ment smoothing ·when the scale bctor is large. Since enlargement is pertcm11ed by replicating some of the pixels, the center ot· any span of three pixels will be identical to one of irs neighbors when scaling u p by
J bctor of two or more. As a result, the center pixel affects the resulting image, since it is replicated either to the lefT or to the righ t. The ( l /2, 0, l/2 ) tilter aftords the gn.:atcsr degree of smooth ing that can be achieved with a three-tap tilter.
These tiltcr hmctions are simple to implement in hardware. The implementation req uires storing only the two preceding pi xels and perf(mning ri·om one to three addition or su btraction oper:ttions. The sharpen ing filters req uire :�n additional clamping step to
Table 1
Sm ooth ing a n d Sharpen ing Fi lters
Smoothing Fi lter
Degree of
Filtering
Sha rpe n i ng Fi lter
(0, 1 , 0)
('f,, 'h, 'f,)*
('h, 0, 'h)
U nfi ltered
Moderate
Aggressive
* Ava i l able o n ly on the TGA2 c h i p
(0, 1 , 0)
( -'h, 2,
-'h)
( - 1 , 3, - 1 ) * ensure that the result is in the range
0 to l .
Better tit teri ng functions could be obtained by using tive t;lps i nstead of three taps but only by significantly incre:�s ing the logic required tor filtering.
Pre- and Postfi ltering
The order in which filters
JI'C applied depends on whether the image is being enlarged or red uced. When reducing an i mage, the
Bresenham scaling a lgorithm elimi nates pixels h·om the source i mage. This can result in severe a liasing arti facts un less a smoothing filter is applied bdore scaling.
The smoothing til rer spreads out the contribution of each source pixel to adjacent source pixels.
When enlarging an i mage, the smoothing filter is applied after scaling. This smoothcs out the edges between rep licated blocks of pixels. The smoothing fil ters eliminate t h e block etkct entirelv w h e n enlarging up to rwo times the source image size . The AccuVideo d ithering algorithm also contributes to smoothing out the edges between blocks . Another way to smooth out the edges is to use higher-order i mcrpolation to rind destination pixel val ues. Such methods requ ire more logic and do nor necessarily prod uce a better- looking result, particu larly f()r modest scale bctors.
If sharpening or edge enhancement is desired, a sharpening fil ter is used in addition to whatever smoothing fil ter is selected . For reducing an image, the sharpening ti ltcr is applied alTer scaling-sharpen ing an image bef(Jre reducing its size would only exag gerate al iasi ng effects. For enlarging an image, the sharpening filter is appl ied bdorc scaling-sharpening
Jn i mage alTer e n larging its size wou ld on!�· amp! in' the edges between blocks. As a result,
II' hen both sharpen ing and smoothing filters JI'C used, one is applied betore scaling and the other is applied alTer scaling.
AccuVideo Dithering Algorithm
AccuVideo dither i n g technologv is Digital's propri etarv high-qual ity, highly d1icicnt method of render i ng video with Jn arbitrary number oLwailable colors.
Included is Y U V ·to- RG B conversion, if n ecessary, with carefi.il our-of�bounds color ma�1ping. The gen eral algorithm is described i n two orber papers in this issue ofthejuurnal. which discuss the implementation of the }300 video adapter and sottll'arc-only \'ideo p layers 3·; In the ch ips described in this paper, ll'e sim plified the general implementation of the AccuVidco technology by setting constraints on the number of available colors.
Review of the Basic Algorithm
The de, elopment of the general mean- preserving mu ltilevel d ithering algorith m is presented in "Video Renderi ng," ll'hich appears in an earl i er issue of the
journal -''
Figu re 4 i l lustrates the theoretical development of the fu n damental algorithm for ditheri n g a simple compon ent of a color image.
As stated in the earlier paper,
Digital Technical journal
Vol . 7
:-.io.
4
199S
8 1
L
'
� b
:
:-A���;;L
.. _ _ _ _ _ _
..
L
1
Fig ure 4
Multilevel Di theri ng Algorithm Used in the }300, with the Gain Function Separated from the Adjust LUT a mean-preserving dithered output level L., can be pro duced by quantizing the sum of an element from a normalized di ther array and an input level L1 by simply shifting the sum to the right by R bits. This simplified quantizer, that is, a quantizer with step size D.Q
=
2�<, is possible only if the range of i nput to the adder L;, or the n u mber of in put levels N1, is properly scaled by a gain C. In the J300 and software-only i mplementa tions, C is i ncluded in an adjust LUT. In Figure 4 , we explicitly separate
C from the adjust LUT. The adjust
LUT is optionally used to control characteristics such as contrast, brightness, and saturation.
The components of this dithering system can be designed by speci�ring three parameters: l .
Nn the number of raw input levels of the given color component
2 .
N,, the number of desired output levels
3 . b, the width of the adder in bits, and the number of bits used to represent the input levels
Using the results from the multilevel dithering algo
R
=
{
(
2 " - l
)
-
}
N.,
- I ensuring that all the raw input levels will be distin guished by the dithering system . z =
O causes N, < Nr.
This situation results in some loss of resolution of raw input levels, because, in all cases, the number of per ceived output levels from the dithering system wil l be at most N1•
Using this information and the expressions of R and
C, it is straightforward to show that R
= p - q + z, and
N1 =
(2'' - 1) 2 11 + 1 .
Further, c
=
( ( 2'' - l) 2
I<
+l ) - l
2 P - I
( 2
,,
- 1 )
2 1'
( 2 P - 1 )2 ''1 "
A key approximation made at this point is and the gain is where
C = '
- 1
N, - 1 '
The effect of the gain is multiplicative. That is, L1
=
L,
X C, where L,. is the raw i nput level. In the absence of an adjust LUT, this multiplication must be explicitly performed .
Simplified Implementation of Gain
In the above sum mary of the basic d ithering algorithm, the values of
N, and N., can be any i nteger, where N,. >
N,,.
Consider the important special case of restricting these values to be powers of two. Introducing the three integers p, q, and z, we specif)r that
N,.
=
2P, N,,
=
2'1, and b
= p + z, where z is the number of additional bits used to repre sent L1 over L,.. z
> 0 guarantees t hat N1 > Nn thus
Note that this approximation becomes better as the number of bits, p, in the raw input increases.
( 2
,,
- l)
2 ' ' '
0
= 2 -
--
.
2 ' q z > l J
With this value of
G
, the resulting modified input levels will be proportionally less than ideal by a factor of
C 2P - I
c
--
2 "
·
The fact that this error is negative guarantees that overflow will never occur in the multilevel d i thering system. Therefore, a tru ncation step is not needed in the implemen�ation. Figure 5 i l lustrates the imple mentation of C, vv.h ich consists of the su btraction of a (q - z)-bit right shift of Lr from a z- bit left shift of L,.. This simple "mu ltiplier" is what is implemented in Dagger, TGA2 , and the ZLX family of graphics accel erators, where the power-oft:wo constraint on the output levels is made.
Consider, for example, the case where jJ
=
8
(
/\� =
2 5 6
)
, q = 3
(
N., =
8 )
, and z
= l .
From the equations j ust presented, R =
6, h =
9, an
�
N;
= our approximation for the gain,
C= (2
-
449.
Although
1/4)
= 1 . 7 5 ,
82 Digital
TechnicJI journal Vol.
7
No. 4
1995
(q z)
L;
Figure 5
Parallel -shifter Implementation of the Gain Fu nction
is not eq ual to rhe ideal gain ,
G'
=
448/255
=
1 .757,
the ratio
GIG
=
0.996 is so close to uni ty that any resulting differences in output are i ndistinguishable.
Shared Dither Matrix
Another simplification can be made by having aJ l the color components in the render ing system share the same dither matri x . As defined i n
"Video Rendering," a di ther template is an array o f
N, unique elemenrs, with values
TE
! 0 , l ,
. . .
, (N, - l ) j .6 matrix element d [x,y]
. d for each location [
= i n t
{ x, y] as follows:
(
T
}
For any real number
A and any positive integer
1(, the rollowing is always true: int
{�}
= int
{ Ul� A}.
H� for each color component,
1\�, is a power of two, we can exploit this fact by storing only a single d ither matrix designed for the smal lest value of
N,,.
Speci fical ly, this would be
N,
=
2'" -
"·· I, where h is the width i n bits of the adder and R"' is the largest val ue of
R in the system. For the other larger number of output levels
N,,'
==
2'"
- ��· 1 dither matrix values d'[x, y] can easily be derived by a simple right shift by (R,, - It) bits of the stored dither mau·ix , as shown in the following equation : d [x,y]=mt
11
_ •
2 "'
"
.
Since our dither matrices are typically 32 by 32 in size, the hardware savings i n storing only one matrix is significant. Also, the stored values can be read -only memory ( ROM) instead of the more costly
RA.t\1 .
Typically, RAM requires u p to eight times the area of
ROM i n either gate array or custom implementations.
Color Conversion Algorithms
The result of the preceding pipeline stages is three 8-bit values that represent either RGB or YUV color chan nels. If this format is to be written to the frame butter, then no further processing is necessary. I f a different destination format is specified, t.hen Dagger and TGA2 must perform a color format conversion. Both chips use the same algorithm to dither RGB values down to a smaJler number of bits per color channel. Both chips aUow writing YUV pi..:xcls to the frame buffer, although
TGA2 allows the writing of only the 32-bit YUV for mar. FinaJ ly, both chips can convert YUV pixels i nto the RGB color space, but they use markedly ditkrent algorithms to perform tJ1is conversion.
Although
YlN pixels can be written to the fi·a me buffer in both Dagger and (to a more limited extent)
TGA2, neither chip su pports displaying YUV pixels to the screen. YUV pixels may be stored only in the off screen portion of the frame buffer as intermediate val ues for further processing. This is because it is Elr more efficient to convert
YlN to RG B in the rendering stage than to perform the conversion in the back-end video logi c . At the rendering stage, it need only be done at the i mage update rate of up to 30 fps. I f performed i n the back-end video logic, the YUV-to
RGB conversion must also be performed at the screen update rate of up to 76 fps. This extra , higher-speed logic may be justified if preconverting YUV to RGB noticeably reduces the image q uality. Given the
AccuVideo dithering algorithm, however, postconver sion is not necessary.
RGB-to-RGB Color Conversion
Even if both the source and the destination pixel formats represent RGB color channels, i t may still be necessary to perform a bit depth conversion . I nput pixels are expanded out to
8 bits per color channel f()r processing through the video rendering pipeline. Destination pixels may have
8, 1 5 , 1 6, or 24 bits tor RGB and so may need to be dithered down to a smaller nu mber of bits per pi xel .
TGA2 also supports 1 2- bit RG B, as described later i n this section .
Dagger and TGA2 differ somewhat in the specific formats that they su pport. Dagger allows writes to the frame bu ffer of 3/3/2 , 5/5/5, 5/6/5, and 8/8/8
RGB pixel formats. TGA2 supports all these as source pixels but does nor al low writes of 5/5/5 and 5/6/5
RGB, because TGA2 does not su pport 1 6-bit pixels in the frame buffer. Dagger supports 1 6- bit p i xels because they are very common in the PC industry. In the workstation ind ustry, however, which is TGA2's market, 1 6-bit pixels are al most unknown. As the
Windows NT operating system gains in popu larity, this situation is likely to change.
I nstead of supporti ng 1 6-bit pixels, TGA2 allows writes to the frame butter of 4/4/4 RG B pixels, with
1 6 possible shades for each of the red , green, a nd blue
Digital
Tec hnical Jou rnal
Vol . 7
No. 4 1 995
83
color channels. This is a standard pixel format tor workstation graphics, since it allows two RGB bufters to be ston:d in the space of a 24-bit, 8/8/8 RGB pixel.
This in turn a llows double buffering, i n which one image is drawn while the other image is displayed .
Double bufrering is essential tor animation applica tions on large screens, since the rendering logic gener ally can not re paint the screen bst enough to avoid fl icker dkcts.
YUV-to- RGB Color Conversion on the Dagger Chip
The key design t(x us tor the Dagger chip was to sup port low-cost graphics options with the h ighest possi ble performance and displav quality. As a result, although Dagger supports up to 32 bits per pixel, its design center is r(x 8-bit-per-pixel displays. Therefore, the algorith m that Dagger uses f()r converting YUV to
RG B produces the best possible results given a limit of just 2 56 resu ltallt colors.
The resulting ditheri ng system design is shown in
Figure 6. Note that the same system is used to dither both RGB data and
YUV data . Because the nu mber of output levels t()r each component is always a power of two, we can use the simple gain circuit of Figurc 5 and s hare the same dither matrix by right-shifting i ts contents, as derived in the last section. I n hardware, this shifting simply requi res a mu ltip lexer to select the most signitlcant bits of the data. The dither matrix is 7 bits wide to support dithe ring down to 2-bir bJuc val u es in 3/3/2 RG B , bur only 6 dither matrix bits arc used for 3-bi t output, and only 5 bits arc used for
4 -bit output.
YUV data is al ways dithered to 4 bits of rand 3 bits each of
U and V An additional bit is provided f()r the
Y channel because the eye is more sensiti1·e to changes of i ntensity than to changes of color. These
10
bits arc input to a color con1·crr LUT, \\'hich is implemented as a ROM . I ts contents arc generated by �1n a lgorithm with some out-of bounds mapping.'·' Approxi mately three- [()urths of the possible combinations of
YUV values are outside the range of colors that can be spec i ricd in the RG B color space. In these cases, the color com·ert LUT ROM prod uces an RGB value that bas the same l u minance but a less saturated color.
The color com·crt LUT ROM represents these 2 5 6 colors a s a n 8 - b i t index that i s stored in the ti·ame buffer. One addi tional bit per pi xel in offscrcen mem ory specifies which pixels resu lt from YUV conversion and which arc used by other applications. When pixels
J re read fi·om the ri·Jme bu ffer t()r dispLw to the screen,
Dagger's internal
R.A.LvlDAC reads th<1t addi tional bit per pixel to decide whether to map each byte through a standard 2 56-cntry color map or through a
ROM that is loaded with the 2 5 6 col ors selected in the color convert
LUT
RO M . As a resu lt, Dagger allows selection of the best 256 colors r()r Y UV- to- RG B con version , in add ition to a llowing color-mapped applica tions ro store 8 - bit index 1·alues in the ti·J me butkr.
DISPLAY ADDRESS
LEAD SIGNI FICANT BITS
5 5!
7
DITHER
MATRIX
7
1
,024
BY
7
BITS
COLOR
CONVERT
LUT
1 ,024 BY
8
BITS
8
R G B
COLOR
I N D E X
( Y U V I N PUT)
GIY
8
'---' �
9
8
3/3/2
R G B
( R G B I N P U T)
B/U
--=.-8.........,�
9
R4
Figure
6
Dithering <1nd YUV-ro-RGB Conversion in rhc Dagger Chip
Digital Tc�hnical jound
Vol . 7
No. 4 1 99S
It is possible to extend this approach ro use more bits of dithered YUV to prod uce more finely quantized
RG B colors. The size of the req uired look-up ROM quickly gets out of hand, however. Dagger uses a l
K by-8 - bit ROM to convert 4/3/3 YUV into 256 RGB colors. Using 4/4/4 Y UV wou ld make the ROM rive times larger ( 4K by 1 0 bits ) . To produce 4K RGB col ors wou ld req uire a R O M with l 6 K 1 2 -bit entries.
YUV-to-RGB Color Conversion on TGA2
The
TGA2 gr:1phics chip pert(>rms dithering and color conversion in the reverse order, as compared to the Dagger chip.
I n TGA2, a Y U V pixel is first convened into an RC B pi xel at 8 bits per channe l . This 24 bit RG B pixel is then either written ro the fi-Jme buffer or dithered down to 8- or 1 2- bit RGB before being written to the ri·amc bufkr. Figure 7 shows the dithering system that is used in the TGA2 chip.
The key advantage of the TGA2 approach over the
Dagger approach is that it allows deeper frame bu fters to usc higher-quality color conversion . I f a 24-bit frame buffer is being used , TGA2 allows YUV to be converted to full 8/8/8 RG I3 . On the Dagger chip,
Y UV-to - RG B conversion produces only 256 different colors, rega rdless of the fi·amc buffer depth. This is acceptable on Dagger, where 24- bit frame buffers arc far ri·om the design center. Also, the Dagger method uses rcwcr gates, which is an important consideration rc>r the cost-constr:tincd D:tgger implementation.
Another advantage of this algorithm tor TGA2 is that the set of colors used t(>r video image d isplay is the s:tmc one used by fu ll -color synthetic graphics applica tions, such as a solid modeling package or a scientific visu:tlization application . This :.1l lows a common color
map to be used by both i mage applications and shaded graphics applications. Unl i ke the Dagger chip, TGA2 does not have an integrated
RAMDAC:
and uses an external
RAMDAC.
Typical low-cost
RAM DAC:
chips provide only one 2 56-entry color map, so it is i mpor tant for TGA2 to allow i mage applications to share this color map with other applications.
Figure 8 i l lustrates how the TGA2 chip pert(m11S
Y UV -to- RG B color coiwersion. Bv the standard defin ition of the YUV fcm11at, the conversion to RGB consists of a 3 - by-3 matrix m u ltiplication operJtion in which three terms equal 1 and two terms equal 0.2
The TGA2 chip pcrfcmm this matrix multiplication using f()ur LUTs to perf(mn the remaining t()ur mu lti plications, together with some adders. A fin:tl multi plexer is required to clamp the resulting vJi ues to the range 0 to 2 5 5 .
The TGA2 color conversion a lgorithm h:ts one dis advantage: the ::tlgorithm does not handle CHit-of range YUV val ues as well as the technique used in the
Dagger chip. In D<lggcr, each Y UV tripl e that is out of range has an optimal
or
ncar-optimal RGB triple com puted f(x it and placed in the table. With the TGA2 technique, the red , green, and blue components are computed separately. The i ndividual color compo nents are clamped to the range bou ndaries, but if a YUV triple results in an out
-
ofran
g
e value fc>r green , this cannot aftcct t h e red or blue values. T h e result is some color d istortion for oversaturated images . If such a resu lt wou ld be u nsatisr:1ctory, it is necessary to adjust the colors in sofhvare, e.g., by reducing the sat uration or the i ntensity of the source image so that most YUV triples map to valid RGB colors.
DI SPLAY ADDRESS
LEAD SIGNI FICANT BITS
X y
5
DITHER
MATRIX
7
7 u y �
ADJUST
LUT
256
BY 8 BITS
8
8
R
8
YUV-TO-RGB
CONVERT
G
8
9
R
G
9
B v --------�8�---------+1
Figure
7
Dithering
System i n rhc TCA2 Chip
Digital
Tccbnic1l Journal
Vol . 7 No. 4 1 995 8 5
y
8 v
8 u
8
Figure
8
Y UV-ro- RGB Conversion in the TGA2 Chip
R
G
B
Implementation Cost and Performance
Both the Dagger and the TGA2 chips have the design goal of integrating as many as possible of the J300 design teatures i n to a single-chip graphics and video solution. Dagger and TGA2 include dif ferent teatures and implement some common tea tures i n difterent ways because each chip focuses on a different market. As mentioned earlier, Dagger is a
PC graphics accelerator chip, and TGA2 is a work station graphics accelerator chip.
Gate Cost
Table 2 shows the number of gates required to add the various imaging operations to the TGA2 chip. TGA2 is implemented in
I
BM
' s
SL standard cell technology.
The video rendering logic represents less than 1 0 per cent of the total TGA2 1ogic. The chip contains no addi tional gates tcJr video scaling or dithering logic, since nearly all the gates needed to implement those fi.mctions are already required in TGA2 to implement Bresenham line drawing and dithering of 3-D shaded objects.
Table 2 clea rly shows why the luminance adj ust
LUT was omitted fi·om Dagger. On the TGA2 chip, the LUT requires more than half the total gates used tor multimedia support.
Display Performance
The peak hardware performance t(x image operations on the TGA2 chip depends primarily on the internal clock rate, which is 60 mega hertz
( M H z ) .
The TGA2 chip is fu l ly pipcli ned, so that one pixel is processed on each clock cycle, regardless of the fil tering, conversion , or d ithering that i s required . Reducing the i mage requires one clock cyc le per source pixe l . Enlarg ing the image requires one clock cycle per desti nation pixe l . Actual hardware pedormance is never quite equal to peak rates, but TGA2 performance approaches peak rates. For example, TGA2's hardware performance limits support rendering a common
Table
2
Gates Used by the TGA2 Video Rendering Log ic
Number of Cel ls
Number of Gates
Gates per
Total Gates
( Percent)
Log ic B l ock
P ix e l
Formatting
Look-up
Ta ble
F i ltering
Color
Convert
M isce l l a neous
Total
778
9, 590
2, 265
3,486
2,2 1 0
1 8,329
584
7, 1 92
1 , 699
2, 6 1 4
1 , 6 58
1 3,747
4.2
52.3
1 2 .4
1 9 .0
1 2. 1
1 00.0 intermediate format (CJF) image that is scaled up by a factor of three in both di mensions at over 30 fps.
Actual system performance depends on many factors besides hardware performance. Typically, mul timedia images arc stored and transmitted in com pressed torm, so that display pertormance depends on the speed of the decompression hardware or soft ware. "Software-only Compression, Rendering, and
Pl ayback of Digital Video" contains tables that show the performance of a variety of Alph a Generation sys tems with software-only rendering and with J 300 ren dering hardware that implements hardware algorithms similar to those in rhe TGA2 and Dagger ch ips 5
Table 3 s hows the results of p relimi nary tests of
TGA2 video display rates on AlphaStation 250 4/ 1 66 and AJphaStation 250 4/266 workstations, which use
DECchip 2 1 064 CPUs. The table shows performance in frames per second for displaying the standard
Motion Picture Experts Group
(
M
PEG ) tlower gar den video clip, comparing performance ro software algorithms that use the TGA graphics accelerator. Like
TGA2 , the TGA chip su pports fast image transfers to the frame buffer; however, TGA does not provide any specific logic to accelerate video display.
The first two lines of Ta ble 3 show perf()rmancc f(>r displaying images at their original size. Allowing
TGA2 to convert decompressed
YUV
pixels to
RGB
im proves performance by 34 to
4 5
percent, depend ing on CPU performance. This performance improve ment drops to 1 8 to 25 when data transkr times arc included . Possibly, this gap can be reduced by fi.1 rther coding to better overlap data transfer with MPEG decompression . Note that the TGA2 performance can include image filtering and a lu minance adjust table lookup at no loss in performance.
The third line of Table 3 shows pcdorm:wce when the video clip is d isplaved at two times the size in both di mensions. The flower garden movie covers an area of 320 by 240 pixels, which is very small on a l
,280by- l ,024-pixel monitor. Therefore, it is highly desir able to display an enlarged image. In this case, TGA2
86
Digital
T�chnical juurn,ll
Vol.
7 No. 4 1995
Ta ble
3
Frames per Second for Displaying M PEG F lower G a rden Video C l i p
Al phaStation 2 50 4/1 66
AlphaStation 250 4/266
TGA
(fps)
TGA2
(fps)
Increase
(Percent)
TGA
(fps)
24.7
23. 1
1 2 .7
35.8
28.9
26.4
45
2 5
1 08
Software d ecode rate
1 x
2x video playback rate video playback rate
Source:
T o m Morris, Technical Di rector, Light and Sound Engi neering, Dig ital Equi pment Corporation
47 .9
44.0
2 3 . 1
TGA2
(fps)
64.2
5 2 . 1
44.9
Increase
(Percent)
34
1 8
95 displays the video clip at twice the speed of the soft ware algorithm that uses the
TGA graphics chip. The su bjective difference is even greater, since
TGA2 applies a smoothing filter to improve the quality of the resu lting images. The software algorithm on the
TGA chip performs no filtering because this would dramati c;dly reduce chip p e rf o rm ance.
The performance data in Table 3 are tor displaying
8-bit images to the frame buffer.
TGA2 is able to disp l ay
24 bit images at the same performance, up to the limit of its frame buffer bandwidth. For the examples in Ta b le 3,
TGA2 is able to p r od uc e either 8-bit, 1 2-bit, or 24-bit images at essentially the same pertormance.
Software algOLithms would experience a dramatic drop in performance, simply because they would have to process and transfer three times as much data. Therefore, rJ1e
TGA2 chip allows significanrJy higher-quality images to be displayed "�rJ1out sacrificing pertormance.
Conclusions
This paper describes two graphics accelerator chips that i ntegrate a set of image processing operations with tra ditional synthetic graphics operations. The image oper ations a r e carefully chosen to al low significanrJy higher pertormance with minimal extra logic; the operations that can be performed in software are left out. Both chips take advantage of the PC! bus to provide the bandwidth necessary tor image data transfers.
The Dagger and TGA2 video rendering logic is based on the AccuVideo rendering pi pe li n e as im ple mented in the
J
300 family of video and audio adapter�.' The following restrictions were made to integrate this logic into these graphics chips:
1 . Color preprocessing-Eliminate RAM tor dynamic chrominance con trol . For the Dagger chip, a l s o eliminate RAM con trol. for dynamic brightness/contrast
2 . Filtering-Support just one sharpening and one smoothing filter (other than the identity filters) in the
Dagger chip. For the
TGA2 chip, support just two sharpening and two smoothing filters.
3. Color output For the Dagger chip, allow only
256 output colors tor Y UV input [ 3/3/2 t<x
RG B input ] . For the
TGA2 chip, support only
RG B col ors with a power-oftwo number of values in each channel.
The quality of the resulting images is excellent. The
AccuVideo
32-by-32 void-and-duster dithering algo rithm provides quality similar to error diffusion dither ing algorithms:' Error diffusion is a technique in which the difference between the desired color and the displayed color at each pixel is used to control dithering decisions at ad j a ce nt pixels. Error diffusion dithering requires considerably m o r e l og ic than
AccuVideo dithering and cannot be used when ren dering synrJ1etic graphics.
The high quality of the AccuVideo algorithm is especially important when dithering down to 8 bit pixels ( 3/3/2
RGB
). Even in this extreme case, apply ing the Accu Video d i t h e r i n g algorithm results i n a slight graininess b u t few visible dithering artifacts.
Applying AccuVideo dithering to 1 2 - bit (4/4/4
RG B ) pixels resul ts in screen images that are almost indistinguishable ti-om 24- bit ( 8/8/8
RG B ) pixels.
We plan to continue evaluating new multimedia katures tor i nc l u s i o n i n our synthetic graphics chips.
Areas we are investigating include more elaborate fil tering and scaling operations, additional types of color conversion, and inexpensive ways to accelerate the compression/decompression process.
References
1 .
PCJ lvfultimedia
Design Gu ide, r e v 1 .0 ( Po rtl a nd
,
Oreg.: PC! S p e c i a l I nterest G ro u p , March 29, 1 994).
2 . Encoding
Parameters of Digital Television for Stu dios,
CCIR Report 60 1 - 2 ( Geneva: I n ternational Radio
Consultative Commi ttee [ CCI R ] , 1 990).
3 .
K.
Correll and R . Ulichney, "The ]300 Family ofVideo and Audio Ada pters: Architecture and Hardware
Design ," Digital Tech n ical jou rnal. vol . 7, no. 4
( 1 995, this issue ) : 20-3 3 .
4 . R. Ulichncy, " Bresenham style Scaling," Proceedings of tbe IS&T
Annual Conference
( Cambridge, Mass.,
1 99 3 ) : 10 1-103.
Digital Technical )ourn,ll
Vol . 7 No. 4 1995 87
S.
1'. lhh l ,
P. G<luthier, and R . Ulich ne,·, "Solt\\'<m:-on lv
Compression, Rendering, and PLn·back of Digital
Video,"
Digitcil Tecbnica!Joumal. v o l
.
7, no. 4
(
1 99 5 , this issue): 52-75 .
6. R . U l ichney, " Video Rendering,"
Oip,ilal Technical joumal. ,·o l . 5 , no. 2 ( Spri ng 1 993 ) : 9-1 8 .
7. R . U l ich ncy,
"Method and
App<lr;ltus t(>r L'vbpping a
Digiul Color I mage from a First Color Space to a Sec ond Color Space,"
U.S.
Parent 5,233,684 ( 1 99 3 ) .
8 .
R . U l ich nev, "The
Void-and-Cluster
J'vlcthod for Gen erating Dithn Arravs," !SE-7/S!'Ih. 5) '111}Jusium on h'lec tmnic lmap,inp, Science ancl T'c>clmo!og) '.
Calif. ,
\'O I .
1 9 1 3 ( Febntary 1 993 ): 332-343.
San Jose,
Biographies
Larry
D.
Seiler
LilT\' Seiler is a consul tant cngi n.:cr working i n Digi ta l's
C raph ics and M u l t i m e d i a
Gwup within the vVorkwstcms
Busi ness Unit. D u ring his 1 5
\'c a r s at D ig helped design i
LI
I
,
Larrv has a ,·arietv of graphics prod ucts. Most recent!\·, he was the ;uchittct for the
TGA2 graphics chip that is used in Digita l 's PowerSrorm 3D30 and PowcrStorm
4 D20 graphics options.
Prior to that he architccted the
SPX series of graphics options for Vf\X workstations. Larry holds a P h . D . in com pllter science ti·om the M assachuse tts
I nstitute of'T'cchnology, as wel l as B.S. and
,'v! .S. degrees from the C1lifornia I n stitute ofTechnology.
Robert
A .
U lichney
Robert U lich nev t
· c c c i ve d a
Ph.D. fi·om the LVbss<Khuserrs
I nstitute ofTechnology in electrical engineering and com puter science and a
B .S . in physics and com puter science ti·om the UtJivtrsiry of
Dayton, Ohio. He joined D igital in 1 978. Bob i s currently a sen ior consu l ting engineer with
D igit
<
1 i's Cam bridge Resc;1rch LJbOr<Horv, \\'here he leJds the Video and I mag.c Processing project. He h ;1S ti led sev
Ct'<11 p a t e nts t(Jr comriburions to Digi t
<
1
1 prod ucts in the
<1 1"C1S of hard\\'Jrc ;md soft\\'are-onlv motion \'ideo, graphics comml lers, ,1 11d lurd copv. Bob is the au thor of
OiJ.!,ilaf
!-faljiuu iu,c� and scr,·es as a reteree t(n a n u m ber of technical societ ies, including
I EEE, of\\'hich he is a senior mem ber.
88
Di�iral .kchnical
JournJI
Vol. 7 No. 4 1 995
Technical Description of the DECsafe Available
Server Environment
I
Lawrence S. Cohen
John
H. Williams
The DECsafe Available Server Environment (ASE) was designed to satisfy the high-availability requirements of mission-critical applications run n i ng on the Digital UNIX operating system.
By supplying fail u re detection and failover proced ures for redu ndant hardware and soft ware subsystems, ASE provides services that can tolerate a single point of failure. In addition,
ASE supports standard SCSI hardware in shared storage configurations. ASE uses several mecha nisms to maintain continuous operation and to prevent data corruption resulting from network partitions.
The advent of shared storage interconnect support such as the small compute r system i n terface (SCSI ) i n t h e Digital U N I X operating system provided the opportunity to make existing disk-based services more available. Since high availability is an important feature to mission-critical applications such as database and file system servers, we started to explore high -availability solu tions tor the U N I X operating system environ ment. The outcome of this effort is the DECsafc
Ava i lable Server Environment (ASE), an i n tegrated organization of computer systems and external disks connected to one or more shared SCSI buses.
In the first section of this paper, \Ve review the many product requirements that needed to be explored . We then define the ASE concepts. I n the next section, we discuss the design of the ASE componen ts. I n subse quent sections, we describe some of the issues that needed to be overcome during rhe product's design and development: relocating client-server applications, event monitoring and notification, network partition ing, and management of available services. Further, we explain how ASE deals with problems concerning mul tihost SCSI ; the cross-organizational logistical issues of developing special ized SCS I hardware and 6rmware features on h igh-volume, low-priced standard com modi()' hardware; and modifications to the Network
File System ( NFS) to be both highly available and back ward compati ble.'
Requirements of High-availability Software
The availability concept is simple. If two hosts can access the same data and one host fai ls, the other host should be able to access the data, thus making the applications that use the data more available. This notion of loosely connecting hosts on a shared storage i nterconnect is called high availability. High availability lies in the midd le of the spectrum of availability solu tions, somewhere between expensive fault-tolerant sys tems and a well -managed, relatively inexpensive, single computer system.'
Bv eliminating hardware single points o f failure, the environment becomes more available. The goal of the
Digiral Technical Journ,ll
Vol .
7
No. 4
1995
89
ASE project was to achieve a product that could be configured for no singl e point oftailure with respect to the availability of services. Thus we designed i\SE to detect and dynamically reconfigure around host, storage device, and network fai l u res.
Many requ irements i ntluenced the ASE d esign . The most overriding requirement was to eliminate the pos sibility for data corruption . Existing single-system applications implicitly assumed that no other i nstance was running on another node that cou ld also access the same data . If concurrent access did happen, the data wou ld likely be corrupted . Theretore the preemi nent challenge for ASE was to ensure that the applica tion was run only once on only one node.
Another req uirement of ASE was to use industry standard storage and interconnects to perform i ts tlmction . This essentially meant the use of SCSI storage components, and this did pose some chal lenges tor the project. I n a later section, we d iscuss the chaJienge of ensuring data integrity in a m u ltihosted
SCSI environment. Also, the limitation of eight SCSI d evices per SCSI storage bus confined the scaling potential of ASE to relatively small environments of two to fou r nodes.
Less obvious requirements affected the design. ASE would be a layered prod uct with minimal impact on the base operating system. This decision was made tor maintainabil ity reasons. This is not to say we did not make changes to the base operating system to support
ASE; however, we made changes only when necessary.
ASE was required to support multiple service types
(applications) . Original ly, it was proposed that ASE sup port only the Network File System (NFS), as does the
HANFS product t!·om International B usiness Machines
Corporation.3 Customers, however, required support tor other, primarily database applications as well. As a result, the ASE design had to evolve to be more general with respect to application availability support.
ASE was also req uired to allow multiple service types to run concurrently on all nodes. Other high availability products, e.g., Digital's DECsafe Failover and Hewlett-Packard's SwitchOver UX, are "hot standby" solutions. They req uire customers to pur chase additional systems that could be idle during normal operation . We fel t it was important to allow all members of the ASE to run highly available applications as wel l as the tradi tional, hot-standby configuration.
The remaining requirement was time to mar ket. IBM's H A/6000 and S u n M icrosystems'
SPA RCclusterl products were in the market, offering cluster-like high availabil i ty. We wanted to bring out
ASE q uickly and to follow with a true U N I X cluster product.
One last note tor readers who might try to compare
ASE with the VMScluster, a fu lly functional cluster product. ASE add resses the availabil ity of singlethreaded applications that require access to storage.
For example, it does not add ress parallel applica tions that m ight need a distributed lock managn and concurrent access to d ata. Another effort was started to address the req uirements of clusters in the
U N I X e nvironment!
ASE Concepts
To u n derstand the description of the ASE design, the reader n eeds to be famil iar with certain availability concepts and terms. I n this section, we detine the AS E concepts.
Storage Availability Domain
A storage availability domain (SAD ) is the collection of nodes that can access common or shared storage d evices i n an ASE. Figure l s hows an example of a
SAD. The SAD also i ncludes the hardware that con nects the nodes such as network d evices and the stor age i ntercon nects. The network device can be any standard network interface that supports broadcast.
This usually i mplies either Ethernet or a fiber d istrib uted data interface ( F DD I ) . Although the SAD may include many networks, only one is used tclr com m u n icating the ASE protocols i n t h e version 1 .0 product.
To remove this single point of failure, future versions of ASE will allow tor comm unication over multiple networks. Other networks can be used by clients to access ASE services. The storage i nterconnect is either a single-ended or a fast, wide-differential SCS I . The shared devices are SCSI d isks or
SCS I storage products like HSZ40 controllers.
Symmetric versus Asymmetric SADs
There are many ways a SAD may be configured with respect to nodes and storage . In a symmetric configuration (see Figure l
), a l l nodes arc connected
CLIENT
SERVER
1
SHARED SCSI BUS
STORAGE
NETWORK
STORAGE
Figure 1
Simple Availabl e Server Environment
SERV E R 2
90 Digital
Technical
)ounul
Vol . 7 No. 4 1995
to all storagc .
An asym metric configuration exists when a l l nodes arc not connected to a l l thc storage deviccs. Figure 2 shows an asymmetric configuration.
The use of asymmetric configurations improves pert(xmance and increases scalability. Performance is better because fewer nodes share the same bus and have less opportunity to saturate a bus wi th I/0 . Scal ability is greater because an asymmetric configuration allows for more storage capacity. On the other hand, asymmetric configurations add significant implemcn tation issues that are not presem with symmetric configurations. simpli�ring a ss u
S ymm etr i c configurations al low tor m p ti o n s i n device naming, detecting network partitions, and preventing data corruption.
By assuming fu lly con nected configurations, we were able to simpl i�r the ASE design and increase the software's reliability. For these reasons, we chose to support only symmetric configurations in version ofASE.
1 .0
Service
\rVe use the term service to describe the program
(or programs) that is made highly available. The service model provides n e twork access to shared storage through its own client-server protocols.
Examples ofASE services are NFS and the O RACLE? d atabase. Usual ly, a set of programs or processing steps needs to be executed sequentially to start up or stop the service.
If any of the steps can not be exe cuted successfi.dly, the service either cannot be pro vided or cannot be stopped . Obviously, if the shared storage is not accessible, the service cannot begin .
ASE provides a general i n frastructu re for specif)ring the processing steps and the storage dependencies of each service.
SERVER
1
S E RV E R 2 S E RVER 3 SERVER
4
STORAGE
1
Figure 2
Asymmetric Configuration of ASE
STORAGE 2
Events and Failure Modes
ASE monitors its hardware and software to determine the status of the environment. A change in status is reported as an event notification to the ASE software.
Examples of events include a host failure and recovery, a fai led network or disk device, or a command from the ASE management utility.
Service Failover
The AS E software responds to events by relocating services from one node to another. A relocation due to a hardware tailure is referred to as service failover.
There are reasons other than failures to relocate a ser vice . For example, a system manager may relocate a service tor load-balancing reasons or ma�' bring down a node to perform maintenance.
Service Relocation Policy
Whenever a service must be relocated , ASE uses con ti.gurable policies to determine which node is best suited to run the service. The policy is a fi.mction of the event and the installed system-management prefer ences for each service. For example, a service must be relocated if the node on which it is r u nni ng goes down or if a
SCSI cable is disconnected .
The system manager may specify the node to which the service should be relocated . Preferences can also be provided tor node recovery behavior. For example, the system manager can speci�r that a service always returns to a specified node if that node is up. For services that take a long time to start up, the system manager may speci�' that a service relocate only i f its node should fai l . Additional service policy choices are built into ASE .
Centralized versus Distributed Control
The ASE software is a collection of daemons (user-level i ndependent processes run in the backgrou nd ) and kernel code that run on all nodes in a SAD. When we were d e sig n i n g the service relocation po li c y , we cou ld have chosen a distributed design in which the software on each node participated in determining where a ser vice was located . I nstead , we chose a central ized design in which only one of the mem bers was responsible tor implementing the policy. We preferred a simple design since there was l ittle benefit and much risk to develop ing a set of complex distributed algorithms.
Detectable Network Partition versus
Undetectable Full Partition
A detectable network partition occurs when two or more nodes cannot communicate over their networks but can still access the shared storage. This condi tion could lead to data corruption if every node reported that all other nodes were down . Each node could try to acquire the servic<.:. The service cou ld run
Digital Tcchnic1l Journal
Vol .
7
No. 4
1 995
9 1
concu rrently on m u l tiple nodes and possi bly corrupt the sh :1red storage. ASE uses several mechanisms to prevent data corru ption resu lting ti·om ner..vork parti· tions. First, i t relies on the abil ity to communicate sta rus over the SCSI bus. I n this way, i t can detect ner.vork partitions and prevent multiple instances of the service. When communication cannot occu r over the SCSI bus, ASE rel ies on the disjoint electrical con nectivity property of the SCSI bus. That is, if Server l
:md Server 2 cannot contact each other on the SCSl bus, it is impossible tor both servers to access the same storage on that bus.
As a safeguard to this assumption, ASE also applies device reservations ( hard locks) on the disks. The bard lock is an extreme t:1ils:�k mech;mism that should rarely (if ever) be n eeded . As a result, ASE is able to adopt
:1 nonquorum appro:�ch to ner.vork partition handling. In essence, if an appl ication can access the storage i t needs to run, it is allowed to run . Quoru m approaches require a percentage ( usually more than half) of the nodes to be available tor proper operation.
For r.vo- node configurations, a tiebreaker wou ld be req uired : if one node tailed , the other could conti nue to operate . In the OpcnVMS svstem, for example, a disk is used as a tiebreaker. We chose the nonquorum approach tor ASE because it provides a higher degree of availability. t:J '
' situation i n which data could become corrupted: <1 ful l partition cou l d occur during shad owed storage.
Shadowi ng transparently replicates data on one or more disk storage devices. In a fu l l partition, r.vo nodes can not communicate via a ner.vork, and the SCSI buses are disconnected in a wav that the first node sees one set of disks and the second node sees another set.
Figure 3 shows an undetectable full partition.
Even though this scenario does not allow t()r com mon access to disks, it is possible that storage that is replicated or shadowed across two disks and buses could be corrupted . Each node believes the other is down because there is no com munication path . I f one node has access to half of the shadowed disk set and the other node bas access to the other half, the service be
:
ome out of sync, causing data corruption when i ts halves were merged back together. Recause the poss
, t:J simal, we provide an optional policy to:· running a service when less than a fu ll shadowed set is available.
Service Management
AS E service management provides th ree functions: ser\'ice setup,
SAD monitori ng, and service relocation.
The ma nagement program assists in the creation of services by prompting for intc>n11<1tion such as the t\'pe
CLI ENT
NETWORK
SERVER 1
)(
SHARED SCSI
BUS
1
- - - - - - - - - - - - - - -
1
SHARED SCSI
BUS
2
:
I
STORAGE
I
I
I
I
I
I
I
STORAGE
:
_ s�!'Sl'?��� p�s�
_ s
_
E
:!"
I
�
Figure
3 full
Partition
CLIENT
SERVER
2 of service, the disks and ti le systems that arc req uired , and shadowing req uirements. ASE gathers the require ments and creates the command sequences that will start the service. It thus intcgr:ues complex su bsvstems such as the local tile svstcms, logical storage manage ment ( LSM ) , and N FS into a single ser\'ice.
AS F version
1 . 0 supports three scn·ice r.·pes: user, disk, and NFS .
A user sen ·ice requires no d isks and simply allows a user-supplied script to be executed whenever a node goes up or down . The disk sen:ice is
:� user service that also req uires disk access, that is, disk and ti le system i n formation . The disk service, tC.)r example, wou ld be used for the creation of a high ly available database. The
N!-S serl'ice is a special ized ver sion of the disk service; it prompts f(x the additional int(>rmation that is germane to N I-"S, t()r example, nport i n formation.
The monitoring feature p ro\'ides the status of a ser vice, ind icating whether the scr\'ice is running or not and where. It also provides the st:ltus ut"each node.
The service l ocation fcJture :�llows svstem managers to move services manually by simply specif},in g the new location.
Software Mirroring
Sottware mi rroring (shadowi ng) is a mechanism to replic1te data across t\\'O or more disks. I f one disk tails, the data is avai lable on :ll1othcr d isk. ASE relies on
Digita l 's LSM prod uct to pro, ide this rl:ature .
92
Digital Technical
)ourn�l
Vol . 7 No 4 1995
ASE Component Design
The ASE prod uct componellts perform d istinct opcrations that correspond to one of the fol lowi ng catcgories: l . Con figuring thc availability cnvironmcnt and scrvtces
2 . Monitoring the St�HUS of the avJi lability environ ment
3.
Controlling and sync hronizing service relocation
4. Controlling and pcrforming single-system ASE management opcrations
5 . Logging events tor thc availabil ity environment
The configuration of ASE is divided into thc instal lation and ongoi ng contiguration tasks. The ASE i nstal lation process cnsurcs that all the members arc running ASE-compli:mt kernels and the required the mons ( independent proccsses) k)r monitoring the environ ment and performing single-system AS E opcr ations. Figu re 4 illustratcs these components. Thc sharcd networks and distributed time services must also bc contigured on cach mcmber to guarantce con ncctivity and synchronized time. The most cu rrent
ASE configuration infcm11ation is determined from time stamps. Configuration information that uses time stamps docs not changc oti:cn or frequently and is pro tcctcd by a distributed lock.
Thc AS E configuration bcgins by running the ASE ad ministrative command ( AS EMGR) to establish the membcrship list. All the participating hosts and
ETHERNET daemons must be available and operational to complete this task successfully. ASE remains in thc install state until the membership list has been succcssfi.Iily pro cessed. As part of thc ASE membership processing, an
ASE configuration database (ASECD B ) is crcatcd , and the ASE membcr with the highest Internet Protocol
( I P ) address on thc primary network is designated to run the ASE director daemon (ASEDI RECTOR ) . The
ASE director provides distri bu ted comrol across the
ASE members. Oncc an ASE director is running, the
ASEMGR command is used tO configure and control i ndividual services on thc ASE members. The ASE agent daemon (ASEAGENT) is responsible tor pcrf<m11ing all the single-system AS E operations required to m:mage the ASE and related services. This local system manage ment is usually accomplished by executing scripts i n a specific order to comrol the start, stop, :1dd, delete, or check of a service or set of services.
The AS E d i rector is responsible for controlling and synchronizing the ASE and the avai lable services depend ent on the AS E. All d istributed dccisions arc made by the AS E director. It is necessary that only one
ASE director be running and controlling an AS E to provide a cemra l ized point of control across the AS E.
The ASE d i rector provides the distributed orchestra tion of service operations to effect the desired recov ery or load - balancing scenarios. The ASE d irectOr controls the availabil ity services by issuing sets of ser vice actions to thc ASE agents running on each mem ber. The AS E director implements all Elilovcr strategy and control .
ASE
MANAGEMENT
UTILITY
ASE
D I R ECTOR
LOGGER
DAEMON
ACTION
PROGRAMS
ACTION
PROGRAMS
SE
GENT
ASE HOST
STATUS
MONITOR
SE
GENT
ASE HOST
STATUS
MONITOR
1
I
1
1 j' - - - - - - - - - - - - - - - - -I , - - - - - - - - - - - - - - - - _ ,
I
I
I
I
I
I
I
ACTION
PROGRAMS
ACTION
PROGRAMS
SE
GENT
ASE HOST
STATUS
MONITOR
SE
GENT
ASE HOST
STATUS
MONITOR
I
I
I
I r- - - - - - - - - - - - - - - - - _,
I
I
I
I
Figure 4
ASE Cornponenr Contigur;nion
Vol . 7
No. 4 1 995 93
The
ASE agent and the
ASE director work as a team, reacting to component faults a nd performing fai lu re recovery for services. The
ASE events are generated by the ASE host status moni tor and the avai lability man ager (AM), a kernel su bsystem. The
ASE agents use the AM to detect device failu res that pertain to
ASE services. When a device fai lure is detected, the
AM informs the ASE agent of the problem. The ASE agent then reports the problem to the AS E d irector if the f�1ilure results in service stoppage. For example, if the tailed disk is part of an LSM mirrored set, the service is not affected by a single disk failure.
The ASE host status monitor sends host- or member state change events to the ASE director. The ASE host status monitor uses both the networks and shared storage buses, SCSI buses, conflgured between the
AS£ mem bers to determine the state of each mem ber.
This mon itor uses the AM to provide periodic
SCSI bus messaging through
SCSI target- mode technology to hosts on the shared
SCSI bus.
The ASE agent also uses the AM to provide device reservation control and dev ice events. The ASE host status monitor repeatedly sends short messages, pings, to all other members and awaits a reply. If no reply is received within a prescribed time out, the monitor moves to another interconnect until a l l paths have been exhausted withou t receiving a reply. If no reply on the shared network or any shared SCSI is received, the monitor presumes that the mem ber is down and reports this to the
ASE director. If any of the pings is successful and the member was previously down, the monitor reports that the membe r replying is up.
If the only successful pings are SCSI -based, the AS E host sta tus monitor reports that the members are experienc ing a netvvork partition . During a network partition, the AS E con figuration and current service locations are frozen until the partition is resolved.
All ASE operations performed across the mem bers use a com mon distri bu ted logging facility. The logger daemon has the ability to generate multiple logs on each i\S E mem ber. The administrator uses the log to determine more detail about a particular service failover or configuration problem.
ASE Static and Dynamic Sta tes
As with most distributed applications, the
ASE prod uct must control and distri bute state across a set of processes that can span several systems. This state takes two forms: static and dynamic. The static state is dis tribu ted in the
AS E conflguration database . This state is used to provide service availability configuration information :md the
AS E system membership l ist.
Although most changes to the
ASE configuration data base are gathered through the
ASE administrative com mand, al l changes to the database are passed through a single point of control and distribution, the ASE d irec tor. The dynamic state includes changes in status of the base availability environment components and services.
The state of a particular service, where and whether it is running, is also dynamic state that is held and con trolled by the ASE director. Figure 5 depicts the tlow of control through the ASE components.
ASE Director Creation
The AS E agents are responsible for control ling the p lacement and execution of the ASE di rector. vVhenever an
ASE mem ber boots, it starts up the
AS E agent to determine whether an
ASE director needs to be started . This determination is based on whether an
AS E director is a l ready running on some mem ber. n
---z._
USER
REQUEST
0
----
HARDWARE
FAILURE
SCSI I/O
-z___
ASE
MANAGEMENT
UTILITY msgs
I
ASE
D I R ECTOR msgs l
ASE HOST
STATUS
MONITOR select
I
AVAILABI LITY
MANAGER
(AM) msgs
ASE AGENT exec rl
ACTION
PROGRAMS
11 exec r---
ASE AGENT exec r--
ACTION
L_
PROGRAMS
1-1 exec
�
ASE AGENT exec d
ACTION
PROGRAMS exec
Figure 5
ASE
Control Flow
SYSTEM
PROGRAMS
SYSTEM
11
PROGRAMS
SYSTEM
PROGRAMS
Vol. 7 No. 4 1995
94
DigirJI
Technical Journal
I f no AS E director is running and the ASE host status moni tor is reporting that no other members are up, the ASE agent forks and executes the ASE director.
Due to intermittent t:1ilures and the parallel initiali zation of members, an ASE configuration cou ld find two ASE directors running on two different systems.
As soon as the second director is discovered , the younger director is killed by the ASE agent on that sys tem . The IP address of the primary network is used to determine which member should start a ctirector when none
JS running.
ASE Director Design
The ASE director consists of four major components: the event manager, the strategist, the environment data manager, and the event controller. Figure 6 shows the relationship of the components of the ASE director.
The event manager component handl es all i ncom ing events and determines which su bcomponent should service the event. The strategist component processes the event if it results in service relocation.
The strategist creates an action plan to relocate the ser vice. An action plan is a set of command l ists designed to try all possible choices tor processing the event. For example, if the event is to start a particular service, the generated plan orders the start attempts from the most desired member to the least desired member accord ing to the service policy.
The environment data manager component is responsible for maintaining the cu rrent state of the
ASE. The strategist will view the cu rrent state before creati ng an action plan . The event controller compo nent oversees the execution of the action plan. Each of the command lists within the action plan is processed in parallel, whereas each command within a command list is processed serially. Fu nctionally, this means that services can be started in parallel, and each service start-up can consist of a set of serially executed steps.
ASE Agent Design
The ASE agent is composed of the environment man ager, the service manager, a second availability manager (AVMAN ), and the configuration database manager. Figure
7 shows the ASE agent components.
All the ASE agent components use the message l ibrary as a common socket communications layer that allows the mixture of many outstanding requests and replies across several sockets. The environment man ager component is responsible for the maintenance and initialization of the communications channels used by the ASE agent and the start- up of the ASE host status monitor and the ASE ctirector. The environment manager is also responsible for handling all host-status events. For example, i f the ASE host status monitor reports that the local node has lost connection to the network, the environment manager issues stop ser vice actions on all services currently being served by the local node. This forced stop policy is based on the assumption that the services are being provided to clients on the network. A network that is down impl ies that no services are being provided ; therefore, the service will be relocated to a member with healthy network connections.
If the AS E agent cannot make a connection to the
ASE host status monitor during its initialization, the ASE host status monitor is started. The start-up of the ASE director is more complex because the ASE agent must ensure that only one ASE director is run ning in the ASE . This is accomplished by first obtain i ng the status of all the running ASE members. After the member status is commonly known across all ASE agents, the member with the h ighest IP address on the primary network is chosen to start up the ASE direc tor. If two ASE directors are started, they must both make connections to all ASE agents in the ASE. In those rare cases when an ASE agent discovers two directors attempting to make connections, it will send
REPLIES
MESSAGE
LIBRARY
EVENT
MANAGER
STRATEGIST
•
_ _
I
I
I
I
- - 1 t t
ENVIRONMENT
DATA MANAGER
I
I
I
•
EVENT
CONTROLLER
!
Figure 6
ASE Director
Digital
Technical J ournal
Vol
7 No. 4 1995
95
MEM BER
MAINTENANCE
INITIALIZATION
REQUESTS
_ _ _ _ _ _ _
.,..
REPLIES
MESSAGE
LIBRARY
ENVIRONMENT
MANAGER
R E P LIES
+
_ _ _ _ _ _
I
I
J
, - - - - ·
I
- -
-
-
- - - �
REQUESTS
AVAILABILITY
MANAGER
(AVMAN}
- - - - - - -
-
,
-
- - - - - ,
I
I
I
I
_ _ _ _
CONFIGU RATION
....,
DATABASE
+
MANAGER r-
1--
SERVICE
MANAGER
I
REPLIES
Figure 7
ASE Agent an exit request to the younger director, the one with the newer start time.
The service manager component is responsible tor performing operations on ASE services. The :ervice manager performs operations that use spec1fic ser vice action programs or that determine and report sta tus on services and their respective devices. The service actions are forked and executed as separate processes, children of the agent. This allows the ASE age n t to continue hand ling other para llel actions or requests.
The ASE agent is aware of only the general stop, start, nor aware of the specific application derai ls required to implement these base avai lability fi.mctions.
_A more detailed description of the ASE service interfaces can be fou n d in the section ASE Service Definition. vVhen the service manager executes a stop or starr service action that has device dependencies, the ASE agent provides the associated device reserves or unreserves to gain or release access to the device. Services and devices must be configured such that one dev1ce may be associated with only one service . A device may not belong to more than one service.
The agents' availability manager (AVMAN ) compo nent is cal led by the service manager to process a reserve or unreserve of a particuiJr device fix a ser vice stop or start action. The AVMAN uses ioctl( ) calls to the AM to reserve the device, to invoke SCSI dev1ce pingi ng, and to register or unregister tor the tc>llowing
AM events: l . Device path failure-an
I/0 attempt failed on a reserved device due to a connectivity fail ure or bad device.
2 . Device reservation f�1ilure-an
I/0 attempt tailed on
a
reserve device because another node had reserved it.
96 Digital Technictl Journal
Vol. 7 No. 4 1 995
3.
Reservation reset-the SCS I reservation was lost on a particular device due to a bus reset.
A reservation reset occurs periodically as members reboot and join the AS E. The ASE agent reacts by rereserving the de\'ice and therebv conti nuing to pro vide the service. I f' the reser\'ation reset persists, r iK
ASE agent inf(mm the ASE director. If a device path fai lure occurs, the ASE agent informs the ASE director of rhe device path fai l u re so that another member can access the device and resume the service . The de,·ice reservation t�1ilure can occur only if another member has taken the reservation . This signifies to the AS E agent that an AS E director has decided to run this ser vice on another member without first stopping it here.
The con figuration database manager component hand les requests that access the ASE configuration database. vVorking through the configuration database manager component, the ASE agent provides
all
access to the AS E configuration database tor all other com pone nts of the ASE .
ASE Availability Manager Design
The
availability
manager ( AM ) is a kernel componcm of ASE that is responsible tor providing SCSI device control and SCSI host pi nging with target mode. The
AM provides SCSI host pinging to the ASE host st:�tus monitor daemon through a set of iocrl ( ) cal ls to the
"/devjam_host* " devices.
As
has been mentioned , the
AM provides SCSI device control for pings :� nd event notification to the ASE agent through ioctl( ) call s to the "/dev/ase" device. All ASE SCSI device controls for services and SCSI host pinging assu me that all members are symmetricallv con figured with respect to SCSI storage bus addressing.
ASE Host Status Monitor Design
The ASE host St;ltus monitor (ASEHSM) component is responsible for sensing the status of members and interconnects used to com mun icate between mem bers.
As previously mentioned , this monitor is designed to provide periodic pinging of a l l network and SCSI interconnects that arc sym metrically configured between ASE members. The ping rate is h ighest, l to seconds per ping, on the first configured
3
ASE network and SCSI bus. Al l other shared interconnects are pinged at a progressively slower rate to decrease the overhead while stil l providing some i nterconnectivity state. The ASE host status monitor provides member state change events to both the ASE agent and the ASE director. The ASE agent i nitial izes and updates the monitor when mem bers are added or deleted from tbe
ASE configuration database. The ASE host status monitor is designed to be fkxible to new types of net works and storage buses as well as extensi ble to incn::1.sed numbe rs ofsh:1.n.:d interconnects.
ASE Service Definition
AS E has provided an interface ti·amework for available applications. This framework detines the avai labili ty configu ration and fai lm·er processing stages to which an appl ication must contemn . The appl ication intcr t:Kcs consist of scripts that Jre used tO start, stop, add, delete , query, check, and modi�' the particu l ar service.
E;1ch script has th<.: ability to order or stack a set of d<.:pcndent scripts to suit a m u ltilayered application.
Th<.: rS
Service
Failovcr section i n this paper pro vides an example of a multilayered service that AS E supports "out of the box ." AS E assumes that a service can be in one ot' th e td lowi ng states: l .
Nonex istent-not con figured to run
2. Off l ine-not ro be run but configured to ru n
3 .
Unassigned-stopped and contigured to run
4 . Ru nning-ru nning on a member
At initialization, the AS E director presumes all con figured services shou ld be started except those in the off line stare . Whene\'er a new membe r joins rhc ASE, the add sen·ice action script is used to ensure that the nc\1' rm:mbcr has been con tigured ro have the abi lity to run the service . The delete service script is used to remove the abi l i ty to run the service. The delete scripts an.: run whenever a s<.:rvic<.: or member is deleted . The st:trt service script is used to start the service on a par ticular member. The stop service is used to stop a ser vic<.: on a particular mem ber. The check script is used to determ ine if
;1 sen·icc is running on a particu lar mem ber. The qu<.:ry script is used to determine if a par ticular dc,·ice bilure is sufricicllt to warrant tailovcr.
AS t strives to keep a service i n a known state . Con scquemly, i f a st:�rt anion script fails, ASE presumes that executing th<.: stop action \\'ill return the scn·icc to an unassigned state.
Likewise, if an add action bi ls, a delete action will retu rn the service to a none xistent state.
If any action bits in the processing of an action l ist, the entire req uest has tai led and is report<.:d as such to the ASE director and in the log. For more det:�ils on ASE service action scripts, see the
G11 ide to the
DECsaje A uailah/e Sercer.'
NFS Service Failover
In this section, we present a wa lk-through of an NFS service fai lover. We presume that the reader is familiar with the workings of NFS . ' The NFS service exports a tile system that is remotely mou nted by clients and locally mounted by the member that is providing the service. Other ASE mem bers mav a lso remotely mount the NFS tile system to prO\·idc common access across a l l ASE members .
For this exam ple, assume that we have set up an N FS service that is exporting a U N ! X til e syst<.:m ( UFS) named /foo_nts. The U FS resides on an LSM disk group that is mirroring across two volu mes that span four disks on two difkrellt SCSI buses. The NFS ser vice is cal led tcm_nts and has been given its o\\'n I P address, 1 6 . 1 4 0 . I 2 8 . 1 2 2 . All remote c lients who w:�llt to mount jtoo_nts wi l l access the server using the service name t(>o_nts and associated I P address
1 6 . 1 40. 1 28 . 1 2 2 .
This network address i n formation was distri buted to the clients through the Berkeley
I nternet Name Domain
( 131 D) service or the net work intonnation service ( NI S ) . If several NFS mount points are commonly used b�· all cl ients, they can be grouped into one sen·ice to reduce the nu mber of I I' add resses requi red . Although grouping di rectories exported tl·om NFS into a single ser\'icc red u ces the management m-erhcad , it al so reduces fl exibil ity t(x load balancing.
Further, assume that the NfS service t(>O_nts has four clients. Two of the clients are the mem b<.:rs of the
AS E. The other t'\vo clicms are non- Digi tal systems.
For simplicity, the Sun :�nd HP clients reside on the same net'\vork as the scn·ers ( but they need not ). The
ASE NFS ser\'ice foo_nts is currenth' running on the
ASE member named M U NCH . The other ASE mem ber is up and named ROLAIDS .
Enter o u r system admin istrator with h i s :t!tcrnoon
Big G u lp Soch. He places the Big Gulp Soda on top of
1VI UNCH to tl-ec his hands tor some serious console typing. Oh
1
We forgot one small aspect of the sce nario. This ASE site is l ocated in Calitc>rnia. A small tremor later, and
Gulp
MUNCH gets :1. good taste oftbe Big
Soda.
Seconds later, MU1 CH is \'CI'Y upset and fails. The AS E host status monitor on ROLAI DS srops receiving pings ti-om MUNCH and decl ares
M U NCH to be down . I f the ASE director had been running on
Digital Tcdmic.1l }ounul
Vol . 7
0:o. 4 1 995
97
M UNCH, then a new director is started on ROLAIDS to provide the much-needed relief. The ASE director now running on ROLAI DS determines that the foo_nfs service is not currently being served and issues a start plan for the service. The start action is passed to the local ASE agent si nce no other mem ber is available.
The ASE agent first reserves the d isks associated with the foo_nfs service and runs the start action scripts.
The start action scripts must begin by setting up LSM to address the mirrored d isk group. The next action is to have U FS check and mount the /foo_nfs file system on the ASE hidden mount point /var/asc/mnt/ foo_nfs. The hidden mount poin t helps to ensure that applications rarely access the mount point directly.
This sateguard prevents an unmounting, which would stop the service. The next action scripts to be run are related to NFS . The NFS exports fi les must be adjusted to i nclude the foo_nfs tile svstem entry. This addi tion to the exports files is accomplished by adding and switching exports include files.
The action scripts then con tigure the service address
( i fconfig a lias comman d ) , which results i n a broadcast of an Add ress Resolu tion Protocol (ARP) redirection packet to all listening clients to redirect their I P add ress mapping f(x 1 6. 140. 1 28 . 1 22 from
1\t! UNCH to ROLAIDS.'' After all the ARP and router tables ha,·e been updated, the clients can resume communications with ROLAJDS for service foo_nfs . This entire process usually completes within ten seconds. The storage recovery process ofi:en contri butes tbe longest d u ration. Figure 8 summarizes the time-sequenced events for an NFS service fai l over.
This scenario works because NFS is a stateless ser vice. The server keeps no state on the clients, and the clients are willing to retry forever to regain access to their NFS service . Through proper mounting opera tions, all writes are done synch ronously to disk such that a cl ient wil l retry a write if it never receives a suc cessful response .
If ASE is used to fail over a service that requires state, a mechanism h as to be used to relocate the required state in order to start the service. The ASE product recommends that this state be written to fi l e systems synchronously i n periodic checkpoints. In this manner, the tailover process coul d begin operation at the last successful checkpoi nt at the time the state d isk area was mounted on the new system. If a more dynamic tailover is required, the services must syn chronize their state between members through some type of network transactions. This type of synchro nization usually requires major changes to the design of the appl ication.
Implementation and Development
We solved manv i nteresting and logistical ly difficul t issues during the development of the AS E product.
Some of them have been discussed , such as the asym metric versus symmetric SAD and distri buted versus centralized policy. Others arc mentioned in this section .
I
SUN CLI ENT
J
1
1
:
:
1
1
1
,'
,
/ /
/
- - - - - - - - - - - -
/
To - Initially, MUNCH serves NFS service foo_ nfs.
T 1 - NFS clients mount foo_nfs from MUNCH.
' ,
'
HP CLIENT
'
' '
T2 - MUNCH goes down.
T
3 -
ROLAIDS senses that MUNCH is down and begins failover by acquiring the disk reservations foo_nfs service. for the
T4 - ROLAIDS broadcasts an ARP redirection for the I P address associated with foo_nfs.
T
5 -
HP and SUN clients update their route tables to
T6 reference ROLAIDS for foo nfs.
Clients resume access to f oo
_nfs from ROLAIDS.
'
'
'
I
I
I
I
I
I
'
I
I
I
I
COMMON NETWORKS
COMMON NETWORKS
ASE SERVER ROLAIDS
ASE SERVER MU NCH
SHARED SCSI BUSES
SHARED SCSI BUSES
I
EJEJ
I
Figure 8
Time-sequenced
Evcnrs tor NfS railover
Vol. 7
No. 4 1 995 98
Digit;l l TecbnicJI
J ournal
The SCSI Standard and High-availability Requirements
The SCSI standard provides two levels of req u ire ments: mandatory and optional . The AS E require ments fal l into the optional domain and are not normally implemented in SCS I control lers. In particu lar, ASE requires that two or more initiators ( host SCSI controllers) coexist on the same SCSI bus. This feature al lows for common access to shared storage. Normally, there is only one host per
SCSI, so very l ittle testing is done to ensure the electrical i ntegrity of the bus when more than one host resides. Furthermore, to make the hosts uniquely addressable, we needed to assign SCSI
I Ds and not hardwire them . Lastly, to support its host sensing needs, ASE requires that
SCSI controllers respond to commands fi·om another controller. This
SCSI feature is called target- mode operation.
In addition to meeting the basic functional SCSI requirements, we had to deal with testing and qualifi cation issues. 'vVhen new or revised component s were used in ways tor which they were not origitHIIy tested, they could break; and i nvariably when a controller was fi rst inserted into an ASE environ ment, we found problems. Additional qualifications were req uired tor the SCSI cables, disks, and optional
SCSI equipment.
AS E required very specitic hardware ( and revisions of that hardware ) ; it wou ld be difticult to support off the -shelf components.
Note, however, when
<111 was said and done, only one piece of hardware, the Y cable, was invented for
AS E. The Y cable <11 1 ows tbe SCSI termination to be placed on the bus and not in the syste m. As a resu lt, a system can be removed without corrupting the bus.
The challenge for the project was to convince the h ardware groups within Digital that i t was worth the expense of all the above n.:qui rements and yet provide cost- competitive comrol lers. Fortunately, we did; but these issues are ongoing in the development of new controllers and disks. Our investigation contin ues on alternatives to the target mode design. We a lso need to develop ways to reduce the q ualitication time and expense, while i mproving the overall quality and avail abi lity ofthe hardware.
NFS Modifications to Support High Availabi lity
The issues and design of N rS bilover could consume this entire paper. vVe discuss only the prominent poi nts in this section.
NFS Client Notification
The tirst challenge we raced was to determine how ro tel l N rS c lients which host was serving their tiles both d uring the initial mount and after a service relocation .
The ideal solu tion would have been to provide a n IP address that all nodes in the SAD could respond to . If c l ients knew only one address, aJI
NFS packets would be sent to that address and we would never have to tell the cJient the location had changed . The main prob lem with this solution is performance. Each node in the
SAD would receive all NFS traffic destined for all nodes. The system overhead for deciding whether to d rop or keep the packet is very high. AJso the more nodes and NFS services, the more likely it would be to saturate individual nodes. Unfortunately, this solution had to be rejected.
The next best solution, i n our minds, is per service
I P addresses. Each
NFS service is assigned an IP address ( not the real host add ress ) . Now each node in the SAD could respond to its own address and to the addresses assigned to the
NFS services that it is run ning. The main issues with this approach are the fol lowing: ( l ) It could use many I P addresses and ( 2 ) I t is more difficult to manage because of its many add resses. However, there were no pertormance trade-oHs, and we could move services to locations in a way that was transparent to the NFS clients.
Notifying the clients after a relocation tu rned out to be easy because of a standard feature in the AR.P that we could access through the ikon fig alias command of the Digital U N I X operating system.'' Essentially, al l clients have a cache of translations t()l"
J P addresses to physical add resses. The AR.P teature, which we reterred to as ARP redirection, allows us to invalid ate a c lienr-cac bed entry and replace it with a new one. The iteonfig com mand indin:ctly generates an A RP redi rection message . A s a result, the cl ient N rS software believes it is sending to the same address, but the net work layer is send ing it to a different nod e .
Similar functionality cou ld have been achieved by requiring m u l tiple network controllers connected to a single network wire on the SAD nodes. This solution , however, requi res more expense i n hardw:Jt"C and is
Jess t1exible since there is only one address per board .
Essential ly, the latter means the granularity ofNtS ser vices would be much larger and could not be d istri b uted among many SA D nodes without a great deal of hardware.
NFS Duplicate Request Cache
The
NFS duplicate request cache im proves the pcrtor mance and correctness of an
NFS server.' Although the duplicate req uest cache is not preserved across relocations, we did not view this as a significmt prob lem because this cache is not preserved across reboots.
Other Modifications: Lock Daemons and mountd
We mod i fied only two pieces of software rel ated to
NFS failoYer: the lock daemon and the mountd. \Ne wanted the lock d aemon to d istinguish the .locks asso ciated wirh a specitic service address so th:n only those
Digital Technical )ournJI
Vol . 7 No. 4 1 995
99
locks \\'ould be removed d uring a reloc:J.tion. After the service is relocate d , we rely on the existing lock reestabltshment protocol . \Ve modified the mountd to support
N�S loopback mounting on the
SAD, so that a tile system could be accessed d i rectly on the
SAD
( as opposed to a remote client) :md yet be relo cated transp:�rentlv.
Future of ASE
Digital's
ASE prod uct was designed to address a small, symmetrically configured availability domain. The implernent:1tion of the
ASE product was constrained by time, resou rces, and i mpact or change in the base system. Consequentl y, the
ASE prod uct lacks extensi bility tO l arger asymmetric configurations and to more complex application availability requirements, e . g . , support of concurrent d istributed applications. The n e x t g ene ra t i on :tvailability product must be designed to be extensible to varying hardware contlgurations and to be flexible to various application availabilitv requirements.
Acknowledgments
We thank the t(Jilowing people t(Jr contributing ro this docu ment through their consu ltation and artwork:
Terry Linsey, Mark Longo, Sue Ries, Hai Huang, :1nd
\Vavnc CJrdoo.
References l .
Sun Microsysrc ms, Inc.,
/vi-S
.Vetu •ork File Svsteill
Protocol Speciji"cution, RFC
1 094
(
N c r work Inrrmm tion Cemcr, S IU I n ternation a l
, 1 98 9 ) .
2 . J . Gr�1·,
"
H igh A1·ai labilirv Com p u ter Svstellls,"
!FhF
0Hnjillll'r ( September 1 99 1 ) .
3. A . 13hidc, S. I'vlmga n, and E .
E l t
1 o z ah 1
·
,
"A
Highh'
A1·ail�1blc Ncrwork File Server,'' Couji>rence Prucecd il ll-iS ji·onl the l
:\·enix Confi'reuce. Dal las, Te x .
( vVinr c r 1 99 1 )
4. VV.
Ca rd m.
c
1
, F G l on:r, and W. StL1man, " Design o�· the Digital U � ! X
Cl uster
S
1 s tc m ,
" f)fgiwl
TL'ch u iwl
.foun wf. t< m h com i ng 1996.
5.
Cr'11 ide to the f)f:Csufe AI'Uilol;le
Sl 'II 'Cr ( Mavn ard ,
Mass. : D igi ta l
Equ i p ment Corp or ar i on , 1 995 ).
6. D. Plum mer, Uhernet Address Nesolution
Protocol:
Or Cou/'crtiu,q Nettt•ork
P r u t oco l
A ddresscs to 8-hit l:'lhcrue/ Address jiH T!'Cti/SIIlission
011 t:thernet
H({ rdu ·rm'.
R.I:'C 0826
(
;\!ctll'ork In
�( m n arion Cenrcr,
SlU I mc rnarion�1l, 1 9 82 ).
7. C:. J uszc z,1k,
"I m p ro1
· i n g rhc Pc rr(mn:-tncc and Cor r e ctn e s s of an � I:'S Sen·er," Conji:reuce Proceedings ji-om the [lsenix Conference, s�1n Diego, Cal if
( Wi nter 1 9 89 )
Biograph ies
Lawrence S. Cohen
LJtTI' Cohen led the A1·aiL1ble Scn·er £ n,·ironmcnr project. He is �1 principal sofiwanO> en g; i n cc r i n Digitc1 l 's
U � I X Engineeri ng Croup, ll'here he is currenrll- 11·orking on Digital's C � I X c l uster products. Since joining Digi tal in I 983, he has written network �nd terminal device d rivers and II'Orked on the o r igi na l U LTRIX port of BSD sockets and the TCP /1
P itnplcmc nration ti·o m BSD
U �I
X Larrv
:tlso parrici pcHed in rhc im plementation of Digi tal's 1/0 porr <l l·c hitecrurc on sion l
' l .TRI X and in the porr of � FS ,·cr
2 . 0 ro the DEC : OSF/ 1 1-ersion of U l\: I X . Larn· 11·as previous
I\·
U l\: I X ro L'\llX Cop1· Prog,1'.1111 m
�
H h ( 1 976) 3nd an M.S. in compute r science
( I 98 I ) , borh fi-om the
Univcrsitv of Connecticut.
emp loved at Bell Labs, wh ere he 11 orkcd on the
( UUCP).
He h �
1 s
�1
B.S. in
John
H . Williams j o h n Wi lliams is a p ri nc ipJI soft11 arc engineer in Digita l 's
U N I X Engi nening; Croup. John led the ath·anccd de,·c l o p m c n t eftorrs r(lr the U N I X c l us tel' prod uct c1nd 11 as r h e p r o ject leader �(u· r h c DECsafc Available S<.:l'l'cr Elll·iro n mcnr ,·ersion 1 . 1 a n d \ c rsion 1 . 2 . 13etorc rhar, john dc.,igned
�1 nd implcmet1tcd rhc scc urin· i nrcdacc arc h i tecture r(u· the
DEC
OSF/ l opcr�Hing sl·stc m . Cu rre ntlY, ) o h11 is respon si ble r(>r the L' :-\ I \ clusrn mc111agement �caturcs. John recci,·ed c1
B .S . i n computer science from the M ich igan
Tcchnologiul
Uni1crsirv in 1 9 7 8 .
1 00
Di�iral Tcchnic:ll /ou nul
Vol.
7
�o.
4
1 995
L
I
Michael Palmer
Jeffrey M. Russo
Parasight: Debugging and Analyzing Real-time
Applications under
Digital U N IX
Conventional UNIX debug and analysis tools, with their static debugging model and low resolution-sa mpling profiling techniq ues, are not effective in dea ling with real-ti me appl ica tions. Encore Computer Corporation has devel oped Parasight, a set of debug and ana lysis tools for rea l-time applications. The Parasight tool set ca n debug running progra ms, debug multiple programs, constantly mon itor local and global variables, and perform on-the-fly execution analysis. Th us, Parasight provides much improved debug and analysis capa bilities, which application developers can use on both static and dynamic applications. Parasight can be used on any of Digital 's Alpha platforms run ning under the Dig ital U N I X operating system.
Because of their time -critical n:�turc, real -time applica tions do not respond wel l to the perturbations that conventional
U N I X debug and :�nalysis tools cause.
For i nstance, the static debu gging model of the dbx debugger req uires that a program be stopped bdorc it can be debugged . Also, execution an:� l ysis using the profiling tec h niques of the prof profiler often provide erroneous results tor real-time appl ications because of the low-resolution sampling employed .
This paper describes the critical aspects of debugging real-time applications, the ddiciencies fou n d in con ventional
C NIX tools, and the methodologv Encore
Computer Corporation used to develop Parasight, a set of easy-to- use graphical user i nterface tools that debug and perform execution analysis on real -time programs while they are running. Parasight can be used on any of Digital's Alpha platforms that operate under the Digital
UNIX operating svstem.
Real-time Appl ications
Real - time applications pertorm
J wide variety of fu nctions, trom tlying state-ofthc-arr military aircraft to controlling nucle<1r power plants. Al l real -time <1ppli cations ha"c one common dcnomin:�tor: Thev must complete their calcu lations before
:t dead line cxpin.:s.
T:�king too long to calculate the correct ans\\'er can ha\'e just as detrimental an effect
JS a rri,·i ng at an i ncor rect answer; either result could c1usc an :�ircraft to crash or a nuclear power plant to experience a meltdown .
Most real-time applications consist of one or more programs th:lt arc scheduled to run in response to an e\'ent. The triggering C\'ent is usualh· transmitted in the torm of an i nterrupt and can be generated randomly by an extern<ll e\'ent or regularly lw
J interval timer run ning at a tixcd rate, such as
60 times per second . Once the interrupt is received, the applic:�tion must pcrtcm11 the allotted t<1sk bdc)JT the next interrupt occurs.
The cleme nts of a real -time application com m u n i cate \\'ith each orher dvnam ic:�l l\'; th;n is, the results of the calculations of one element are used immedi atel v t(x the calcu btions of another clement. Real - time applications arc often referred to
3 S dynamic applica tions, since the\' react dvnamicall\' to changes i n their
Vol.
7
No. 4
1 <.195 I O L
environment and often refer to elapsed time i n their calculations. In contrast, static applications have resu lts that rarelv depend on changes in their environ ment or on elapsed time.
The Problems Associated with Debugging and
Analyzing Real-time Applications Using
Conventional UNIX Tools
Debugging a real -time application during execution, debugging and analyzing m u ltiple programs, con stantly monitoring variables, and analyzing program execution are all activities that debug and analysis tools have to deal with. This section d iscusses the capabil i ties a n d limitations o f conven tional U N I X tools and describes the features req uired of effective real-time debug and analysis tools.
Running Programs
Debugging a static program typically involves control l ing the execution tlow and examining the values of variables withi n the program . Stopping a real -time program or even delaying i t by single stepping, how ever, is usually not possible without adversely affecting the application. Debugging real -time applications is therefore limited to examining the values of program variables while the program is still running.
Conventional UNIX debuggers are not able to examine variables during program execution and therefore cannot be employed on running real - time applications. Consequently, these debuggers are usefu l only i n the early stages of real-time program develop ment, essentially while the program is sti ll static.
The traditional methods of debugging real-time applications i nvolve placing all the critical data into one or more global, shared memory regions. A data monitoring tool , usually written by the user, runs as a normal UNIX process and attaches to the global region. The tool can then be used to inspect and/or change the values of the global variables. This t ech nique is nonintrusive in that it does not afkct the real time application programs in any way. Unfortunately, the debugging is restricted to global data, and , unless the programs are designed with this in mind, this restriction can be a severe limitation . Modi�1ing exist ing programs to change local data into global data tor debugging purposes can result in a whole new set of problems in managing the separation of data.
An effective real-time dehug,u,.ing tool must be ahle to attach to a nmning prop,ram without stopping it and then be able to nonintrusively inspect and/or change the global data
.
Debugging and Analyzing Multiple Programs
Real-time applications typically consist of several pro grams working together. I nvoking multiple copies of the dbx debugger to debug each program individ ually is cumbersome and precludes studying the interaction between programs.
A real-time debugger must be able to work with one or more programs at the same time, providing the user with an integrated and cohesive debugging environment.
Monitoring Variables
The one-shot variable evaluation capability of conven tional UNIX de buggers is of limited use for programs that are running. These debugging tools provide the user with only one previous value of a variable, not necessarily the current value.
A real-time debugger must be able monitor the values
to
constantly
ofglohal
variables. The minimum and maximum values that each variable allained should optionally be available as a record oftransient conditions,
Execution Analysis (Profiling)
Since performance is important in real -time applica tions, program execution analvsis is often needed to locate areas of a program where the pedonnance cou l d be i mproved . A real -time appl ication may have a strict execution order requirement, whereby one routine must run prior to the execution of another routine. This requirement may be accomplished easily if the romines are in the same program; however, often the routines are in different programs or are executing on difterem CPUs in a symmetric m u l tiprocessing
(SMP) environment.
The Digital UNIX profil ing tools provide two kinds of execution analysis: l . PC sampling, which involves i nterrupting the program periodically to record the val u e of the program coun ter.
2. B lock counting, which inserts profi ling code at key points in the program to cou nt the number of times each basic block of code executes. ( A basic block is a region of the p rogram that can be entered onlv at the begi n ning and exited only at the end . )
Both techn iques involve the t(>ll owing basic steps: l .
Preprocess the program to prod uce the desired profiling information .
2 . Execute the program to produce a profiling data ti le.
3 . Postprocess the program with the profi ling tools to view the data collected.
The normal sampling period employed bv the
PC-sampling method is based on the hard clock
( C:LOCK_RENTIME) of the Digital U N I X operating system . This method resul ts in l
,024 samples being
I 02 Digiul T<:chniC<11
Joumal
VoL 7 No. 4 1 995
taken per second, which provides a timing resolution of976 microseconds, or approximately l millisecond.
The routines that make u p a real-time application typically take from a tew microseconds to several milliseconds to execute . Attempting to measure the execution time of routines that take less than l m i l lisecond t o execute with a clock resolution o f l milli second can lead to erroneous results. A test on a
1 50-megahertz ( MHz) Alpha 2 1064 CPU showed that the prof tool, using the normal PC-sampling rate, reported the execution time of a routine to be 4 milli seconds when the true execution time was 20 micro seconds. (The true execution rime was measured using the Parasight tool set. )
It is possible to increase the sampling rate using the uprofile uti lity, but doing so also proportional ly increases the number of interrupts per second that the system must handle. For instance, to obtain even
10-microsecond resolution would require the system to handle 100,000 interrupts per second. This amount of i nterrupt activity would rapidly swamp the system, l eav ing little or no CPU rime to execute the program being instrumented. The PC-sampling method of execution analysis is therefore not suitable for the short execution times typical in real-rime application rou tin es.
The block-counting method, a lthough capable of high-resolution measurement, sufters from the inabil ity of the pixie utility to work with programs that receive signals. Most real-time applications use signals for program scheduling and are therefore disquali fied from using the block-counting method .
In addition to the problems just discussed, the tradi tional U N I X profiling tools are unsuitable for real time program execution analysis for the following reasons:
•
•
A program must be preprocessed for protlling prior to execution. Adding or r e m ovi n g profil ing requires stopping, processing, and restarting the program . This assumes that the problem area is known before the application starts to r u n . If a problem suddenly develops after an u ninstru mented program has been running for 24 h ou rs, the user will have lost the opportunity to determine which routine is causing the problem.
A program must be profiled as a whole, unless source code modifications are made to the program to control the protiling. This can cause excessive overhead, which real - time programs usually cannot tolerate.
•
The profiling results cannot be seen until the pro gram termi nates, unless source code modifications are made to the program to permit the results to be dumped on command . The user needs to see the results while the program is running a nd often needs to repeat a rest several rimes to get the
• desired results. Stopping and restarting the applica tion once for each test could be laborious.
Only the average and cumu lative times for each routine are available. That is, the i n d ividual execu tion times tor each call to a routine are not avai l able. This a lso precludes the examination of the cal ling sequence .
•
The results cannot be cross-correlated between programs to provide i n formation about the rela tive calling sequences between programs or across processors.
A real-time execution ana�vsis
tool must operate
with sufficient resulution to measure the execution time
of
a routine that may take
execute.
The in st ru mentati
10 microseconds to
on
should be dynamical�y insertable into the current areas of interest and should be able
to
move to new areas of interest as required-all without stopping and restarting the real-time application.
Parasight: A Solution for Real-time Debugging and Program Analysis
Parasight is an integrated set of rea l-time debugging and analysis tools with a graphical user interface . The tool set consists of a debugger
( Debug ) , a data monitor
( Data Mon
), and a program analysis tool
( Paragraph ) .
The Parasight tool set solves the real - time deficiencies found in d bx, prof, and the other conventional U N I X debug a n d analysis tools used u nder t h e Digital
UNIX operating system . Parasight is able to debug applica tions in either a dynamic (running) or a static (stopped ) state; i t can perform debugging and program execu tion analysis on several programs simultaneously, with out adversely affecting the dynamics of time-critical applications.
Parasight's Foundation
The Parasight tool set features the use of a symbol table, the
/prof tile system, global memory, and scanpoints.
The Symbol Table, .pg File, and /proc File System
Parasight's source of knowledge about the target application is derived from the symbol table and the
.pg fi le. Both are generated at compile time as a result of the
-para special compiler option.
Parasight manipulates target applications by using the
/proc file system services available under the Digital
U N I X operating system. The
/proc ti le system enables
Parasight to control the program flow and to read and write any memorv address i n the target application.
Global Data
Just as the traditional means of debug ging real-time applications depends on global memory regions, Parasight uses the global memory access
Digiral Tt:chnical Journal
Voi . 7 No. 4 1 995 1 03
l
04 concept as th<.: basis For accompl ishing most oF its advanc<.:d capabilities. Parasighr eith<.:r accesses th<.: or uses global memory to acc<.:ss cbta gathcr<.:d t("Jr
Parasight b�· one of its scanpoints .
Sca n po ints
Th<.: Parasight tool s <.: t uses global m e m o r y access whencv<.:r possible t o prm·ick nonintrusivc access to the target appl ication . Certain functions, howev<.:r, r<.:quire access to data that is local to a pro gram. Parasight accesses this data through smal l seg m ents of cod<.: called scanpoin ts.
A scan poi nt is a tl1nction that is dvnamically inserted i nto the target program bv Parasight. The scanpoint function then runs i n the same context as the target program and thus has access to all the local data of the program. The scanpoint function works as an agent for Parasight, gatheri ng data that Parasight does not have direct access to. The Parasight tool set uses two principal types of scanpoints: datamon-scanpoints, which arc used bv
Data Mon to perform local data monitori ng, and sensor-scanpoints, which are used by
Paragraph to perform program execution analysis.
Inserting the scanpoin ts does not require modi�•ing the application's source code or preprocessing the application's object code. The only requirement is to link each program with the special
-para option . This adds a memory buffer to the target program for use by
Parasight. The buffer is benign u ntil used bv Parasight.
Parasight d\'11amicallv inserts scanpoints bv using the
/proc service to build a scanpoint template in the specia l buffer of the target program . This can occur even while the program is running. The template code contains the fu nctionality to
•
•
•
•
•
•
Sa,·e the register state that existed when the pro gram count<.:r \\'aS at the scanpoim insertion location
Set up the arguments to the scu1poinr function , including the register state
Call the scanpoinr fi.mction
Restore the register state
Execute the instruction that ,,·as originally at the insertion location
B ranch back ro the i nstruction fol lowing the inser tion location
Parasight then dynamically alters the template code according to the insertion location and the instruction con rained therein. If the i nstrucrion was a branch con trol instruction , Parasight alters the instruction's dis placement so that the location corresponds to the instruction's nn,· displaced location ,,·ithin the tem plate. All other instructions, including j u mp control instructions, do not require altering and arc simply copied to the new d isplaced location .
Once this code is constructed in the bufter,
Parasight COlll[)lctcs the scan point insertion process by
Digiral Tc(hnic.tl )ourm l
\'ol. 7 l:'\o.
4 1 995 ov<.:rwriting the instruction at the insertion location with a branch to the newly g<.:nerated scanpoint template. The f!xed instruction length o f Digital's
Alpha microprocessors si mplifies this step e normously.
It is important to note that the scanpoint is built bv
Parasight, not the target program. The target program is aftected onlv Lw the tlnal step of the scanpoint inser tion, when Parasight 0\'CniTites the instruction at the insertion location. This design prc,·enrs <.:xcessi,·e inter terence of the target program. Scan points an.: written in highly optimized code to minimize the impact on the target application when thev are executed.
Parasight dvnamically deletes scmpoi ms bv writing back the original i nstruction at the insertion location.
This design al lows Parasight to disable a scmpoint even if the scanpoinr function has not completed .
Meeting Requirements
Parasight has the capabilities required ofdkctive reai time debugging and analvsis tools.
Debugging Running Programs Com e ntional C:\IX de buggers deliberate!\' stop a program '' hen attaching to it, because these tools do not operate on running programs. vVhen Parasight's debugger,
Debug, attaches to a program, there is no impact on the program.
Conve n tional U N I X debuggers refuse to access any data while a program is running, even though global data resides at fixed memorv l ocations that are accessi ble at all times through the
/proc se n ice. The reason tor this limitation of the conventional
U N I X tools is unclear. Parasight's debugger is able to examine and to change the value of any global dara while the program is running or stopped .
Coll\·entional
U N I X tools also refuse to set any breakpoints in a program \\'hile the program is run ning. Agai n , the reason t(x this constraint is un kno\\'11 .
Par:.1sight's debugger is :�ble to i nsert bre:�k.points into running programs, a feature that is val u :�ble in debug ging error conditions in real -time applicJtions.
Debugg ing Mu lti ple Programs
PaLlsight's
Debug,
DataMon, and
Paragraph components constitu te an integrated set of tools capable of \\'orki ng on one or more applications simultaneoush·, as sho\\'n in Figu re
1 . The Parasight main window displavs the programs
( and any child ren th<.:y create ) attached to Parasigbr.
The window a lso provides an easy mechanism to access the Parasight tool for each specific progr:1m.
Monitori ng Va riables Constantly
Parasight's
DataMon tool allows the user to simultaneouslv monito r the val ues of anv local or global variables in one or more stopped or running programs. Parasight constantly monitors the v:-tlues Jnd shows any ch:111ge on the
DataMon display scree n .
DataMon is �1lso capable of displaving th<.: m i n i m u m , maxim u m , :-t nd J\Tragc
69
70
71
72
73
74
7'5
7li
77
78
79
00
81
63
••
65
66
67
68
50
51
52
53
54
55
56
57
511
59
GO
61
62
41
42
43
••
�
46
4
7
97
98
99
100
101
102
103
1Q4
87
88
99
90
91
92
93
94
95
9G
7'J
80
81
92
83
11-1
85
86
105 lOG
107 lOll
109
110 ltl
U2
113
114
115
116
117
UB �
119
128
,.;
47
..
49
5o
51
31
32
3J
3-1
35
36
37
38
39
40
41
42
43
..
45
GO
61
62
63
64
65
66
67
52
53
54
55
56
57
58
59
68
6
9
70
71
72
..
<15
46
..
34
35
3li
37
38
39
40
41
42
43
57
58
59
GO lil
62
63
49
5o
51
52
53
54
55
56
..
65
66
2!i
21>
27
28
29
31)
31
3
2
33
J celD'a accoW'tt
I
.. lnt. iJ
,init(illt to;krMm) dp-)cpunun[ted:f'k.tl"]
• o; inl ar.LusecCint ta:oknun)
I r-eturn ( Unt ) t11'18f'64_uo•c
J lend if
( dp->cpu
ExpressiCM1
.........
I VWIUJ'I h • : 1 ' 1 0 1 5-7 I l jtp->O>IIICUme�J
I unSlgntd lnt tKocUme
•
16J (DI<aJ)
� dp->stoptiM[tukrM.M I
• pt
.. usec( tas-ltllll'f'l) : dp->mcectb•eltHki"'UMI if
= dp-htopUne(taaknuNJ
.. dp->t:tartt.-lt.,..ounh tt!-��:=��::.:�"�11Xt".;Ai\":.t.oa) ) if
( dp-)exectine[tuknl.l\]
< dp-)t�lntinc(taskrutl ) dp-)ftintl,.. ltukrtu��l
• dp-)exectl�te[ted:.ll!.Ml; dp->rraftti..eltasknu"
1 • dp-htoptine[ tllskMmJ - clp-)etartiMIOl
J dp->t.oHinelt.sknur.l
•• dp-)oHoctino[t.ukRW�lJ dp-)pa;o:.cnt.li(tulmutl
]HJ dp-)toskstathasknun)
•
CDHPLETEl
H < taaknu..
== dp•>1uttask ) dp-)donetillleldp-)(t'a.llll]
• pt_usec(tackntM) J
Fig ure
1
P;lr;lsighr's Debugger
W o rkin g "·irh FiH: Tasks Simultaneouslv values attained ror any variable. A scrolling history display along with a time stamp is also ava ilable for solving transient problems.
The variables to be monitored can be selected using the mouse on the Debug browser or entered i nto a dia log box using the kq,board. The Data Mon graphical user interrace has a poi nt-and-edit capability, which a l lows the user to edit the mnemonic data ( i. e . , name, display format, va lue, location, or com ment) directly on the screen . The user can store mnemonic lists on d isk for fast n.:tric\·::d \\'hen requi red . Figure 2 shows a DataMon d isplay screen.
DataMon is able to monitor global data completely and nonintrusivdy using the /proc service and uses a datamon -scanpoint to i mplement local data mon i toring. The daramon-scanpoi nt i s attached to the
Data Mon database, which is a shan:ci memory region connecting all the scan points and the Data Mon display progra m . The datamon-scanpoin rs deposit the val ues of local data into the database for the displav program to show on the screen . Datamon-scanpoints are :�lso used to change the values of local data, depositing the value fl·om the database into the specitled variable.
Data Mon uses the
Debug tool's expression evaluator to parse the required m nemonic to derive the location of the value ro be displayed . This mav include register access for l ocal variables saved on the stack. Multiple m nemonics can be monitored locally at the same location since a datamon-scanpoint function can tra verse a list of mnemon ics to be monitored.
Note that DataMon monitors data asynchronously; therefore, DataMon cannot guarantee to d isplav evcrv
Digital Tcchni(al
Jou rn.1l
Vol . 7 No. 4 1 99S 1 05
DataMon
!:!_elp stllrtime dp->stoptime[ tasknuml d p ) e x ec t i rn e ( ta sk d p - > m i n t i m e[ taskn n um] uml dp->maxtime[ tasknuml dp- > f'r�rf'l t l m e ( t�sknurnJ dp-> passcnt s [ OJ dp- > passcnts [ 1 l d p - > passcn t s [ 2 J d p - > passcnt s [ 3J d p - > p a s s cn t s [ 4 ] l oops dp-> numtasks tasknurn
Dec
Dec
Dec
Dec
Dec
Dec
Dec
Dec
Dec
Dec
Dec
735
4 2
1
6
< 735 <
4 2
1
6
4 2 1 5
4 2 1 5
4 2 1 4
49449067
1 0
4
2064
[ 1 56 4 ]
[ 1 56 4 ]
[ 1 564]
[ 1 564]
[1564]
[ 1 56 4 ]
[ 1 56 4 ]
[
1 5
64
]
[ 1 56 4 ]
[ 1 56 4 ]
[ 1 56 4 ]
[ 1 572]
[ 1 565]
[ 1 573]
**
Global account . c : 60 accou n t , c : 6 0 account . c : 6 0 a c c o u n t , c : 6 0 accou n t , c : 6 0 task3 , c ; 1 02 mai n ( ) exec . c : 1 4 4 mai n ( ) account . c : 51
16�07:0] llp->rrr4mlilut!:[l.,.!lkfllllll) '"'
J�!'i
1&:07:03 dp->lnunlimo(l831<numJ - 7�7 r•:oJ:Ol dp-�ftiUTltlmt(ta'SblumJ
=
79'
1&:07:0-1 dp->lnunlime(l831<numJ - 804 r':OJ:O.J np- ·.•rrarntlmt[ta1t.numJ
=
791
1&:07:0-1 dp- >lnunlime(l831<numJ - 80G n::UI :U·l dp- :-friUTltJme[tasknumJ - /ti:J
16:07:05 dfo - •framtime(l&knumJ
•
700 n:ut:Ju ttp-:-rr.mt�me!tasknumJ - 1�-1 flii:OC�
Ht-!lp
Fig ure 2
The DataMon Display Screen with History Window value that the variables reach . For global data,
Parasight records only the minimum and maximum v a l u e s that
DataM o n sees. For local data, however, the scanpoint keeps track of the minimum, maximum, and average val ues, so these can be guaranteed.
Parasight can also moni tor global data by usin g a datamon-scanpoint to moni tor the value at a particu lar line of code.
On-the-Fly Execution Analysis Paragraph displays static source-code call graphs of the application's programs, i l lustrating the hierarchy of function calls, system calls, and statement-level control flow. Point and -click operations allow the user to quickly view the source code for any program or function, thus simpli
fy
ing the task o f analyzing source code. Figure
3 shows a
Parag raph call graph and browser. lnslnlment VIew Conte><!
�etup
•inc lude '"dp .h" linclude .. exec . h " int perJodc.olc{int
I t�sknun) stdtic un�igned int l.c�s:t._sldrtine[ MRXTASJ(+l] � dp->star·ti"e[ tasknunl
= eut_usE:c{tasknun } ; i f ( dp-)pi)SScnts[tasknun J
> 1 >
I dp->per lod£ tasknunl
= dp-}lotperiod( tasknurtl d p
-> s tart irt e[ tasknun
J
- last_slart 'inc [ task nun l ;
• = dp->per�iodl tasknu" l ; i f ( dp-)period ( t.asknun J
< dp-)niuperiodl tasknu.,) dp-)n.inperiod [ tasknunJ
= dp-)p"riod(t-c�sknuft]; if
( dp-)per iocHtasknunJ
) dp-)naHp�riod( tasknunl dp-)nawperiod [ tasknunl
= dp-)peri.od.(t.l�knurtl �
I� Min p�r.iod
•I
/* HaM period 'f>/ last._st.artlne[t.asknunl
= dp->starti"e[ task nun J ;
Figure 3
Paragraph Ca l l Graph and B rowser
1 06
Digital Technic<! Jomnal
Vol . 7 No. 4
1 99 5
Call graphs are also used to define where to insert i nstru mentation in an application . The i nstrumenta tion is used to perform execution timing anal ysis on a part or the whole of one or more of an application's programs. The instrumentation is inserted dynami cally into a running program, vvithout the need for source-level changes or object code preprocessing and without significantly affecti ng the dynamics of a run ning application . The inserted instrumentation may be deleted or added to at any time.
Paragraph
uses sensor-scanpoints to measure how long a function takes to execute. The sensor scanpoi.nt fu nction is placed at a branch-to-subrou tine i nstructio n . The fu nction takes a time stamp from a nanosecond -resolution timer before and after the i nstruction to note the exact time the fu nction started and ended . The sensor-scanpoints are attached to the
Paragraph
database, a shared region accessible to the sensor-scanpoinrs and
Paragraph.
Data is written into the database each time an i nstrumented function is executed. The results of the i nstrumentation may be viewed immediately, even while the program is running. The graphical view shows each fimction call as it occurred in time. Each program has a different bar, so the user can determine the relative time between fu nctions called in different programs or even across multiple processors in an SMP environment.
The zoom capability may be used to measure time peri ods down to a single microsecond. Figure 4 shows the
Paragraph
graphical d isplay, called Bargraph, and the zoom capability.
Data gathering is continuous until the instru men tation is removed, so new data can be added onto the previous snapshot's view at any time. Multiple
Bargraph windows can be used to recall previously saved timing data to easily compare current results with past results.
The nanosecond-resolution timer used by
Paragraph
is derived from the process control counter ( PC C) register available on all AJpha m icroprocessors. This
32-bit, free-running timer operates at the same rate as the m icroprocessor and therefore provides a
Process
22928
22920
22912
21090
22918
22936
22902
I
�
•
•
I m �
I •
I
�
•
I u
, I
I
•
I
I
D •
0 w
•
M
•
D
I
Ill � I
rn
I
I
� I Ill � I
� m
I
•
I
I
u
�
I
I
I I
�
� 0 w I
I I I
� I
D
I
I m rn
0 f!le ?IJom !!_uffer �w getUp
Qwsor: 44.281 s Mal1<e!': 44.28 1 •
----------------
0
Oifferet�Ce: 1 D7us
------------------
Process
22928
22920 account() ta:.kl.c:l 00
22912
2 1090
22918
22936
22902
1.J0rk40
I
Figure
4
The
Paragraph
Graphical Displ ay, Bargraph, Showing Zoom Capabil ity work2( )
I work.J( )
Digital Technical Jounul
Vol . 7 No. 4 1995 107
3.6-nanosecond-resolution timer on a 2 75 - M H z
Alpha C P U . U n fortunately, since i t i s only a 32 bit timer, it wraps every 1 5 .6 seconds. Parasight keeps track of the wrap count to create a 64-bit timer that allows problem-rl·ee ti ming for more than 2,000 ve:�rsl
Adverse Effects
Although, ideally, the Parasight tool set should be com pletely nonintrusive and thus not afkct the appl ication in any way, such operation is not com pletely achievable for all fu nctions. Capabilities such
JS inspecting
( Debug) and monitOring
(DataMon) global variables req uire no intrusion to implement; howe\'er, mon i toring local variables and analvzing program exe cution do require a small amou nt of intrusion .
\Vhile most real-time applications cannot tolerate exceeding the time anilable for the completion of the task, t hev do have some spare time availa ble :�fter completing the task. Without this spare time, the risk of exceeding the deadline bef()re program comp letion wou ld be too great. This spare time can be used judi ciously r()f the mildly intrusive functions of Parasight.
Summary
This paper discusses several capabilities required to effectively debug and analyze real-time applications.
These capabilities i nclude debugging of running pro grams, constant monitoring ofvariables, and on-the-fly executjon analysis. The paper also details some of the problems associated with conventional
U N I X tools, such as the inability to debug running programs, the adverse effects on target progr:�ms, the erroneous pro fi l ing results, and the cum bersome operation . Encore
Computer Corporation's Parasight tool set ofkrs a solu tion to these diffic u l t problems. The p:�per describes the methodologv behind the product :�nd the capabil ities that make Parasight an im·aluable tool f(x debugging and analyzing real-rime applications.
Acknowledgments
The authors would l i ke to acknowledge the eft(Jrts of the following Parasight team members for their contri butions to the product: R.aghuvccr Chak.ra,·art h i ,
Di leep Katta, Carlos Gonzalez, Deborah G ri mstead , and Ken Shafter.
General References
Z. Ar,l l , ! . Germer, and C . Slufkr, " Hticienr Debugging
Prim itin:s tor
Mu ltiprocessors" ( fon Lauderdale,
HL:
Encore Computer CorporJtion,
1 989 ).
DhC 0.'!1-/7 Pmgranuner�<;
Guide.
Section 6 (MavnJrd ,
Mass.: Digital Equipment Corporation, August
1994 ).
Biographies
Michael Palmer
Michael Palmer is
J principal member of Encore Computer
Corporation's technical StJff and has led the Parasighr team for the past three years. Prio1· to joining En core in
199 1 ,
Mike worked to 1· several major tl igh t simulation ,·end ors throup;hour the worl d , a<hancing from computer s\·stems engineer to lead sofr"·are en gineer tor a 550 mil lion, du,\1dome tactical tighter simulator. H e has used his real-rime si mu brion backgrou nd to mold Parasighr into a lead i ng tool set tclr real - time dc,·clopmcnt. Mike holds a
B .Sc.
( Hono1·s) i n electronics from Newcastle Polytechnic,
Ncwclsrle upon Tyne, E ngbnd .
Jeffrey M. Russo
JctfRusso ha s been emplovcd by fi\M since June
1 99S.
H e i s
'lll
Ad,·ison· Programmer wmking a s a ream lc,\<kl· to r rhe
OS/2 operating S\'Stc m . Prior to joining
IBM, )dl
\\·orkcd ar Encore Computer Corpor.nion to1· 1 0 ,·c.J I·s, ad,·,111cing ti·om rhe position of soti:" '1rc engineer to t h<lt of Senior Section Mana�er responsi ble tor se,·cral re,l l -timc sofuv;1rc groups. H e lus sign itica nt experience with !·cal rime, microkcrncJ- b,\sed opn,uing systems, as well JS with rhc accompa111·i ng critic1l, real - rime tool set. jeff c'\rned a R.S. in computer engineering ti·orn the U ni,·crsif\· of
Florid,! in
! 9R S .
I 08
Digital Tccllllic.JI
Journal
Vol.
7
N<>.
4 1 995
Call for AL1thors from Digital Press
DigitJ I Press is an imprint of Butterworth-Heinemann, a major international pub lisher of professional books and a member of the Reed Elsevier group. Digital
Press is the authorized p u blisher for Digi tal Equipment Corporatio n : The nvo companies arc working in partnership to identi�' and publish new books under the
D igital Press imprint and create opportunities for authors to publish their work.
Digital Press is committed to publ ishing high-quality books on a wide variety of subjects. \Ve wou ld like to hear tl·om you if you are writi ng or thinking about writing a book.
Contact: Mike Cash, Digital Press Manager, or
Liz McCarthy, Assistant Editor
D I G fTA L P RESS
3 1 3 Washington Street
Newton, MA 02 1 58- 1 626
U .S.A.
Tel : ( 6 1 7 ) 928-2649, Fax : (6 1 7 ) 928-2640
E-mai l : Mike.Casb@BHe i n . reJ .co.uk or
LizMc@world .std .com
mamaomo
,.
ISSN
0898-90 1X
Printed i n
LI S.A. l:Y U002 F-TJIY6 3 1 4 1 8.S
Copwight Digit.1l Lqui pmL·rn C :mpor,nion.
Download
Advertisement
Key features
Natural-sounding text-to-speech synthesis
Versatile color-map rendering
High availability for Digital UNIX applications
Real-time debugging and analysis tools
Software reliability and data integrity
Support for multiple platforms and operating systems
Integration with computer's audio subsystem
User-generated dictionaries
Multimedia audio formats
Telephony application support
Frequently asked questions
The paper discusses various text-to-speech technologies, including concatenative synthesis, formant synthesis, and diphone synthesis.
DECtalk Software consists of eight processing threads: text queueing, command parser, letter-to-sound converter, phonetic and prosodic processor, vocal tract model (VTM), audio, synchronization, and timer.
The API allows applications to play speech, write speech samples to files, and write samples to buffers.