DISPLAY KEYBOARD KEYBOARD
US 20030151628A1
(19) United States
(12) Patent Application Publication (10) Pub. No.: US 2003/0151628 A1
(43) Pub. Date:
Salter
(54) INTERACTIVE GAME PROVIDING
Aug. 14, 2003
Publication Classi?cation
INSTRUCTION IN MUSICAL NOTATION
AND IN LEARNING AN INSTRUMENT
(51)
Int. Cl? .
(52)
Us. 01. ............................................................ ..345/773
.... .. G09G 5/00
(76) Inventor: Hal Christopher Salter, Seffner, FL
(57)
(Us)
ABSTRACT
A musical keyboard is connected to a computer. The com
Correspondence Address:
puter implements a graphical user interface for teaching
MCDERMOTT, WILL & EMERY
600 13th Street, NW.
users to play the musical instrument. A computer readable
music ?le, such as a MIDI ?le, is used to drive the creation
Washington, DC 20005-3096 (US)
of game objects that travel from a point of origination along
(21) Appl. No.:
10/273,353
a path toWard a key of a virtual keyboard. In one form, When
a user presses a key of the musical keyboard Within a certain
(22) Filed:
Oct. 18, 2002
time WindoW of arrival of the game object at the correspond
ing key of the virtual keyboard, the user is aWarded With an
audio presentation, a visual presentation and/or With game
points. In a more structured learning mode, the game can be
Related US. Application Data
(60)
Provisional application No. 60/347,554, ?led on Oct.
20, 2001.
played With selectable, progressively more dif?cult chal
lenges that the user masters on the road to pro?ciency.
DISPLAY
CPU
KEYBOARD
/
MUSICAL
KEYBOARD
150
Patent Application Publication Aug. 14, 2003 Sheet 1 0f 12
US 2003/0151628 A1
DISPLAY
r-_
CPU
4
/
100
KEYBOARD
MUSICAL
FIG. 1
Patent Application Publication Aug. 14, 2003 Sheet 2 0f 12
US 2003/0151628 A1
TIME = MIDICLOCK
FUTlURE
TIME = MIDICLOCK
MINUS TRAVERSA
TIME
FIG. 2
Patent Application Publication Aug. 14, 2003 Sheet 3 0f 12
US 2003/0151628 A1
Patent Application Publication Aug. 14, 2003 Sheet 4 0f 12
US 2003/0151628 A1
330
/
LIBRARY
OF
OBJECTS
300
320
310
GRAPHICS
ENGINE
350
/
MIDI
FILES
MIDI
KEYBOARD
INTER
PRETER
GAME
OBJECT
GEN.
DISPLAY
PARA
METERS
AUDIO
OUTPUT
150
FIG. 3
AUDIO
ENGINE
360
370
Patent Application Publication Aug. 14, 2003 Sheet 5 0f 12
400
/
US 2003/0151628 A1
<---Chunk--->
type
MIDI MThd
FHI-E MTrk
length
Data
6
<format> |T<tracks>
<Iength> ‘
<division> y\410
<delta_time><event>
-\
.
420
MTrk <Iength>
<delta_tlme><event>
\420
FIG. 4
Header Chunk
Chunk Type
length
Data
4bytes
4bytes
<--Ien9fh(=6byte$)-->
(ascii)
(32-bitbinary)
16-bit
16-bit
16-bit
MThd
<Iength>
<format>
<tracks>
<division>
FIG. 5
Patent Application Publication Aug. 14, 2003 Sheet 6 0f 12
Track Chunk
type
length
4 bytes
4 bytes
<-- length bytes -->
(32-bit binary)
(binary data)
(ascii)
MTrk
_
Data
‘
<length>
<de!ta_time><event>
FIG. 6
US 2003/0151628 A1
Patent Application Publication Aug. 14, 2003 Sheet 7 0f 12
US 2003/0151628 A1
FF <type> <length> <data>
where:
<t e>
is a single byte, specifying the type of meta-event.
The possible range is 00-7F. Not all values in this range are de?ned, but programs must be
able to cope with (ie ignore) unexpected values by examining the length and skipping over
the data portion.
<length>
is the number of bytes of <data> following.
This is a variable length quantity
0 is a valid <length>
<data>
zero or more bytes of data
FIG. 7
Patent Application Publication Aug. 14, 2003 Sheet 8 0f 12
800
Mtrk
oo
leng
820-1
US 2003/0151628 A1
820-2
AA
<delta time><>event>----<de|ta time><>event>
FIG. 8
Patent Application Publication Aug. 14, 2003 Sheet 9 0f 12
US 2003/0151628 A1
Three note sequence--C followed by E followed by G
Dellaqvime
Message
Dellalvime
Message
00
92
3C
44
08
82
3C
40 '
10
92
40
40
18
82
40
40
20
92
43
3E
28
82
43
3E
|
|
l
|
|
l
ON
NOTE VELOCITY
OFF
k
NOTE VELOCITY
J
FIG. 9
Patent Application Publication Aug. 14, 2003 Sheet 10 0f 12
US 2003/0151628 A1
e \ as\
$maséE20m5m02zo,50?w3m2l1g8
52gx%bém
Ass2%22vA232$2.5
mEvSzwo3b?x5m
GEof
Patent Application Publication Aug. 14, 2003 Sheet 11 0f 12
US 2003/0151628 A1
SPLASH /“°°
SCREEN
—-’
1140
V
———\
WELCOME /
1110
SCREEN
‘ SETUP
GAME
1120
QUICK
1100
1110
1100
PLAY
/
/
/
GAME
PLAY
FREEZE
MENU
CAREER
\
MODE
1100
1150
STARTING
OPTIONS
Patent Application Publication Aug. 14, 2003 Sheet 12 0f 12
112
/
106
110
)3‘)
l """" ‘7L “““““ ‘7L """" '7’ "" ":
i
NNN
MSPLAY <: 5
ROM
MEMORY
N4
INPUTDEVICE <:
108
US 2003/0151628 A1
ii
EL)
STORAGE
i
DEVICE
5
SERVER
3 J12’ @
lNTERNET
5:
BUS
ISP
116
/
126
5
CURSOR <1 i
CONTROL
128
5
122
i
PROCESSOR
COMMUNICATION
ENETWORK
INTERFACE
; LINK
i ______ ",1 ____________ __/_ ________
104
118
LOCAL
NETWORK
120
16;]
HOST
/
Aug. 14, 2003
US 2003/0151628 A1
INTERACTIVE GAME PROVIDING
INSTRUCTION IN MUSICAL NOTATION AND IN
LEARNING AN INSTRUMENT
CROSS REFERENCE TO RELATED
APPLICATIONS
[0001] This application claims priority from and is related
to US. Provisional Application Serial No. 60/347,554, ?led
Oct. 20, 2001, entitled “AN INTERACTIVE GAME PRO
VIDING
INSTRUCTION
IN
READING
MUSICAL
NOTATION”, by inventors Hal C. Salter. The contents of the
provisional application are hereby incorporated herein by
reference in their entirety.
methods, but piano and music tutorials have not been as
successful, due to several signi?cant problems inherent With
the subject.
[0007] A major problem With these prior art learning
systems is that they fail to take into account several impor
tant differences betWeen learning to type and learning to play
music. The ?rst and most obvious difference is that musical
notes cannot merely be played in a correct sequence, but
must be played in a rhythmic pattern in relation to the notes
before and after it. The correct timing of the notes is one of
the most difficult aspects to teach in music, by any tech
nique; and up to noW feW applications have been able to
adequately address this challenge, and then only in isolation
from other concepts. The second challenge is that most
COPYRIGHT NOTICE
programs start With musical notation to teach a song to the
[0002] At least a portion of the disclosure of this patent
document contains material that is subject to Copyright
protection. The Copyright oWner has no objection to the
facsimile reproduction by any one of the patent disclosure,
user. Reading and understanding musical notation is a bit
like reading BeoWolf in the Old English, and it’s roots are
just as old. The ?ve line staff originally represented a monk’s
hand in the Middle Ages, and has been adapted to deal With
all the musical innovations since.
as it appears in the Patent and Trademark Of?ce patent ?les
or records, but otherWise reserves all Copyright rights What
soever.
CROSS REFERENCE TO CD-ROM APPENDIX
[0003] This application contains an Appendix comprising
(1) CD-ROM containing speci?cations of computer pro
grams used to carry out the invention disclosed herein. The
contents of the CD-ROM are described in more detail in
paper AppendixAattached to this document. The contents of
the CD-ROM are incorporated into this disclosure by ref
erence in their entirety as if they Were set forth completely
Within the text of this application.
[0008] There is also the problem of a vertically oriented
notational system originally designed to convey pitch for
voice (hence the concepts of higher and loWer). But on the
piano, pitch is experienced horiZontally (i.e. perpendicularly
to the notation—left to right instead of loWer to higher). This
difference, While conceptually easy, is a dif?cult transition
for students to make, both visually and experientially. The
combined dif?culty is such that feWer than 20 percent of all
piano players are estimated to be able to read music ?uently.
[0009]
For the novice player, then, there are at least four
additional challenges to learning the piano compared to
learning to type. They must learn the keys to the piano, they
must learn musical notation, the notation is perpendicular to
FIELD OF THE INVENTION
[0004]
The present invention relates generally to an inter
active game, and more particularly to a process for inter
preting or compiling MIDI or other standard music ?les into
their piano, and they must hit the right key(s) at the right
time. The frustration at these simultaneous challenges
causes most users to quit, and makes the task seems more
like Work than play.
an interactive game that instructs its user in musical notation
[0010] Examples of prior patents in the general area of
While playing the game.
music instruction are outlined beloW. Each of these refer
ences is incorporated by reference in their entirety.
BACKGROUND
[0005]
MIDI technology has been a stable source of piano
[0011] US. Pat. No. 4,416,182 to Wise, et al. discloses a
keyboard teaching device for the self-instruction of a student
and music sequencing programs for editing, printing, play
of keyboard musical instruments. They system enables the
ing and composing music for almost tWenty years. Some
early, relatively primitive music and piano tutorial programs,
student to correlate the positions of the keys on a musical
instrument keyboard With the positions of the notes on a
as Well as some games, have resulted from this technology.
musical scale. A keyboard having a plurality of keys corre
For example, some types of computer aided instruction,
including piano instruction, have been used in the home
sponding to the notes of a musical scale generate a ?rst set
of control signals in an initialiZation or set-up mode and
key-note correlation signals in an instruction or game mode.
There is at least one storage element storing a predetermined
combination of logical signals providing a source for a
market for several years.
[0006]
In the area of music instruction, the computer has
been used, for the most part, to provide a student With an
interactive vieW of musical notation, Where the student can
press a key on the accompanying electronic piano and it Will
light up the note, or the note Will light up and a draWing Will
indicate Which key he should press. Other systems have a
series of lights or ?ngering illustrations to attempt to com
municate to the user Which key should be played next. These
programs and systems, While alloWing the student to prac
tice and obtain feedback, generally fail to involve musical
novices suf?ciently, especially With regard to rhythm. Sev
eral typing tutorials have enjoyed great success using such
pseudo random sequence of one or more notes over a
predetermined range in the game mode. The system gener
ates a second set of control signals in response to the
actuation of one or more keys in either game mode and a
mechanism for generating audio tones and displaying video
images in accordance With the ?rst and second sets of
control signals enables a student to visually and audibly
check his or her selection of one or more keys.
[0012]
US. Pat. No. 5,183,398 to Monte, et al. discloses
an apparatus and method for instruction of a student Which
Aug. 14, 2003
US 2003/0151628 A1
includes interactive guidance of the student through a series
of lesson frames. This disclosure provides the student With
a keyboard having a plurality of keys corresponding to the
notes of a musical scale and generates a key relation signal
in response to each depressed or released key. A video
display and an audio tone generator associated thereWith
enables the student to respond to the visually displayed
images and audio tones by selecting one or more of the keys
on the keyboard. The student is presented With a lesson
frame representing an instructional activity requiring a
response by the student on the keyboard. The student
keyboard response is compared With a performance standard
and an absolute performance evaluation result in generated.
The absolute performance evaluation result is compared
With an acceptable achievement level for the particular
instructional activity and a neXt frame selection signal is
generated. A neXt frame is selected for presentation to the
student based upon the neXt frame selection signal.
[0013] Us. Pat. No. 4,997,374 to Simone discloses a
teaching device that includes a changeable tWo channel
prerecorded program source, and a console unit including a
Work booklet. The ?rst channel of the program source
includes an audio program comprising a series of spoken
Words Which are audibly reproduced by the console unit and
?nder WindoW Which displays an alphanumeric code Which
corresponds to a like code appearing neXt to the score of
music to be played. A foot pedal advancing mechanism is
used Whereby the user can control the speed Which the lights
display the keys to be struck, using the Foot Pedal to
advance one action. The display also can shoW the music to
be played at a tempo set by the user automatically changing
from action to action Without using the pedal. The panel
articulates so that it may be stretched in one or more places
so that it can ?t over various dimensions of keyboards
Without interfering With keys to be played.
[0017] US. Pat. No. 4,331,062 to Rogers discloses an
apparatus for visually displaying music notes on a note
display panel mounted on an electronic piano With a support
arm. The electronic piano has a keyboard electrically
coupled to an electronic circuit operable to produce an audio
output in accordance With the depression of one or more
keys. The arm is rotatable in a mount attached to the piano
for movement about a ?rst upright aXis. A ?rst motion
limiting unit attached to the mount and arm limits the
rotation of the arm about the piano. A second motion
limiting unit attached to the panel and this arm limits
rotation of the panel about an upright aXis relative to the arm.
A modi?ed structure has the arm ?Xed to the piano. The
are operative for actuating lights adjacent prespeci?ed Words
panel has grand staff indicia coordinated With vertically
disposed ?rst light mechanisms corresponding to chord note
in the Work booklet. The operation of the console unit is
information With a second light mechanism diagonally cor
coordinated With the audio program so that lights are actu
ated adjacent to the Words in the Work booklet as the same
responding to ascending note scale information. Akeyboard
representation is located beloW the staff indicia. Athird light
mechanism associated With the piano key indicia of the
keyboard representation is coordinated With the second light
the second channel includes a series of control signals Which
Words are audibly reproduced by a console unit. One
embodiment of the device further includes a plurality of
depressible user response buttons on the console unit for
indicating responses to questions presented in the audio
mechanism to provide visual information as to the keys
depressed on the electronic piano. An electric circuit having
program.
on-off sWitches electrically couples the electronic piano With
[0014]
the ?rst, second, and third lights so that the lights can be
selectively operated. The electronic circuit has a sWitch
Us. Pat. No. 4,781,099 to Koike discloses a musi
cal quiZ apparatus that presents a question chord in sound
and a trainee ansWers by depressing the keys of the chord
constituting notes on the keyboard. The apparatus is capable
of generating a plurality of different chord data respectively
representing chords, and generates a question chord datum
assembly having a plurality of key signature sWitches oper
able to coordinate the ?rst, second, and third lights With the
piano keyboard in accordance With the key signature of the
music that is played.
one at a time more or less randomly selected from among
[0018]
those different chord data and produces sounds of notes
Which constitutes a chord designated by the question chord
tronic piano having a keyboard and an electronic piano
data. When the ansWer is correct, points are added up and a
device to provide a video note display concurrently With the
depression of one or more keys. Akeyboard representation
located adjacent the screen of the CRT device is associated
With lights used to indicate the key or keys that are played.
neXt question chord is presented.
[0015]
Us. Pat. No. 5,392,682 to McCartney-Hoy dis
closes a computeriZed musical keyboard and a method for
using same to play or to learn to play a piano. The comput
eriZed musical keyboard includes a piano keyboard con
nected to a computer. The computer is programmed to select
from a music module a piece of music to be played on the
piano and to generate a signal indicating the proper keys to
be played, the correct sequence in Which the keys are to be
played, and the hand and ?nger to be used in striking each
key, in order to play on the piano the piece of music selected.
[0016] Us. Pat. No. 5,107,743 to Decker discloses a piano
teaching aid having a panel designed to ?t over the keys on
an eXisting keyboard so that lights mounted on a panel
having more than one color or shape may be located directly
above the piano keys to be played. The lights have more than
one color or shape in order to distinguish the hand Which the
user Will use to play the piano. The panel also includes a
US. Pat. No. 4,366,741 to Titus discloses an elec
circuit connected to a micro-processor used to control a CRT
Manually operated controls cooperate With the micro-pro
cessor to alloW the back clearing of the screen one note at a
time, remove all the notes, retain all the notes, indicate sharp
or ?at mode of each note, and indicate the duration that a key
is depressed by elongating the note on the screen. A metro
nome unit is used With the micro-processor to provide a
visual beat marker on the screen that sequentially moves
across the screen. Amovable frame connects the CRT device
to the piano.
[0019]
US. Pat. No. 5,864,868 to Contois discloses a
computer system and method for controlling a media play
ing device. The system provides a user interface for alloWing
a user access to media pieces stored in a media database. The
interface is also for controlling a media playing device, like
a player piano or movie playing video device, that is coupled
to the computer to play the accessed or selected piece of
Aug. 14, 2003
US 2003/0151628 A1
media. In one embodiment there is a computer interface that
allows a user to display only music that relates to a selected
a song, rising from near the bottom of the screen toWard the
virtual piano keyboard, and their vertical relationship to
category, like jaZZ or classical music. Another embodiment
each other shall be a representation of relative musical time.
alloWs the user to direct the media playing device to auto
As these game objects approach the virtual piano keyboard,
matically play selected music pieces that are related to a
the corresponding piano key along that path is visually
selected music category. Another embodiment alloWs a user
obvious, and the objective of the player is to hit the
corresponding key on a musical keyboard at precisely the
time the game object is located Within a prede?ned hit
WindoW vis a vis the virtual keyboard. If the player strikes
to direct the media playing device to automatically play
selected music pieces that are related to the selected music
composer or artist.
[0020]
US. Pat. No. 6,204,441 to Asahi et al. discloses
techniques for displaying musical information and particu
larly for visually displaying musical notes, beats and tempos
the correct key at the correct time, audio and visual feedback
Will reWard the player.
[0029]
The player’s main task in the invention is to hit the
using personal computers or game devices that run musical
softWare programs. The disclosure teaches the use of dif
corresponding piano key for as of the many moving objects
as possible at the right time, repeating the challenge until
ferent colors and different brightnesses to distinguish certain
they have reached a certain percentage of correctness and
move up to the next level of complexity, and in doing so,
subconsciously learn to play a song, and in the later levels,
to recogniZe and read the sheet music.
types of musical notation. It has a display screen Which
shoWs both base and treble clefs as Well as the keyboard and
timing indication.
[0021] US. Pat. No. 6,388,181 to Moe shoWs computer
graphics animation, used With a live video interactive
method for playing keyboard music While the user guides his
?ngers to the keys targeted by the animation With each key
to be struck Within one beat of time is designated by a
colored “sprite”.
[0022]
US. Pat. No. 6,066,791 to Renard et al. shoWs a
system for instructing the playing of a musical instrument,
displaying an image on the display device, and instructing
the student to focus on the image While preferably using a
musical instrument to play the notes on the staff.
[0023]
US. Pat. No. 5,540,132 to Hale shoWs techniques
for teaching musical notation to children. Each note is
associated With the distinctly identi?able color Which is then
associated With an object Which naturally occurs in this
color. They utiliZe cartoon characters Which apparently
enhances association Within the mind of the child.
[0024]
US. Pat. No. 6,337,433 to Nishimoto et al. shoWs
an electronic musical instrument having performance guid
ance function, performance guidance method, and storage
BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The objects, features and advantages of the system
of the present invention Will be apparent from the folloWing
description in Which:
[0031] FIG. 1 is a block diagram of a hardWare architec
ture for carrying out some aspects of the invention.
[0032] FIG. 2 is a diagram shoWing a virtual keyboard,
game object trajectories and an exemplary relationship of
those to a treble clef of a standard musical notation.
[0033]
FIG. 3 shoWs a block diagram of exemplary hard
Ware and softWare modules for carrying out the invention.
[0034]
FIG. 4 illustrates the format of a MIDI ?le.
[0035]
FIG. 5 illustrates the format of a header chunk of
a MIDI ?le.
[0036] FIG. 6 illustrates the format of a track chunk of a
MIDI ?le.
[0037]
FIG. 7 illustrates the format of META events from
medium storing a program therefore With a plurality of
a MIDI ?le.
display devices arranged in association With the perfor
mance operating elements, respectively, each comprising a
[0038]
pair of display elements corresponding to left and right
hands of the player, respectively.
[0025] US. Pat. No. 6,284,961 to Kimmel, Jr. shoWs a
system of musical notes With the notes being associated With
a color and utiliZes stickers for application to the keys of the
musical instrument to correspond to the colors of the note
Which it plays.
SUMMARY OF THE INVENTION
[0026] The invention is related to a hand-eye coordination
game Which has the purpose of teaching one hoW to read
musical notation and hoW to play a musical instrument such
as the piano.
[0027]
FIG. 8 is an example of a portion of a MTrk chunk
of MIDI data together With exemplary semantics.
[0039]
FIG. 9 illustrates an exemplary How of data illus
trating MIDI messages for the sequential playing of three
notes.
[0040] FIG. 10 is a How chart of an exemplary process for
interpreting the MIDI data stream of FIG. 5 for generating
game objects for display.
[0041]
FIG. 11 is a block diagram shoWing an exemplary
arrangement of game modules in accordance With one aspect
of the invention.
[0042] FIG. 12 is a block diagram of an exemplary
computer that can be used to implement various aspects of
the invention.
The invention simpli?es musical notation into a
grid, With pitch moving left (loWer) to right (higher), parallel
With the piano keyboard layout, and the dimension of time
moving vertically.
[0028] In one embodiment, in the beginning phases of the
game, the player Will see game objects, representing notes of
DETAILED DESCRIPTION OF THE
INVENTION
[0043] FIG. 1 is a block diagram of a hardWare architec
ture for carrying out some aspects of the invention. FIG. 1
illustrates a computer 100 comprising a processing element
Aug. 14, 2003
US 2003/0151628 A1
(CPU) and a display. A typical implementation Would be a
personal computer commercially available from a number of
board Which interfaces With a virtual keyboard. The game
detects When the user presses the correct key at the proper
sources. Details of the internals of computer 100 and its use
in a netWork environment in FIG. 12.
moment.
[0044] Apiano keyboard, 150, preferably having the capa
[0049]
FIG. 3 shoWs a block diagram of exemplary hard
Ware and softWare modules for carrying out various aspects
bility of generating controlling data in accordance With the
of the invention. Although this particular embodiment of the
MIDI standard is connected to the computer through the user
standard MIDI interface.
invention is described With respect to the use of MIDI ?les
and music formatted in accordance With a MIDI standard,
other formats for music are Well knoWn and can be utiliZed
in alternative embodiments of the invention.
[0045]
FIG. 2 is a diagram shoWing an exemplary virtual
piano keyboard, game object trajectories and an exemplary
relationship of those to a treble clef of standard musical
notation. Avirtual keyboard is an object that is shoWn on the
screen of the computer’s display as part of a graphical user
interface. The virtual keyboard Will comprise a plurality of
individual keys 200i corresponding to the White keys of a
piano and a plurality of keys 200j corresponding to the black
keys of the piano. Thus, in this embodiment, the virtual
keyboard represents the standard arrangement of keys on a
keyboard such as a piano. Also associated With each key of
the virtual keyboard is a trajectory 210i corresponding, in
this example, to only the White keys of the piano keyboard.
One of the White keys and an associated trajectory corre
spond to each of the notes of an octave of musical notation,
in this case, starting With middle C on the left and progress
ing up the musical scale until reaching the C above middle
C. Trajectories shoWn in bold correspond to those note found
[0050] In this exemplary embodiment, one or more ?les
300 formatted in a MIDI format are stored in, for example,
a library, or doWnloaded in real time and fed to interpreter
310 Where the MIDI data ?le is interpreted and translated in
to commands Which drive the game object generation mod
ule 320 Which causes the display of game objects on the
graphical user interface on the display screen and provides
audio output 370 to the speakers of the computer. Conve
niently, a graphics engine 350 and an audio engine 360
simplify the translation of game objects into audio and
visual components driving the visual display and the audio
output. The game object generator has access to a library of
objects 330 and to a set of game control parameters 340.
Each of these modules are described in more detail herein
after.
on a line of the treble clef in this example.
[0051] FIG. 4 illustrates the format of a MIDI ?le. As
noted above, the MIDI ?le format is utiliZed for purposes of
[0046]
illustrating the invention, but the invention is not limited
hereto. An example of another ?le suitable for carrying out
From this vieW, it is clear that pitch increases as
one progresses up the musical scale from left to right as
shoWn in this vieW. HoWever, in standard musical notation,
the invention Would be that of MPEG-4.
middle C begins on the line beloW the ?rst line of the
standard treble clef shoWn in FIG. 2 and the C above middle
C is shoWn in the top most space of the standard treble clef.
Thus, musical notation progresses in pitch on treble clef
[0052] An understanding of a MIDI ?le and the operation
from bottom to top starting With loW pitch value and
increasing to higher pitch value. HoWever, When vieWed
from a keyboard, pitch increases from left to right rather
than from bottom to top. As noted above, this difference of
representation creates at least one factor of dif?culty in
learning to play the piano.
[0047] As Will be described more hereinafter, game
objects are introduced at a location opposite the virtual
keyboard along the trajectories 210i. Game objects then
of the MIDI protocol Will greatly simplify understanding of
the invention.
[0053]
The MIDI File Format
[0054] MIDI ?les are structured into chunks. Each chunk
consists of a 4-byte chunk type, a 4-byte indication of length
indicating of the length of the bytes contained in the data
?eld.
[0055] There are tWo types of chunks, namely header
chunks (Which have a chunk type of “MThd” (410)) and
track chunks Which have a chunk type of “MTrk” (420).
progress in animated fashion along the trajectory from near
the bottom to the point Where the game object encounters the
key of the virtual keyboard 200i. There is associated With
[0056]
each of the trajectories a traversal time so that if three notes
[0057] FIG. 5 illustrates the format of a header chunk of
a MIDI ?le. The header chunk comprises three ?elds,
Were introduced sequentially one after the other, they Would
become visible at the bottom of the trajectories appearing in
a sequence, and progressing for the duration of the traversal
time from bottom to top Where they intersect their respective
keys at respective instants of time displaced from one
another.
A MIDI ?le consists of a single header chunk
folloWed by one or more track chunks.
namely, chunk type, length and data. For a header chunk, the
chunk type is MThd. The length ?eld contains the length in
bytes of the chunk data part. The “format” portion of the data
interface. As the user plays the game, the user develops a
?eld includes the MIDI ?le format Which can be only
formats 0, 1 and 2. The “tracks” portion of the data ?eld as
a binary number indicating the number of track chunks
contained in the MIDI ?le. The “division” portion of the data
?eld of the header chunk de?nes the default unit of “delta
time” for the MIDI ?le. If the most signi?cant bit of the
“division” ?eld is a 0, the remaining 15 bits indicate the
number of “ticks per quarter note” to be utiliZed in repre
certain facility for mapping the visual expression of the
senting and reproducing the music. If the most signi?cant bit
notes from a musical ?le Which drive the generation of the
is a logical 1, then there are tWo components indicated by the
remaining 15 bits. Bits 8-14 Would indicate the number of
[0048]
One object of the interactive game is for a student
to press the key of a (preferably MIDI) keyboard substan
tially at the same instants that the game objects intersects
they key of the virtual keyboard on the graphical user
game objects into actual ?ngering positions on a real key
Aug. 14, 2003
US 2003/0151628 A1
frames per second (indicated as a negative number) and the
least signi?cant 8 bits represents the number of ticks per
SMTPE frame.
[0058]
MIDI ?les come in three variations. Format 0
contains a single track. Format 1 contains one or more tracks
Which are all played simultaneously. Format 2 contains one
or more independent tracks Which can be or are played
independently of the others.
[0059]
FIG. 6 represents the format of a track chunk of a
MIDI ?le. It comprises a chunk type Which, by de?nition is
a MTrk type and a length ?eld Which indicates the length of
the data portion in the track chunk. The data portion of a
track chunk comprises tWo elements. The ?rst is a “delta
_time” portion and “event” portion. delta_time is the num
ber of “ticks” from the previous event and is represented as
a variable length quantity. There are three types of events
that are de?ned Within the standard. They are a “MIDI
event”, a “SYSEX event” and a “META event”. There are no
explicit delimiters betWeen the “delta_time” and “event”
instances. This is possible because both ?elds have clearly
[0069] A marker event marks the signi?cant point in the
sequence such as the beginning of a verse.
[0070] A que point is utiliZed to include queues for events
happening on-stage, such as “curtain rises”, “exit”, and the
like.
[0071] An end of track event must be utiliZed to give the
track a clearly de?ned length. This is essential if the track is
looped or concatenated With another track.
[0072] A set tempo event sets the tempo in microseconds
per quarter note. This means a change in the unit-length of
the delta_time tick. The default tempo is a 120 beats per
minute. A SMTPE offset event speci?es the SMTPE time at
Which the track is to start. A time signature representing the
standard time signature for a piece of music, such as 3/4 or
6/8 or 2/2. Akey signature event can specify the number of
sharps or Hats and a major or minor ?ag. Finally, a sequence
speci?c META event alloWs a manufacture to incorporate
sequencer speci?c directives into a MIDI ?le.
[0073]
FIG. 7 illustrates the format of META events from
de?ned lengths.
a MIDI ?le.
[0060] A MIDI event has any MIDI channel message.
These include channel voice messages and channel mode
messages. Messages other than MIDI channel messages to
chunk of MIDI data together With exemplary semantics. As
be included in a MIDI ?le can utiliZe the SYSEX event.
[0061] Most system exclusive messages are quite simple
and are sent as a single packet of bytes, starting With F0 and
ending With F7. HoWever, some system exclusive messages
are used to control device parameters in real time. TWo
different types of SYSEX events are de?ned to accommo
date the different usages.
[0062] META events are used for things like track-names,
lyrics and que points, Which don’t result in MIDI messages
[0074]
FIG. 8 is an example of a portion of an MTrk
shoWn in FIG. 8, an MTrk chunk is introduced With an
MTrk identi?er 800 folloWed by a length ?eld 810. The
plurality of ordered pairs of delta_time and event ?elds 820-l
through 820-j then folloW Which represent the individual
MIDI messages associated With the MTrk chunk.
[0075] The MIDI protocol consists of messages Which are
designed to alloW synthesiZers and sequencers to commu
nicate What-sound-to-play information. Atypical MIDI mes
sage comprises three components. The ?rst component
being sent, but are still useful components of a MIDI ?le.
begins With a hexidecimal 9 is folloWed by an identi?cation
of one of sixteen MIDI channels having a value of O-F. The
second component is a tWo byte sequence representing the
[0063]
MIDI events have the general form shoWn in FIG.
key on the device that has been pressed. This corresponds to
7. MIDI events are preceded With the hexidecimal notation
FF folloWed by a type ?eld, a length ?eld and a data ?eld.
the notes of a keyboard that has been pressed in the case of
The type ?eld is a single byte specifying a type of META
event. The length ?eld contains a number of bytes of data
00-7F. The third component of a MIDI message is the
folloWing that ?eld. The data ?eld includes 0 or more bytes
of data.
[0064] Anumber of META events have been de?ned in the
standard that make implementation of the invention easier.
These include the folloWing:
[0065]
A “sequence number” is an optional event Which
must occur only at the start of a track before any non-0
delta_time.
a piano-type keyboard. The tWo byte value ranges from
velocity component Which speci?es the velocity With Which
the key Was pressed or released. It, too, ranges in value
betWeen 00 and 7F. Thus, if a musician pressed middle-C
key on a keyboard, the keyboard Would send a “note-on”
message comprising: 90 3C 40. When the musician releases
the key, the corresponding “note-off” message Would com
prise: 80 3C 33. In this case, the key Was released more
sloWly than it Was pressed as indicated by the velocity
indication 33 in the release message (compared With the
attack velocity 40).
[0066] This is typically utiliZed to identify each track.
[0076]
Other MIDI messages may include a program
[0067] A text event is utiliZed for annotating a track With
arbitrary text. A Copyright notice event can be utiliZed
Where a Copyright notice is represented in ACSI text. It
timing clock message.
should be the ?rst event on the ?rst track of a MIDI ?le.
[0077]
[0068] A sequence/track name provides the name of a
sequence or a track in the ?le. An instrument name, provides
the transmitter. If a MIDI device does not knoW What to do
With a message, it Will ignore it. MIDI messages Which are
the description of the instruments used on the track. A lyric
event provides the lyrics for a song. Normally, each syllable
speci?c to a MIDI channel are referred to as channel
Will have its oWn lyric-event, Which occurs at the time the
system or an entire MIDI device are knoWn as system
lyric is to be sung.
messages. Channel and system messages are further divided
(instrument) change, a pitch blend message, a control
change message (e.g. pedal/sWitch foot change of state) and
MIDI messages are all one Way. There are no
acknoWledgement messages sent from the receiver back to
messages. MIDI messages Which affect the entire MIDI
Aug. 14, 2003
US 2003/0151628 A1
into several classes. The channel voice messages are mes
sages Which start, alter or stop a sound or sounds being
played. Channel mode messages effect the entire channel.
System realtime messages are those used by sequencers to
regulate and synchronize timing. They do not contain data
bytes. System common messages include messages such as
song position pointer, song select and the like. System
exclusive messages are generally used for device speci?c
extensions to a MIDI protocol.
keyboard With each N ticks of the MIDI clock (1040). If
MIDI keyboard input is received during an open WindoW
and if the value of the MIDI keyboard input equals the MIDI
note value of the object about to impact the virtual keyboard,
video and audio reWard presentation (routines) are activated
(1060). OtherWise, a video and audio sequence representing
a failure can be activated. This process is repeated for each
tick of the MIDI clock in an iterative fashion as indicated by
the arroW going back to the top of step 1040.
[0078] Since MIDI messages are sent and interpreted in
realtime, it is desirable to reduce the volume of data that
must be sent. For ordinary note-on and note-off messages, it
[0084] Returning to FIG. 3, item 330 represents a library
of objects Which alloW the game to be modi?ed by the
is quite common for several notes to be turned off and on
more or less at the same time. In such cases, it is possible to
send a single status command, such as note-on folloWed by
different ?avors of the game. Examples of the types of
objects utiliZed are given hereinafter. Also shoWn in FIG. 3
a plurality of “note identi?er, velocity” pairs Without repeat
ing the same status byte for each note that is turned on or off.
the performance of the game or the behavior of the game in
certain instances.
This reduction in the transmission of status bytes is knoWn
as “running status” Within MIDI messages. Only the data
bytes that change are sent.
[0085] The audio engine 360 is preferably based on
Microsoft® DirectSound 8 application programming inter
face. The audio engine generally has the ability to play short
[0079] Against the background of the MIDI File and
.WAV ?les through the PCM channels of a sound card. The
sound engine is implemented as a class With constructors,
destructors, loading of a .WAV ?le and playing of a .WAV
protocol, an exemplary operation of that invention Will noW
be described.
[0080]
FIG. 9 illustrates an exemplary How of data illus
trating MIDI messages for the sequential playing of three
notes. In FIG. 9, tWo MIDI messages are shoWn per line.
Each is introduced With a delta_time ?eld. Thus, the six
messages shoWn in FIG. 9 are preceded With delta_time
increments of 00, 08, 10, 18, 20 and 28.
[0081] The delta_time increments are folloWed by a MIDI
command sequence. The sequence 92 indicates the situation
Where a note is to be turned on. FolloWing each 92 message
is a message With a command ?eld of 82 Which indicates that
the note is to be turned off. FolloWing each command is a
note identi?er. Which in the case of the ?rst line indicates the
note C is to be turned on and off. In the case of the second
and third lines, the notes identi?ed are the note E and the
note G, respectively. The last component of each message is
a velocity component. In the ?rst line, the velocity 44 is a
measure of hoW fast the key is pressed doWnWardly (e.g.
attack velocity). In the folloWing message, Where the key is
being released, the velocity 40 indicates that the key is
released With a velocity 40.
[0082] FIG. 10 is a How chart of an exemplary process for
interpreting the MIDI data stream of FIG. 5 as part of
generating game objects. When a MIDI message is received
representing a note on object of the form <delta_time>
substitution of various types of objects for corresponding
is a set of parameters 340 Which alloW a user to customiZe
?le. The sound engine is used primarily for playing sound
effects associated With menu buttons, clicks, sWitches and
game play feedback sounds. HoWever, the invention could
also Work With other softWare engines.
[0086] The graphics engine 350 Works in conjunction With
the library of objects 330 and the game object generation
module 320 to produce graphical objects on the screen that
constitute respective implementations of the game objects
utiliZed to play the game. The invention is capable of loading
and playing three types of song ?les: MIDI, MIDI Karaoke,
and .amm. The later is an internal song format described
more hereinafter. Standard MIDI ?les Were described above.
The MIDI Karaoke ?le is essentially a standard MIDI ?le
With song lyrics built into one of the tracks according to a
certain format. The invention supports the .KAR ?le format
Which is a prevailing ?le format in the shareWare/public
domain markets.
[0087]
An .amm ?le is very similar in structure to a
standard type 1 MIDI ?le. HoWever, the .amm format ?le
Will also hold annotations, highlighting and ?ngerings.
Annotations are simple text notes that are stored on a song’s
timeline. These annotations can hold valuable information to
a user and can be displayed during the game play along With
other scrolling game objects. They may be stored as text in
<Command> <Note> <Velocity> (1000) optionally fol
the internal ?le, but on the screen these are shoWn in the
form as speech balloons With or Without arroWs pointing to
loWed by a MIDI message representing a note-off property
other game objects.
of similar format (1010). At time <delta_time> less traversal
[0088] Some notes in an .amm ?le can be highlighted. This
is essentially a ?ag that tells the system that a note should be
displayed With an enhanced video effect such as a halo effect
time, an object is launched on <Note> trajectory With
optional length property of <delta_time (off)>—<delta_time
(on)> ticks (1020). In this form, one can see that steps
1000-1020 represent an interpretation of the incoming MIDI
messages folloWed by the generation of a game object.
Messages are received and interpreted on an ongoing basis
in accordance With the steps 1000-1020. A WindoW is
de?ned Within '5 units of time of impact With the virtual
keyboard (1030).
[0083] Once the object is launched on the note trajectory,
the object is moved along the trajectory toWards the virtual
around it. Fingerings are little clues to a user as to Which
?nger should be used to play a note on a piano keyboard.
Fingerings are assigned to individual notes and hold a
number in a range from 1 to 5. During game play, game
objects that have ?ngerings assigned Will scroll along With
the number placed on them.
[0089]
FIG. 11 is a block diagram shoWing an exemplary
arrangement of game modules in accordance With one aspect
of the invention.
Aug. 14, 2003
US 2003/0151628 A1
[0090] When a game is ?rst loaded, a splash screen 1100
is displayed. This is centered on the desktop and enables the
user to click on icons or links that Would take them to the
sponsoring company’s Website Where they can doWnload
neW songs for the game. Also, there may be a link to HTML
documentation for the game.
coupled to bus 102 for storing static information and instruc
tions for processor 104. A storage device 110, such as a
magnetic disk or optical disk, is provided and coupled to bus
102 for storing information and instructions.
[0097]
Computer system 100 may be coupled via bus 102
The Welcome Screen 1110 alloWs four options.
to a display 112, such as a cathode ray tube (CRT), for
displaying information to a computer user. An input device
First, it alloWs selection of a main menu Which is described
hereinafter. Second, it alloWs access to an instant play mode
coupled to bus 102 for communicating information and
[0091]
of operation, described in conjunction With item 1140 here
inafter, Which assumes all previously set up con?gurations
of the last song played in the game. A third option from the
main menu takes one to the credits screen Where game
credits are revealed in a visual Way. Finally, the main menu
1120 has an option to exit the game and return to the
operating system.
114, including, for example, alphanumeric and other keys, is
command selections to processor 104. Another type of user
input device is cursor control 116, such as a mouse, a
trackball, or cursor direction keys for communicating direc
tion information and command selections to processor 104
and for controlling cursor movement on display 112. This
input device typically has tWo degrees of freedom in tWo
axes, a ?rst axis (e.g., x) and a second axis (e.g., y), that
alloWs the device to specify positions in a plane.
[0092] From the main menu 1120, there are three options.
If a quick play option is selected, the user goes directly to the
[0098]
quick play mode described in conjunction With 1140 Where
cessor 104 executing one or more sequences of one or more
the user can freely load any song, select a number of options
and commence the practice. No pro?le needs to be selected
instructions contained in main memory 106. Such instruc
tions may be read into main memory 106 from another
or created. In the second main menu option, the career mode,
computer-readable medium, such as storage device 110.
a user Will create a pro?le and Will face a number of
Execution of the sequences of instructions contained in main
memory 106 causes processor 104 to perform the process
challenges, prede?ned Within a number of levels and in a
number of different piano methods. These are discussed in
detail hereinafter in conjunction With career mode 1150.
[0093] Also from the main menu, a game set up 1130 may
be selected. In game set up, a number of game objects can
be con?gured as described more hereinafter.
[0094] Instant play 1140 alloWs the user to play the game
Without the hassle of going through start up options. While
Computer system 100 operates in response to pro
steps described herein. In alternative embodiments, hard
Wired circuitry may be used in place of or in combination
With softWare instructions to implement the invention. Thus,
embodiments of the invention are not limited to any speci?c
combination of hardWare circuitry and softWare.
[0099] The term “computer-readable medium” as used
herein refers to any medium that participates in providing
instructions to processor 104 for execution. Such a medium
previous starting option parameters are assumed, including
may take many forms, including but not limited to, non
the last song played if the game Was initially installed and no
songs have been played yet, a default song is assumed.
Instant play has no menu, it is just a short cut to the game
volatile media, volatile media, and transmission media.
Non-volatile media includes, for example, optical or mag
play.
[0095] FIG. 12 is a block diagram shoWing details of an
exemplary computer that can be used to implement various
aspects of the invention. The description of the invention
Which folloWs is exemplary. HoWever, it should be clearly
understood that the present invention may be practiced
Without the speci?c details described herein. Well knoWn
structures and devices are shoWn in block diagram form in
order to avoid unnecessarily obscuring the present inven
tion. At least portions of the invention are intended to be
implemented on or over a netWork such as the Internet. An
example of such a netWork is also described in FIG. 12.
[0096] FIG. 12 is a block diagram that illustrates a com
puter system 100 upon Which an embodiment of the inven
netic disks, such as storage device 110. Volatile media
includes dynamic memory, such as main memory 106.
Transmission media includes coaxial cables, copper Wire
and ?ber optics, including the Wires that comprise bus 102.
Transmission media can also take the form of acoustic or
light Waves, such as those generated during radio-Wave and
infra-red data communications.
[0100] Common forms of computer-readable media
include, for example, a ?oppy disk, a ?exible disk, hard disk,
magnetic tape, or any other magnetic medium, a CD-ROM,
any other optical medium, punchcards, papertape, any other
physical medium With patterns of holes, a RAM, a PROM,
and EPROM, a FLASH-EPROM, any other memory chip or
cartridge, a carrier Wave as described hereinafter, or any
other medium from Which a computer can read.
tion may be implemented. Computer system 100 includes a
[0101]
bus 102 or other communication mechanism for communi
involved in carrying one or more sequences of one or more
cating information, and a processor 104 coupled With bus
102 for processing information. Computer system 100 also
instructions to processor 104 for execution. For example, the
includes a main memory 106, such as a random access
memory (RAM) or other dynamic storage device, coupled to
bus 102 for storing information and instructions to be
executed by processor 104. Main memory 106 also may be
used for storing temporary variables or other intermediate
information during execution of instructions to be executed
by processor 104. Computer system 100 further includes a
read only memory (ROM) 108 or other static storage device
Various forms of computer readable media may be
instructions may initially be carried on a magnetic disk of a
remote computer. The remote computer can load the instruc
tions into its dynamic memory and send the instructions over
a telephone line using a modem. Amodem local to computer
system 100 can receive the data on the telephone line and
use an infra-red transmitter to convert the data to an infra-red
signal. An infra-red detector can receive the data carried in
the infra-red signal and appropriate circuitry can place the
data on bus 102. Bus 102 carries the data to main memory
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement