Mustajuuri – a sound processing application and toolkit

Mustajuuri – a sound processing application and toolkit

Mustajuuri – a sound processing application and toolkit

Tommi Ilmonen

13th May 2002

Contents

1 Introduction 3

2 Related work 3

2.1

CSound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4

2.2

SuperCollider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4

2.3

Max variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4

2.4

Synthesis Toolkit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5

2.5

LADSPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5

2.6

Ordinary hard-disk recording software . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5

3 Software requirements 5

4 Design choices 6

4.1

Plugin architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

4.2

Data transfer paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

4.3

Continuous signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

4.4

Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

4.5

Signal flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

4.6

Real-time framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

4.7

Programming base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10

4.8

Base classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10

4.9

Cache optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10

5 Important classes 11

5.1

class MJ ModuleDsp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

5.1.1

virtual bool dspPrepare() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

5.1.2

virtual bool dspPrepareFast() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

5.1.3

virtual void dspProcessPeriod() . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

5.1.4

virtual bool dspStopFast() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

5.1.5

virtual bool dspStop() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

5.1.6

virtual bool msgProcess(const char *id, MJ Message *m) . . . . . . . . . . . . . .

11

5.1.7

virtual MJ Parameter *msgGetInput(const char *id) . . . . . . . . . . . . . . . . .

12

5.1.8

virtual QCString msgGetInputName(int index) . . . . . . . . . . . . . . . . . . . .

12

5.2

class MJ Satellite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

5.2.1

Signals and slots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

5.3

class MJ ModuleDspUi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

5.4

class MJ ModuleDspGui . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

1

6 Important plugins 13

6.1

Mixer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

6.2

Audio and MIDI input/output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

6.3

Multi-loudspeaker 3D audio panning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

6.4

Polyphonic synthesizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

6.5

Auralization control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

7 Library layout 15

7.1

divabase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

7.2

divaio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

7.3

ac . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

7.4

mjdsp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

7.5

mjwidgets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

7.6

mj . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

7.7

mjmodules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

7.8

mjutils . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

7.9

plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

8 Directories 16

9 File, function and data type names 17

10 Development hints 17

10.1 Plugin programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

10.2 About the compilation system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

11 Further information

12 Future directions

References

18

18

18

2

Abstract

Mustajuuri is a modular audio signal processing application. Mustajuuri can be used as an effects processor and as an audio toolkit. It has been developed especially for the UNIX/Linux platforms.

This paper describes the overall qualities of Mustajuuri and its technical base.

1 Introduction

This paper presents the technical basis of a modular audio application Mustajuuri. Mustajuuri has been created to act both as an audio signal processing test-bed and an end user application. The demand for modularity, flexibility, interactive use, high-performance signal processing, low latency and platform independence have been taken into account right in the beginning. Mustajuuri is an open source effort aimed primarily at the UNIX/Linux operating environment. It is portable to other architectures and operating systems as well.

Mustajuuri is composed of several libraries. An optimized low-level digital signal processing (DSP) library – libmjdsp – is supplied so that it would be easier to write own plugins. Often-needed extra widgets for the graphical user interface (GUI) are supplied in libmjwidgets library. More exotic extensions have their own library – libmjutils. The primary library – libmj – contains the plugin API (application programming interface) and several support classes. The plugin architecture is very flexible. It supports audio streams (constant stream of audio at fixed rate) and control messages (named and time-stamped messages containing arbitrary data).

Mustajuuri can be used in many ways. One can use it as an interactive audio DSP engine (see example in figure 1). This is done starting the application with graphical user interface and using it in normal desktop style. A more involved user can create new plugins for the system and simply use Mustajuuri as a host to run the plugins. This saves the user the trouble of creating from scratch all the necessary infrastructure that is needed in interactive audio applications – audio I/O, real-time considerations.

Mustajuuri has also been used as stand-alone DSP engine without the GUI. In this case it was used to run 15-channel audio system. Rather than having local control it was controlled over the Internet. To achieve this there was no need to modify the base code. Instead the users wrote the necessary control functionality in a few plugins.

Mustajuuri has been written with the C++ programming language. It relies on the Qt class library that provides platform-independent base functionality (file input/output, graphical user interface, C++ template classes). The plugin programming interface has been designed to help create plugins with ease.

To create a new plugin type one only needs to inherit the plugin base class and extend it to match the needs. The strong type safety checks of C++ compilers eliminate many bugs that might creep in with some other languages.

All platform-dependent code has been isolated behind wrapper classes to aid future portability.

Mustajuuri uses the Qt2-library heavily (www.troll.no for info on Qt). Both for user interface and as a generic class library (for templates etc.).

This paper describes the technical features of Mustajuuri. First we introduce related software packages. The original requirements of Mustajuuri are described in chapter 3. Chapter 4 goes thru the design choices we made when creating Mustajuuri and the most important classes are overviewed in chapter 6.

Chapters 7-10 contain information that is highly useful to anyone how plans to develop new Mustajuuri plugins or use the code in some other way.

2 Related work

Modular audio signal processing systems have been in existence since the seventies. There are several toolkits that address only the signal processing tasks while some systems incorporate an interactive

(graphical) user interface to the design. The following sections give examples of the current state of the art.

3

Figure 1: Mustajuuri running on the desktop. Several plugin GUIs are visible in around the large mixer window.

2.1

CSound

CSound is probably the most well known and widely spread free-ware audio processing system. The basic approach is modular – one creates a DSP network and then uses the network to process audio. CSound has been written in C. The system has been created to be used solely with the file and command line interface. Real-time processing has been added only lately. While the system does not directly support any GUI there are several alternatives for that purpose. Unfortunately these do not answer the real-time usage problems. The basic architecture of CSound makes the creation of a properly integrated user interface a daunting task.

CSound is such a popular platform that a full-length book has been written about it (Boulanger,

1999) in addition to manuals and tutorials. CSound was created by Barry Vercoe in the 80’s.

2.2

SuperCollider

SuperCollider by John McCarthy is one of the leading signal processing languages today (McCartney,

1996).

It runs on the Apple Macintosh platform.

SuperCollider has its own object-oriented signal processing language that offers a flexible way to define signal processing operations. The signal processing system can also be changed in real-time by operations done in the language.

2.3

Max variants

The Ircam Max and related software packages were created to allow user to run MIDI and audio processing in real-time with a graphical user interface. They have been born in the Ircam research center. The most up-to-date version of Max is the jMax, which has been written with the Java programming language problems lie in the user interface that supports only one kind of usage. The user also has problems when he would like know how exactly the signal is being processed.

4

2.4

Synthesis Toolkit

Perry Cook and Gary Scavone have written a general-purpose toolkit for signal processing, the Synthesis

ToolKit (STK) ((Cook & Scavone, 1999)). STK is a low-level library that contains a collection of DSP algorithms for sound synthesis.

2.5

LADSPA

Linux Audio Developers Simple Plugin API – LADSPA – is a new standard for audio plugin development for the Linux platform (LADSPA-team, Cited 1.5.2001). The API is written in C and it aims to be as simple as possible. In practice it replicates the CSound signal processing approach – audio signals at fixed rate and control signals at cycle rate.

As the name implies, the LADSPA API is simple. It is based on the lowest common denominator approach – less is more. The API is only concerned with DSP. There are many useful features that it does not support: Custom user interface for plugins, plugin-specific custom data file formats, multi-rate signal processing, recursive plugins, internationalization, custom data types (MIDI, strings).

LADSPA is widely supported in many current DSP applications.

2.6

Ordinary hard-disk recording software

There are many commercial and non-commercial applications for hard-disk recording (ProTools, CuBase,

Sonic Foundry, CoolEdit, Ardour). Most of these applications have been developed for the Windows of

Macintosh environments. While they have sophisticated audio processing capabilities few of them can be used for real-time signal processing purposes. This problem is caused usually by large latencies caused by the operating systems and the internal design of the software.

Few of these applications are for for multi-channel operation (more than ten channels) that is often necessary for virtual-reality applications.

3 Software requirements

The software is based on several requirements that dictated the final design choices. These included both technical (performance and portability) and human factors (user interface) issues. The next paragraphs list the requirements that were given the software in the beginning. The design choices that correspond to these requirements are discussed in chapter 4.

The application will be used as a real-time signal processing tool. To fulfill this requirement the

DSP-process must run at low latency – preferably less than 5 milliseconds (section 4.6).

The system is open-source and plugin-based. Thus third parties can create both open-source and proprietary plugins for the system (section 4.1)

It must be possible to define a completely customized user interface (UI) for any module.

Default UI functionality should exist so that it not necessary to always code a new user interface for a plugin.

It must be possible to create arbitrary signal processing chains with the plugin instances (modules).

Figure 2 shows an example of a simple signal processing network.

There must be multiple usage metaphor options, not just a mixer (a la ProTools) or signal-connection system (like in Ircam Max). It must be possible to introduce new user interface metaphors via plugins.

A plugin should have multiple user interface options. It is not enough to allow the system to simply open a window for the plugin. It should also be possible to edit just one or two parameters of the plugin in an optimized window (section 4.1)

The plugin system should allow consistent intra-application user interface operation with global undo/redo.

The plugin design should scale well from simple plugins (monophonic delay, low-pass filter) to very large systems (reverberator, synthesizer).

5

Figure 2: A simple DSP network.

The plugin design must be able to handle any audio-related application: hard-disk recorder, automated mixer, MIDI sequencer, ZIPI sequencer, oscilloscope etc. (sections 4 and 4.2).

The system must not depend on a particular user interface type. It should be possible to operate the system with or without the GUI.

Mustajuuri should support internationalization. The most basic implication is that all text needs to be translatable to users’ language.

It should be possible to embed Mustajuuri into other applications. Thus it could be used as the sound processing engine without showing anything about it to the user.

Mustajuuri must be portable to different systems. In practice the critical systems are IRIX, Linux and different UNIX-variants in general. The portability to other platforms (especially Macintosh and the different Windows platforms) is hoped for, but not a fundamental requirement.

4 Design choices

In this chapter the most fundamental design decisions in Mustajuuri are introduced. These include the data flow, threading and user interface mechanisms.

4.1

Plugin architecture

The plugin architecture is based on the C++ class system.

There is a base class for all plugins:

MJ ModuleDsp (section 5.1). New plugin classes are created by inheriting this base class and adding new functionality. The new functionality is created by adding new member variables and by overriding the virtual methods that make up the plugin API.

A plugin is not too useful without a user interface. In Mustajuuri a particular module may have multiple user interfaces – both graphical and text-based. Most of the energy has been spent on the graphical user interface (GUI). The programmer can create a customized user interface for a DSP plugin.

6

The user interfaces are created by making a new GUI class. Just as the DSP plugins must inherit the base class, so must GUI-classes inherit their base class: MJ ModuleDspGui.

There are many ways to use a plugin. Correspondingly the user interface must match the need at hand. Three different usage methods were identified: 1) A minimal user interface that only shows a parameter or two (used when many modules need to be shown at once) 2) an intermediate user interface

– about the size of a post card – that can be used along with rest of the application (a few plugins can be viewed at once) 3) a user interface in a separate window. With Mustajuuri plugin system one can create an optimized user interface for any of these cases.

The plugin binaries are stored in shared libraries that are loaded in run-time. The plugin libraries are probed for plugin-factory function that is used to create plugin database entries. The database entries contain necessary factory functions to create the DSP modules (plugin instances) and corresponding GUI elements. This way New features (plugins) can be introduced to the system without touching the code base.

A plugin database entries contain a lot of information about the plugin. The information includes plugin’s author(s), license, name in menu hierarchy.

The plugin entry also includes information about the language translation file that should be used with the plugin. The translation file contains the phrases that are needed by the user interface. The file is in standard Qt translation format

1

. Thus plugin programmer only needs to create a new translation file and attach the name of the file to the plugin database entry. Once this is done Mustajuuri can automatically load the translations as they are needed.

Since all code is licensed under the LGPL it is possible to create commercial plugins that link against

Mustajuuri base libraries.

4.2

Data transfer paths

Mustajuuri plugin architecture uses two data flow alternatives. The most usual flow type is the “signal”

(or audio) flow. It is used where continuous fixed-rate data flow is needed. An alternative method is the message passing interface (typically called “control” interface) that is used to transmit control parameters that need not be changed for each audio sample. These data flow mechanisms are presented in figure 3.

These data transfer paths allow many different plugin types to be created and connected to each other.

CONTROL INPUTS

MODULE

CONTROL OUTPUTS

Figure 3: Data flow to/from a module.

Mustajuuri does not support any intermediate control mechanism. This contrasts CSound/LADSPA approach where DSP-blocks have some signals that come in at audio rate and others at “control rate”.

In the design process of Mustajuuri it was deemed that it is seldom practical to have such control signals as they turn many CPU-heavy algorithms much more heavy. Such approach effectively means that a module has to update its control parameters at each processing cycle (or at least poll critical parameters for any update). In cases where a module has a great number of parameters this causes severe performance

1

Qt includes full multilingual framework that has been opted to Mustajuuri. Thus Mustajuuri relies on ready-made tools rather than reimplementing the code.

7

degradation. For example a 31-band parametric EQ module would need to check its 93 control parameters

(and potentially update corresponding filter coefficients) at each processing cycle.

Another problem with CSound-style control signals is the parameter interpolation. Parameter interpolation is difficult to control. Typically the host may do the interpolation, but there is no practical way to know how it should be done. The time-stamped message mechanism allows the DSP-blocks to decide upon the interpolation mechanism.

4.3

Continuous signals

Signals are data streams that have fixed sampling rate. The streams are implemented as floating point data buffers. Currently the system uses 32-bit floating point numbers, but the type may be changed in the future to some other resolution. Floating point representation was chosen because most signal operations require floating point numbers anyhow. Besides, modern CPUs process integers and floating point numbers at the same rate so there is little or no performance difference between integers and floating point numbers. It was also deemed that 64-bit floating point numbers would be an overkill for nearly any audio application.

A module may have n input signals and m output signals. Each signal may be have different sampling rate. This flexibility is needed to allow multi-rate signal processing systems to be made. Most modules need not take this feature into account.

A module processes multiple samples per one period (also called cycle). This increases intra-application feedback, but it is a necessary choice to reduce overhead caused by virtual function calls. This also affects the multirate signal processing operation. In multirate systems all sampling rates in use must fit exactly to the “master” sampling rate. Thus each input and output buffer has samples that represent exactly the same amount of time.

4.4

Messages

Messages (or events) are used to transmit single parameter changes to the modules. The message mechanism serves many purposes. It is the basis for automatic UI generation. It is used by the undo/redo facilities. It allows modules to pass event-like data to one another (MIDI events for example). It is also used to save module data to the disk.

Messages are time-stamped data packages. Typically messages are passed between signal processing cycles. The time-stamping is sample-accurate so that the receiving module can perform exact parameter interpolation when necessary. The messages are passed explicitly to the module – a module is notified as the message arrives. Thus a module does not need to poll any external data buffers for parameters.

The messages are identified by an identifier string. A more simple method would be to use direct numeric indexing. This can be a problem with plugins that can have an extremely large number of parameters. With this approach the parameters can be named very much like files, for example. This approach is similar to OpenSound Control (OSC) approach (Wright & Adrian, 1997). It is also very useful when the signal processing system is hierarchical. It is possible to pass parameters to a module in the bottom of the hierarchy by issuing the whole path to the parameter, for example: mixer/strip2/module3/delay_time

This turns out even nicer when one thinks about a networked system. It could be possible to address a parameter in the Internet with the string: soundserver.sound.org:12021:/mixer/strip2/module3/delay_time

^server name ^port ^parameter name

While these features are not yet incorporated into the plugins they are merely matter of plugin implementation; the API supports this kind of operations directly.

The virtual method that receives the messages is further described in sections 5.1.6.

8

4.5

Signal flow

Determining the correct call order for the modules is a non-trivial task. The problem is illustrated in figure 4. In the given system the processing order makes a difference. There are two attractive processing orders that could be used: ABCD and ACBD. In this case the system should know what the user wants to happen. There is no practical way to guess the right processing order.

A

C

B

D

Figure 4: Signal timing problematics – what is the optimal call-order?

Different software packages offer different solutions. A signal processing toolkit “Sonic Flow” uses a heuristic algorithm to determine the processing order (SF, 2001). Another approach is to use internal sample counters to guarantee that the signal flow is exact (all feedbacks must have an explicit delay).

Mustajuuri leaves the ordering to control plugins and the user. The mixer (section 6.1) for example has a fairly obvious way to order the processing: left to right and up to down. The heuristic method was not chosen since it might cause non-intuitive signal flow that a user cannot control easily. The internal sample-count mechanism was deemed to have too large overhead to be practical.

4.6

Real-time framework

The DSP process must run at very low latency if it is to be used as a real-time signal processing engine.

On the other hand, many operations that the application must perform (user interface actions, file access) may take notoriously long time. To solve this problem the software needs to be threaded.

Figure 5 shows the threading approach. The DSP thread contains modules that perform the signal processing. The UI thread contains UI objects that are used to control the DSP modules. Additional auxiliary threads are used for audio file I/O etc. The DSP thread can execute always at low latency while the other threads may block.

Unfortunately even such threading may not be enough. Many operating system kernels (for example the Linux kernel) may block at some disk access or other activity. There is little the application can do to overcome this problem. Fortunately patches that address this issue are available (Morton, 2001).

With Linux kernel 2.4.1 and the low-latency patches in place the highest latencies experienced by the

DSP thread have dropped from over 100 milliseconds to less than 5 milliseconds.

The current system runs all the DSP-modules in one thread. An alternative would be to span multiple

DSP threads – in the extreme case one thread for each DSP block. The design choice (minimal threading) was made to simplify programming and debugging. It also translates to high performance on systems with only one CPU. Even on multiprocessor machines this approach can utilize more than one CPU, but not with maximal efficiency. An interesting future possibility is the multi-threading of the DSPprocesses. For example a four-CPU system could run four DSP-threads. The plugin API is thread-safe so implementing this feature in the future is not a problem.

9

LOW PRIORITY GUI THREAD

HIGH PRIORITY DSP THREAD

AUXILIARY THREADS

Figure 5: Threading model.

4.7

Programming base

The C++ programming language was chosen for Mustajuuri. The strong points of C++ are: portability, universality, stability, object-orientedness, performance and availability of support libraries.

To augment the C++ language a base library was needed. C++ does not offer user interface tools.

The template classes tend to be difficult to use and learn. To fulfill these needs the Qt2

2 library by

Troll-Tech was chosen (Troll-Tech, Cited 1.5.2001). The Qt2 library is available as an open source toolkit for the UNIX platforms and as a commercial version for the Windows platforms. Qt supplies a good code base for platform-independent programming: template classes, I/O classes, a comprehensive widget library and internationalization.

4.8

Base classes

Mustajuuri is composed of a few very important classes and a collection of support classes.

The most important class in Mustajuuri is the the class MJ ModuleDsp. It is the base class for signal processing modules. Another important class is the class MJ ModuleDspUi that is the base class for user interface objects. This class is inherited by MJ ModuleDspGui that is the base class for module GUIs.

Class MJ PluginDbeDsp is used to contain plugin-type data and the necessary factory functions for plugin generation. Information about the plugins I/O capabilities (number of channels, sampling rates) is stored to MJ PluginDbeDsp objects.

4.9

Cache optimization

It was found that Mustajuuri consumes different amount of CPU-time in different run times – even on the same system. It was suspected that this could only be caused by improper memory layout – different cache behaviour at different times. Thus cache-optimization trick were taken into use.

In many DSP-applications the modules need some amount of working memory that is needed at each processing cycle (filter coefficients and memory values for example). On the other hand the module may have data that is seldom needed. The hosting system may allocate memory to these needs without knowing which data is often needed and which is not. This in turn may cause the data to be spread all over the memory in a way that degrades cache-hit ratio.

2

Mustajuuri works also with Qt version 3, but does not require it.

10

In the optimal case all of the most often needed data would be in a contiguous memory area. Thus the cache locations of the data would be minimally conflicting. Mustajuuri incorporates an internal memory allocation scheme to make this possible. Thus modules may allocate memory from the continuous area with a call to method MJ ModuleDsp::mjDspMalloc.

The usage of optimized memory allocation seems to have solved the performance variation problems.

After this improvement the CPU-load has been much more stable. The custom memory allocation scheme is something a plugin programmer can simply ignore if they do not want to use it (this is also true for many other parts of the Mustajuuri API).

5 Important classes

5.1

class MJ ModuleDsp

This is the base class for all signal-processing objects. To create new kind of module that does DSP, one needs to inherit this class or some of its child classes.

5.1.1

virtual bool dspPrepare()

This method is called prior to starting the signal processing. DSP data structures should be initialized within this method.

5.1.2

virtual bool dspPrepareFast()

This method is called just prior to starting the signal processing. Usually child classes do not need to reimplement this method. It is only necessary if a module needs to perform some operations just before starting the DSP.

5.1.3

virtual void dspProcessPeriod()

This method is called repeatedly during the signal processing. It should process one cycle of data typically less than 100 samples. The number of samples to be processed is always constant between dspPrepare and dspStop.

Note that the LADSPA plugin interface (section 2.5) does not guarantee that the number of samples per processing period is going to be constant. As a result special care has to be taken when writing plugins that should also work as LADSPA plugins.

5.1.4

virtual bool dspStopFast()

This method is called just after the processing is done. Hardly any modules need to override this method.

5.1.5

virtual bool dspStop()

This method is called after the processing is done. Module should free data structures that are not needed.

5.1.6

virtual bool msgProcess(const char *id, MJ Message *m)

This method receives messages from other modules or from GUI. Each message is identified by its identifier string – “id”. There are several message classes that inherit the MJ Message class. These classes can be used to transmit information. Available messages types are numeric (double and integer), string and

MIDI-event. It is also possible to create new message types if they are needed.

11

5.1.7

virtual MJ Parameter *msgGetInput(const char *id)

This method can be used to the the user interface what are the current parameters of this module. Based on this information the user interface can generate automagically user interface for this module.

The MJ Parameter class (and its child classes) contains information about the parameter besides its value – allowed range, filename filters etc.

5.1.8

virtual QCString msgGetInputName(int index)

With simple plugins the parameters can be identified fixed number of parameters. This method helps the system determine the names.

5.2

class MJ Satellite

Each object that inherits MJ ModuleDsp has a “satellite” – a class member variable that inherits QObject.

It is used to store and transmit GUI-specific run-time information of the module. GUI thread owns the satellite – it does not need to lock the control mutex to access the satellite. Once the satellite is created the DSP-module (running in DSP thread) should never access the satellite.

Why is this class needed ?

A module can have multiple GUIs open at the same time. If the module is modified all GUIs need to be updated (parameter slider etc.). The satellite is an object that is used to trasmit parameter change information. All GUI-objects should attach to the satellite via Qt’s signal/slot mechanism

3

. Once this is done all GUI instances will be notified as the module changes.

Undo/redo information is also stored in MJ Satellite.

The functionality of the satellite could have been included in MJ ModuleDsp. The current design was adopted so that in systems where the satellites are not needed they could be deleted – potentially saving a lot of memory. Additionally it clarifies the situation where the module operates within the DSP thread

(with mutex protection), but the satellite is accessed by the user interface thread.

5.2.1

Signals and slots

This class has some Qt signals and slots available for your use. In some cases you may need more than these. The solution is to inherit this class and create a new satellite type that contains required functionality (by making more Qt slot/signal pairs).

Slots for transferring edit information:

• modified(const char *)

Corresponding signals:

• emitModified(const char *)

Redo/undo information is also stored here. The information is stored in lists that contain single undo/redo pairs. Once some information os added or removed the corresponding signal fire. Thus all interested parties are informed of the new undo/redo information.

• addUndoInfo(MJ UndoInfo *)

• addRedoInfo(MJ UndoInfo *)

5.3

class MJ ModuleDspUi

This is the base class for any user interface object. User interface objects are used to give the user easy access to the plugins. The user interface objects can be arbitrarily simple or complex. This class does not define how the plugin should be shown to the user (if at all). The children of this class implement actual user interface styles (for example GUI or command-line operations). This class does – however – implement a few helpful functions: it contains methods to start and stop signal-processing.

3

The signal/slot -mechanism is the basic element of Qt’s message passing system. Readers unfamiliar with it may read documentation from Troll-Tech’s WWW-site (Troll-Tech, Cited 1.5.2001)

→ “Reference Doc.” → “Signals and Slots”

12

5.4

class MJ ModuleDspGui

The base class for module GUIs.

A module can define a GUI. The new GUI classes must inherit

MJ ModuleDspGui. The base GUI class includes mutex that are used to control access to the DSP data. The GUI objects must always lock and unlock the control mutex when they need to alter parameters in the DSP process.

6 Important plugins

There are several plugins that come with the system.

6.1

Mixer

The mixer can be used as a top-level plugin. It implements a user metaphor that mimics analog mixers’ user interface. A screenshot of the mixer plugin is in figure 7. Figure 7 shows an example of how data might flow inside a mixer.

Figure 6: An example mixer.

MIDI IN

SYNTH

AUDIO IN

INTERNAL IN

REVERB

COMPRESSOR

LEVEL METER

GAIN

FILE OUT

AUDIO OUT

AUDIO

MIDI

EQ

GAIN

MUX

Figure 7: The signal flow inside a mixer. This setup is similar to the mixer in figure 6.

13

The mixer does not reproduce all the anomalies of analog mixers. The routing possibilities are much more flexible; there can be any number of channels in strip (useful for multichannel mixing) and signal transfer to/from buses can be inserted anywhere in the signal chain (not just top or bottom).

The flexibility has its down-sides in that the mixer is not very straightforward to use. More work needs to be done to make it user-friendly.

6.2

Audio and MIDI input/output

An audio application is hardly useful if it cannot play any sound to the user. With Mustajuuri we have created a collection of plugins that are used to perform audio and MIDI input and output operations.

At first they were built directly as Mustajuuri plugins. At later stage it was deemed that is would be nicer to separate them into a smaller stand-alone library DIVAIO that would be used by the plugins.

Such small libraries are also easier to code and test than the corresponding plugins. The problem of this approach is that the system has many levels of indirection between the application main loop and actual audio I/O. As a result it is more difficult to determine just how the various components should interact

(especially in debugging situations).

6.3

Multi-loudspeaker 3D audio panning

Mustajuuri includes two 3D audio panning plugins. These plugins use the Vector Base Amplitude Panning

(VBAP) -algorithm by Ville Pulkki. The two variants were created to fullfil the different needs of people doing audio processing.

The first VBAP plugin is capable of panning sound from n input channels to m loudspeakers. The number of loudspeakers and their configuration is given in a text file. The plugin reads the configuration file as it is started. As the location of sound source changes the plugin calculates new gain coefficients for the sound source and interpolates the loudspeaker gains to avoid clicks.

The second VBAP plugin does the same panning operations as the first one, but adds distance attenuation and delay. These features are are needed when one wants to control also the distance of the sound from the listener. Each source has separate gain and and delay settings that are adjusted independently. To avoid clicks the delay time and gain are interpolated.

6.4

Polyphonic synthesizer

This plugin is an extendible synthesizer.

This synthesizer features a special approach to managing different voices (additive synthesis, sample playback synthesis, granular synthesis etc.): The synthesizer has a pool of voice factories. If one wants to create a new voice type then one only needs to add a new factory to the collection. The factories are named with crahacter strings (“sample”, “granular”). It would be easy to make this a plugin-based system. Then one could add new voice types without even recompiling the synthesizer.

The synthesizer does not have a large collection of voice types yet. The current voice types are:

• sample – play a sample from a file (optionally looping)

• pitchsample – play a sample with specified pitch

• input – direct one input directly to the output

• granular – granular sound synthesis

Each voice type inherits a common base class that has the functions that are necessary for control ofthe synthesis. This API is fairly simple and it offers an easy way for a programmer to add new voice types.

The voices are controlled via messages just like the DSP plugins; they have virtual method “msgProcess” that is used to set their parameters. The parameters are named with with character strings and the message is passed as a pointer. This way one can pass any commands to the notes.

14

Each note has a unique identifier. The identifiers is a simple integer value. When one sends a noteon message to the synth it allocates a new unique identifier to the note and adds the new note to the list of active notes. The individual notes can then be controlled with their note identifiers. The notes are controlled with the normal Mustajuuri message system. As an example if wanted to change the

“worbliness” of note 678 then this would be done by sending the synth a message that controls that particular note. In this case the address of the message would be “note/678/worbliness”.

The synthesizer can have any number of output channels and any note can be directed to any output sound channel. This feature is used in the 3D-audio operation where one output channel corresponds to one virtual sound source. The 3D panner can then pan one audio channel to any position in space and all the notes that attached to that channel will move with the source.

The synthesizer manages polyphony and holds a few common resources for the voices: The synth loads the required sound files so the different voice types do not need to load the files.

6.5

Auralization control

This plugin implements networked auralization server features. The clients can connect to the server via

TCP/IP socket and control the sound synthesis. This plugin is uesd in conjunction with the AC library

(section 7.3).

As a client connects to the server the auralization controller first starts the DSP processing and then transmits the commands coming from the client to the relevant plugins. The client can control the synthesizer- and VBAP -plugins via this method.

7 Library layout

Mustajuuri code is split to several libraries.

Some of these libraries are small stand-alone libraries

(divabase, mjdsp) while some others are largish monolithic packages (mj, mjmodules). There are also in-between versions – nearly independent libraries – mjwidgets and mjutils.

The libraries are listed here in compilation order. This also reflects their mutual dependencies somewhat.

7.1

divabase

DIVA base library provides platform-independent C++ wrappers for threads, sockets, sleeping etc.

7.2

divaio

DIVA IO library provides media IO. Currently this means platform-agnostic C++ wrappers for audio and

MIDI. Currently supported platforms are IRIX, Linux/x86/OSS, Linux/x86/ALSA-0.5.x and Linux/x86/ALSA-

0.9beta3.

There are also audio file classes and threaded audio file classes. The former are an often-needed basic utility and the the threaded audio files are needed remove the file I/O latency from the DSP-thread latency.

7.3

ac

Auralization control library (AC) offers a simple API to control the Mustajuuri sound synthesis and 3Dpanning components. This library controls the Mustajuuri sound engine via a TCP/IP socket connection.

Mustajuuri may be running in any host that can be contacted.

This library can be used in any application easily – it is small and it does not depend on other libraries.

The AC-classes require that the user first starts Mustajuuri with proper setup that is configured to have the necessary plugins. In practice this means that one must create a mixer that contains the synth-,

VBAP- and auralization control -plugins.

15

7.4

mjdsp

Mustajuuri DSP library provides a few optimized low-level signal processing tools. Common filters (first and second order IIR) and filter design algorithms are available. These are maximally simple and efficient filtering implementations. They can be used within plugins to perform common DSP operations. This library is totally stand-alone and it can be used without any of the other libraries (even without divabase).

7.5

mjwidgets

Mustajuuri widget library contains some not-so-common widgets and GUI utilities to complement Qt’s selection classes. Available widgets are level-display widget (vertical bar), slider and text entry with floating point value. There are also utilities to pack numeric data displays to screen.

Among the miscellaneous widgets are the graphics classes. They are meant for quick and simple 2D visualizations.

7.6

mj

Mustajuuri base library contain a whole lot of widgets, DSP classes, plugin-management tools and realtime operation for Mustajuuri. This base library does not contain any plugins or true DSP code. It does – how ever – contain the Mustajuuri application code. The application code does not need to do anything special, it simply creates one instance of MJ Kernel and MJ KernelGui which together form the Mustajuuri application.

7.7

mjmodules

Mustajuuri modules library contains several frequently needed plugins types. These include mixer, level meter, panner, delays etc.

7.8

mjutils

Mustajuuri utility library contains very basic utility classes that are used in some of the plugins. This library contains a few classes that would belong to mjwidgets, but cannot be there (they would make mjwidgets depend hard on libmj and libmjdsp).

7.9

plugins

The plugins-directory contains more plugins for Mustajuuri. Some of these plugins are platform-dependent and will not compile on all platforms. For example plugins under “divaio” need DIVA IO library to function.

8 Directories

Various libraries are in use creating a tree of dependencies.

• divabase – base library containing threads, mutexes etc. – depends on POSIX

• divaio – Platform-agnostic wrappers for MIDI and audio I/O – depends on divabase and operating systems digital media libraries (audio and MIDI I/O device management). Uses “Audio File

Library” by Michael Pruett (of SGI) for audio I/O.

• ac – Auralization control library – stand-alone

• mjdsp – Mustajuuri low-level signal processing library

• mj – Mustajuuri base library – depends on divabase and Qt2

16

• mjmodules – a collection of plugins (also acts as a library) – depends on mj

• mjutils Utility classes

• plugins – more plugins – depend on mj and sometimes on other libraries

• utils – miscellaneous utility programs

– priogive – gives real-time priority to normal user processes – depends on POSIX

– cpuloader – loads host system, used for stress testing

– cdtrunc – Truncates audio files to length that can be recorded to a CD (the CD standard does not allow pieces of arbitrary length)

– rtchz – Test app that is used to check the host has low scheduling jitter with RTC (important in real-time apps on Linux)

9 File, function and data type names

To minimize the risk of naming conflicts with other libraries Mustajuuri uses simple naming prefix system.

All source code file names start with“mj ”-prefix. All globally accessible names (classes, structs, enums, functions, global constants) begin with “MJ ”-prefix. All class or struct member variables have

“m ”-prefix. The first letter of all data types (classes, structs, enums, typedefs) is capitalized. Lower-case letters start member and function names.

10 Development hints

This section gives some hints to developers interested in Mustajuuri.

10.1

Plugin programming

The best way to start plugin programming is to analyze the example plugins. These are included in the source distribution and they are also accessible via the Mustajuuri WWW-pages. We next examine the basic approach to making a new plugin.

First one creates a new directory and copies some other plugin’s source files there. The files and classes are renamed to guarantee they won’t conflict with other systems. Do remember to edit the makefile and change the plugin library’s name (variable “LIBRARY”). Otherwise the new plugin will replace the original. Once these steps are taken you can edit and extend the source code of the plugin.

10.2

About the compilation system

You need GNU make to compile anything with the make system. Most useful top-level (mustajuuri/src/) targets for make are:

• links - Updates the include links. This is needed whenever new header or template files are added to the base libraries.

• dirs - Create directories for the platform-specific stuff

• doc-all - Create html documentation of Mustajuuri and DIVA libraries

• doc-mj - Create html documentation of Mustajuuri

• all - Build libraries, executables and plugins (or what-ever is deemed necessary)

• dep - Update dependency files

17

• doff - Disable dependency generation. Used when the generation takes a lot of time and achieves little (between fast recompilations)

• don - Enables dependency generation

• clean, cleaner, cleanest, yetcleaner - Clean away binaries, backup files etc.

• tgz - Create an archive of the current state of the library.

In source directories (src/xyz/) the most useful make targets are:

• all - Usually builds necessary binaries and translations

• sbin - Create a statically linked executable (smain)

• dbin - Create a dynamically linked executable (dmain)

• slib - Create a static library (libxxx.a)

• dlib - Create a dynamic library (libxxx.so)

• doff, don, clean - as above

11 Further information

This document is an overview of Mustajuuri, not a detailed technical reference manual or tutorial. The

Mustajuuri WWW-pages (http://www.tml.hut.fi/ tilmonen/mustajuuri/) carry more technical information.

12 Future directions

While Mustajuuri has been developed on the UNIX programming environment it can be ported to any platform that has the necessary Qt support (Windows, Mac OSX, certain embedded systems). Right now there is a project to port Mustajuuri over to Windows.

The auralization control library (section 7.3) needs to be refined and extended so that users can use all features in Mustajuuri easily. To enable this we need to adjust the message-passing system to make it possible to send normal Mustajuuri messages across the network.

References

Boulanger, Richard (ed). 1999. The Csound Book: Perspectives in Software Synthesis, Sound Design,

Signal Processing, and Programming. MIT Press.

Cook, Perry, & Scavone, Gary. 1999. The Synthesis ToolKit (STK). Pages 164–166 of: Proceedings of the International Computer Music Conference.

Overview of New Features. In: Proceedings of the International Computer Music Conference.

LADSPA-team. Cited 1.5.2001. Linux Audio Developer’s Simple Plugin API (LADSPA). WWW-site.

URL=http://www.ladspa.org/.

McCartney, James. 1996 (August). SuperCollider: a new real time synthesis language. In: Proceedings of the International Computer Music Conference.

Morton, Andrew. 2001. Linux scheduling latency. WWW-page. Low-latency patches and information for the Linux kernel. URL=http://www.uow.edu.au/ andrewm/linux/schedlat.html.

18

SF.

2001.

Sonic Flow: dataflow-inspired audio signal processing.

URL=http://sonicflow.sourceforge.net/.

WWW-site.

Troll-Tech. Cited 1.5.2001. The Qt class library. WWW-site. URL=http://www.trolltech.com/.

Wright, Matthew, & Adrian, Freed. 1997. OpenSound Control: A New Protocol for Communicating with

Sound Synthesizers. In: Proceedings of the International Computer Music Conference.

19

Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement