A Libaudioverse Status Update

Libaudioverse is my attempt at making 3D audio for native apps, something which is currently lacking in the typical indie price range. There are other systems with fully featured content management pipelines, but these are often overkill for small projects, especially in terms of budgets. If you know Web Audio, you are already basically familiar with most of the core concepts. While there are differences, general architectural experience should be broadly transferable from one system to the other, and Libaudioverse aims to solve many of the same problems as Web Audio.

That said, this post is primarily aimed at those already familiar with Libaudioverse. I will probably have an entire series about what makes Libaudioverse relatively easy for one person to maintain at some point, but this post is about the short-term and long-term future. It recently came to my attention that there are people outside my small circle of developer friends who might be interested in knowing where I stand and where I'm going, so I thought I'd write something up.

This post will proceed roughly by major feature, discussing the status of each. The short version is that Libaudioverse isn't something that I want to call stable, but I'm certainly interested in getting feedback at this point; it's working well enough to be useful and the API is pretty settled.

Core Features: Nodes and Audio Routing

The core of Libaudioverse is an audio routing engine that is capable of allowing the user to build up chains of nodes. Each node is responsible for performing some action with audio: reading from files, filtering, analyzing, etc.

This is complete and has been for a long time. Nodes run as expected, and there are no major bugs in it to my knowledge. It needs a minor rework to deal with a few edge cases and minor bugs, but that's it. This part of Libaudioverse is where the scalability for multiple cores will eventually go, but this has intentionally been delayed, probably until after 1.0. Libaudioverse is fast enough to be useful as-is, and threads add a great deal of debugging complexity.

Basic 3D Audio and HRTF Support

3D audio in Libaudioverse is orchestrated through sources and an environment. At the moment, there is one node for each of these: the SourceNode and the SimpleEnvironmentNode. At a lower level, one may work directly with a so-called multipanner, a node which is capable of using anything from 7.1 surround sound to HRTF through headphones. On a mid-range laptop, getting a hundred sources using HRTF in real-time is a reasonable goal when Libaudioverse is built in its release configuration using SSE. The absolute theoretical maximum on my machine is somewhere around 300, but this fluctuates as all good benchmarks do.

I would also call this part of the library complete.

Occlusion and Reverb

These are two closely related features. Occlusion refers to how sounds sound when a wall is between you and the sound, and reverb to the characteristics imposed on a sound by the room in which it is being played.

Occlusion is simple and not implemented. All that is needed for occlusion is a lowpass filter, something which I have. The problem is that it is not yet evident what API I should use in order to allow easy integration with a game engine, nor do I have anyone lined up with a fully working game engine capable of performing the necessary calculations to see how well my model of occlusion works. When someone gets to the point of wanting it, occlusion can be implemented very quickly. This basically boils down to whoever gets there first helping to define the public API, followed very shortly by me automating and inserting a lowpass filter.

Reverb is a different story. The reverb is an ongoing project that is about half done, though I expect that we'll be improving it for a long time. Reverb is mathematically complicated and difficult to optimize, as well as involving a great deal of what can only be called art. Reverb mostly consists of me waiting on an inspiration; it's not the sort of thing that can be designed ahead of time, and the options and development choices for designing a reverb are not documented in anything less than research journals and digital signal processing textbooks.

Finally, it should be noted that this is a reverb aimed at games, not musicians. I suspect that I shall need to code a musical reverb in the future, as Libaudioverse certainly has applications in that area.

Basic Waveforms, Filters, and Effects

This is another area that is mostly complete, but I suspect that few people will want it right off the bat.

Libaudioverse currently implements sine, square, and configurable noise generators. The square generator is currently sub-par, with noticeable aliasing at high frequencies, especially when swept.

Oscillators may be used as control signals by connecting them directly to properties of other nodes. In addition, it is possible to automate properties in much the same way as Web Audio.

In terms of filters, Libaudioverse implements two convolvers (ordinary and FFT) as well as the famous biquad section and a node capable of representing any IIR filter given its coefficients. The FFT convolver is fast enough to handle impulse responses with lengths on the order of seconds, and the biquad filter section is sweepable. Less interestingly, a ring modulator is available.

Three types of delay lines exist, two of which are currently exposed. The third needs a node definition. The two that are currently exposed are the crossfading delay line and the dopplering delay line. Crossfading delay lines have built-in crossfaders, handling delay changes without clicking by crossfading to the new position. Dopplering delay lines handle delay changes by pitch bending the audio. The third type of delay line is called the interpolated delay line, and exists to allow properly implementing such effects as chorus.

Planned effects that don't currently exist include waveguides, chorus and flangers. Beyond these, this area of Libaudioverse is very open to requests from people who need specific features for their work.

Bindings

Libaudioverse has a fairly complete toolchain for generating bindings. This includes documenting in the native format of the target programming language.

Libaudioverse currently binds to Python. Other languages are pending requests, as I will require feedback and testers at a minimum.

Getting a new language up and running completely and keeping it running for the foreseeable future is only a few days of work. Because of my custom tools, new language bindings practically maintain themselves, needing me to handle only the few special case classes that do not fall under the broad category of "node". The one downside here is that I have to write the initial self-maintaining bindings myself, pending some sort of tutorial on writing a backend for the toolchain.

The only major language which I do not plan to do in the near future is Java. Java is going to require custom JNI code and, save for Android, simply isn't worth it. I would like to take this moment to point out that Java has the worst method of talking to C libraries I could possibly come up with, short of saying it's impossible. I am aware of JNA, which helps to avoid these issues. Unless I am really out of touch with something, Java does not have a large share on the desktop platforms. The question of JNA's status on Android was uncertain at last check.

Three languages I want bindings for in the near future are Rust, Haskell, and Go. All of these languages have vibrant communities and very minimal offerings for game programming; I therefore consider it commercially advantageous to target these markets.

In order for your language to work with Libaudioverse, it must be able to handle C function pointers, pointers to pointers, and thread safety. This last implies mutexes and being able to be called from threads not created by the language itself.

Platform Support

The core of Libaudioverse is written in ANSI C++11 with an external dependency on Libsndfile. This means that it should compile as-is on all three major desktop platforms.

The major blocking factor is cross platform support in my library audio_io, which does not have Linux and Mac audio backends at this time. Writing these backends should not be a large project, though Linux will require learning ALSA. The prospect of learning ALSA fills me with dread; in usual Linux style, it is only half documented, and there are sure to be system differences.

iOS will require a great deal of work. Libaudioverse is likely to compile for it, but the iPhone will need hand-optimized ARM Neon at best and hand-optimized functions written in pure assembly at worst. The reverb is currently expensive even on desktop platforms, providing a further challenge to iOS porting. I intend to release 1.0 and then work on iOS; Libaudioverse will work quite well on desktop platforms, and most bugs will effect all platforms.

Android is shelved for the time being. Libaudioverse requires SIMD to run on phones, and Android is split between at least 3 CPU architectures. In addition, Android requires JNI. As the icing on the cake, however, Android is also known for audio latency issues. Taken together and coupled with the wildly varying power of Android devices, these three mean that it will be difficult for me to provide a quality experience on Android. It may not even be possible.

Documentation

Documentation is somewhere between 50% and 75% done. All nodes are documented (in fact, it is impossible to create a new type of node without documenting it). All functions are also documented, including signatures and individual descriptions for all parameters. Even the enumeration members have reference tables.

I have a rather sizeable basics section (which probably needs some reorganization) and about half the tutorials I want to have.

Finally, there are examples in both Python and C; I intend to port the Python examples to all new high-level languages, though admittedly the C examples exist primarily for testing the library.

Compared to most projects done by one person, my documentation is in pretty good shape.

Licensing and Contributing

This is the one everyone wants to know about, and also the one I need to verify. I intend to consult with a lawyer before 1.0, the first commercial version.

For using Libaudioverse in commercial applications, it looks like I can simply send you an e-mail with some permission statements. Libaudioverse is GPL V3, which specifically has clauses not invalidating any additional permissions I give you. My current plan is to essentially give out a one-time static linking exception that must be removed if code is to be distributed further; this should in theory give you the power to use Libaudioverse in your own commercial application(s) while not allowing you to turn around and effectively sell Libaudioverse yourself (because you can't give out additional permissions along these lines for as long as I hold copyright).

Allowing contributions requires a contributor license agreement. This is something I do not have the expertise to even try to do, so I will need to consult with a lawyer. In order to continue having the power to license Libaudioverse commercially, I must either be the only contributor or get all contributors to sign contributor license agreements saying I can use their contributions. I am very interested in doing this, but am unsure as to the administrative overhead. Given the amount of time I have poured into Libaudioverse at this point, I am being super paranoid about accepting any possible pull requests or similar.

I have yet to decide on a pricing model, but think hundreds of dollars at most. Libaudioverse will always be free for those projects which are capable of abiding by the GPL.

This will all be sorted around the time I put out a call for testers; I intend to give dedicated testers free or discounted licenses, depending on my pricing model and what the tester wants it for. Since testers will need to have a close relationship with me for the duration of testing, I expect this will be somewhat personalized. You can of course test by using Libaudioverse in your GPL application or game, a state of affairs which I'd be more than happy to see happen: I'm at a point where I need feedback to continue.

Conclusion

Probably a month or two to what I'm going to call 0.8, 1.0 when I sort out the legalities and the library seems stable.

Feel free to use it now, though you do need to build it yourself. If you're good at programming, the examples should serve to get you started. I have at least one person using it to develop an FPS at the moment, and so far everything has been at most a half an hour fix. I will get anything reported against GitHub, or you can e-mail me directly; in addition, there will be a mailing list in the near future.