Blindly Coding (Posts about Camlorn_audio)

A Big Status Update and a New Project

Austin Hicks — Tue, 03 Jun 2014 21:37:00 GMT

I've been quiet on this blog for a good while. Life got really busy. Here's everything that's been going on. The most major piece of news is that I'm a founding member of 3 Mouse Technology. We're well under way at this point and I can finally talk about it publicly. I'm also deprecating Camlorn_audio in favor of a new project. More about both of these below.

Clang_helper and the new camlorn_audio

Austin Hicks — Tue, 25 Mar 2014 21:50:00 GMT

Clang_helper, here, is now useful and is also extracting everything available in the OpenAL headers except for comments. It's still a bit rough around the edges, but is definitely workable, especially once I write a better abstraction around C types. Currently, they are strings, all be it standardized-ish ones.

C++ support is unfortunately nontrivial, and probably won't be happening soon. I believe, however, that I can get it understanding and describing C structs. The problem withC++ support is that it is incredibly hard to abstract things without losing meaningful information. As the primary purposes of this project are automatic code generation for languages supporting C FFI and automatic code generation of C functions into C++ classes, this is not a current goal.

The Camlorn_audio approach is going to be a bit unusual: wrap C++ classes into C functions automatically without human intervension, store information on them, and then spit out bindings-at all points in this process, the Camlorn_audio build system will have information on the functions it's generating for itself, and won't need to read the class. Basically, there will be some templates and some yml files, and code will come out the other end. Classes will be declared in yml instead of C++-this gives me sanity, for the moment.

That said, expect me to backtrack on this in a few months and actually implement C++ support into clang_helper. It would be cool if it could, but I am currently at a loss as to how to not lose necessary information. C++ is incredibly complex, and automated C++->C generation in the general case is a hard problem. Simple things like vector<int> start to illustrate this-what about vector<vector<int>>? How about vector<string>? Each of these quickly becomes a special marshalling case as to how to convert the data. It probably requires some sort of dynamic pipeline that can reconfigure itself, at the least. The only way it can be easily done is to limit oneself to some subset-C++ classes that make use of only pointers and C types shouldn't be so bad, and basically staying away from templates is required. Good luck on finding libraries meeting them in the wild, though. Things you can't touch include, well, the Stl. There go half the nice things in the language.

If my current rate of programming skill increase continues, I may be able to handle this in the near future. They say that the worst programmer you know is yourself from 10 years ago; the worst programmer I know is myself from 6 months ago.

Also, the point of not using swig for Camlorn_audio which already handles that difficulty: being able to do things Swig can't, by storing meaningful information (for example, throwing exceptions in languages that support it and being able to not need precompiled C extensions for some things). The meaningful information is not included with C++.

I stand at the point of being able to begin on the Camlorn_audio rewrite, and believe that it will go much faster than the first time. It will probably also be an order of magnitude smaller: the scripts to generate things should be on the order of a few hundred lines, not a few thousand. It will still depend on Boost for the time being, as well as Libsndfile (removing the Libsndfile dependency is a goal; there's no good way around Boost until everyone has C++ 11 capable compilers). I will probably not use SCons. I might write my own small build system, as I've probably got a fairly major nonstandard build step.

In regards to the MMO: I'm trying to get this done first, as the MMO keeps running up against "X needs to be a feature in Camlorn_audio". Most notably, callbacks. I'll discuss my plans for dealing with callbacks another time once I finish figuring them out.

Rewriting Camlorn_audio with libclang

Austin Hicks — Mon, 17 Mar 2014 19:24:00 GMT

Quick update on Camlorn_audio: it's being rewritten.

here's why:

The current code works, but features are missing. This is because everything must be hand-typed, changes must be made across the entire codebase, and the OpenAL spec must be frequently consulted.
The current code works, but has bugs. Most of these are related to openAL error handling, and a few to threads.
Features that need to be implemented can't be implemented without rewriting streaming, which is the only nontrivial code in the entire thing.

In about 3 hours this morning, I succeeded in writing a 50-line script that turns all OpenAL macros into JSON. In about 6 hours this weekend, or when I get the time, I can probably follow it with a hundred-line script that does the same for all OpenAL functions.

A few things fall directly and indirectly out of such metadata:

All low-level classes for anything, now and possibly in future, can be trivially generated. This is easily 4 or 5 hundred lines.
All low-level classes can trivially have their functionality changed, i.e. error checking improvements, better threading, caching/recording for cloning, serialization...etc...
The bindings for Python can be auto-generated. So too can the bindings for your favorite language, whatever that may be.
Properties can be set by name, i.e. from a string.
You can query the metadata directly, getting lists of OpenAL properties and their ranges/possible values.
By folding the documentation into the metadata (human-generated, unfortunately), documentation can be released in the native format of the binding's language. Furthermore, for languages that do not support run-time documentation queries, functions can be provided that return the documentation.
Tables of all OpenAL parameters and functions, as well as those of camlorn_audio itself, can be generated automatically.

The strategy for this is Jinja templates, JSON, Libclang, and a strange combinational approach between data and C++ code. That is, for manually generated things, I will interleave C++ and--probably--YML. The metadata will generate error checking, allowing documentation on it without intervention (again).

And yes, this is basically a compiler, minus the compiling. The compiling used to be the hard part, anyway. So far I've got macros. Macros and plans.