Go to bottom

Designing a sane scripting API

category: code [glöplog]
So, I am currently looking into embedding a scripting functionality into my software (currently considering using Squirrel, which has a similar C API to Lua). I've never done this before, and I have also never really use scripting in other programs, so I could need some hints on common pitfalls, good / bad design decision, etc for designing scripting APIs... - Since I'm rather sure that other sceners here have worked on stuff like this (e.g. in games), I want to ask if any of you are aware of any useful papers or other resources on this topic? Do you have any hints, best practices, and other things to share?
At least it isn't a pony.... -_-
there are many things to check, start with these 2:
- Be thread safe. You may want to script behaviors, and need to parallelize objects update
- Check debugging features. You're at the very beginning, how easy is it gonna be to debug or instrument hundreds of lines of script?
added on the 2012-12-08 16:09:55 by Soundy Soundy
I'd make sure that properties and function can be called at any time, for instance being able to set volume before loading a tune. As obvious as it sounds, I saw plugins failing at this and it get's tedious fast.
Another thing, don't offer too many ways to do one thing, eg setting volume through a property, a config object, on a clip,.. - this'll bite you on documenting and it's confusing, though might be only my opinion.
Look at other scripting APIs, what sucks there, what do ppl have send in as issues or bugs - how'd you use your own engine to solve a problem.

Take care :)
added on the 2012-12-08 18:06:09 by mog mog
Speaking of volume, yes, stuff like that is one design decision I'm not sure about yet. Looking at the scripting capabilities of Renoise, it's relatively easy there because it has a single document interface and there is always a song loaded - while in OpenMPT, with its multiple document interface, there can be any amount of tunes open, including no tune at all. That's why I thought that scripts should probably be bound to single songs instead of the whole application. But then again, I have no idea if that's a good decision, as accessing multiple songs at once would probably offer some cool features (like copying data between them).
I was also wondering if standard callback functions (like a script function that would be called whenenver a new sample is loaded) should have fixed names, or if I should make the user register a custom callback function. Stuff like that is what I want to know about, which decisions are better, which are not so good.
Hmm, that relies on your personal preference actually. Though defining custom callbacks has benefits, like handing the signal to two functions without having a third "handler" function.
For instance if you call "onVolumeChange" in my script, I'm bound to have the function named like this, and it may act as a gate when i actually want to call two functions. Instead something like:
Code: _api.on("volumeChange", updateAwesomeVolumeBar)

gives one the freedom to either bind one function to each event/signal, or have a handler function, though I have to somehow know "volumeChange" exists.

In general, have mandatory functions, like "onScriptEngineReady", "onShutdown",.. (I bet you'll find more descriptive names) available that each script has to implement, so "onModelReady(config)" is called in every script for instance.

I'll make some tea, and see you on IRC then :)
added on the 2012-12-08 18:47:52 by mog mog

script_pos: dc.l script

dc.l Glenz
dc.l wait,100
dc.l Tunnel
dc.l wait,100
dc.l MoreGlenz
dc.l wait,100
dc.l Goto,script

added on the 2012-12-08 19:07:57 by lsl lsl
saga: ask me at tUM ;)

some things which turned out to be useful:

*try to be as generic as possible - e.g., instead of creating a special function for setting each parameter provide a single function which accepts a parameter id and a value. This makes it really extensible, and allows for easier abstraction or mapping on the client script side
this applies to callbacks as well - registering is more generic than predefined functions (unless you make one big callback eating everything - like the win32 message handling. not great for scripting I´d say)

*being able to reload scripts at runtime (maybe even while running without an interruption) is great for both writing scripts as well as for debugging the script host

*similar functionality should behave the same way or even identical, this reduces the learning curve and saves writing extra documentation

*in particular, don´t reinvent a separate structure for every hierarchical level - use a common denominator and extend that (i.e., setting the volume might be done for the whole app, a track, a pattern, an instrument, a sample, or even a single sample value

*concurrent, nested or recursive calls should be considered the normal case when designing stuff - otherwise they are either not supported at all (but might still be used, creating confusion) or tend to break sooner or later

*sometimes, instead of a procedural approach, a data flow or event style interface might serve better

*initialization sucks, try to keep it as low as possible

*try to isolate errors - when something goes wrong it should not break every other scripted part as well

*use reasonable defaults, allowing t write scripts step by step instead of forcing you start ith dozens of parameters you don´t really have an idea why you need them here

*add a lot of working examples to the documentation =)
added on the 2012-12-09 03:26:36 by T$ T$
From the experience of writing several binding from C/C++ to Python and Lua... The binding itself is bare minimum, as low-level as it can be, sticking to the C/C++ implementation, with absolutely no fluff around.

However, I use the scripting language itself to build a nice API from this minimal binding. Things like
* operator overloading
* properties
* input parameter checks, catching & throwing exceptions
* being "Pythonic" or "Luaic" in the API. like accepting any kind of sequence
for a function that process a sequence of items, accepting either a file or a file path for a function working with a file, etc.

Those are doable from the C/C++ side of the binding, but it's lot of boring code to do and error-prone. Doing it with the script language is much easier, from my own experience. It turns out also that I never had much benefit of doing those in C/C++, as it's usually not the speed-critical hings.
Thanks, these are some invaluable hints. T$: We should definitely talk about this at tUM.
marmakoide: I'm not entirely sure about the last thing you said. Do you mean that it could be good to basically have an "ugly" but easy to write C++ backend and the "beautify" it using classes written in the Scripting language that are loaded when the script is loaded, to make the binding layer between C++ and the virtual machine as simple as possible?
I think he doesn´t mean "ugly" but rather "minimalistic" and I can second his advice after learning it the hard way
added on the 2012-12-10 02:36:37 by T$ T$
@Saga Music: just replace "ugly" by "minimalistic". An example : I used Lua to bind the C++ classes of my raytracer, to have a nice way to describe scenes. The Lua objects are 1-1 mapping to the C++ objects. By no means it's ugly, but it smells C++, not Lua. The C++ objects allow you to make a raytracer, but to build and render any 3d scene involve a fair amount of boring code (instantiate this and that, add this into that, etc.).

However, a special script 'setup.lua', is read before reading any scenes scripts. This 'setup.lua' script set many defaults, like a default camera. It provides a 3d transformation stack, and creating a scene object will be relative to the 3d transformation stack by default. The C++ side support rendering of a set of triangles, but reading 3d objects files is done in Lua. I could have done that in the C++ side, but it's faster done in the script language, and allows for great flexibility.
Alright. So basically, the default classes that are available to the script should represent the internal C++ structures as closely, and then I could provide a second layer in some setup script to make handling these structures easier?
I'm not even sure if a second layer would be really necessary when working with a language like Squirell, since it looks a lot like C++, but still has similar mechanisms to Lua.
+1000 to everything T$ said
added on the 2012-12-11 14:44:18 by ferris ferris
this might also sound a bit obvious/annoying/superfluous, but this is also a great task for tdd (once you've gotten some basic design in place).
added on the 2012-12-11 14:48:49 by ferris ferris
So, I haven't worked on this scripting stuff for a long time, but I'm digging it up again. I've stumbled across a problem that I should probably consider in more depth before even starting with the API design. Some of you might have worked with Renoise's (or other programs) scripting API. Since Renoise only ever has one and only one song open at a time, there's no problem with providing a single "Song" object on which the script author can work.

In the case of OpenMPT (for which this scripting API is supposed to be), there can be any number (including 0) of open songs, while one of them is always focussed.

Now the question is... how should I handle this? I've had the idea that a script is always bound to one specific song (but may also access other songs through some function), which implies that the script is created when the song is created (or afterwards) and deleted when the song is closed. The advantage is that it's easy to write scripts that operate on the song's data (e.g. patterns), because you always just get that one Song object and don't have to worry that it doesn't actually exist. I guess this is nice for scripts that run pattern transformations, sample modifications or whatever, because they are basically just "one-shots".

The other way would be that you always have to retrieve the currently active song object first, check if it's not NULL and then operate on it. I imagine this to be more complicated for the script authors, but I imagine it's also slightly more flexible.

If there should be some kind of global script that always runs in the background (e.g. a MIDI filter that listens to incoming MIDI messages and transforms them before they are passed to the active song), it would have to be implemented as a separate kind of script in the first case, while it would just look like any other script in the second scenario.
Of course, in the first case, there could be some kind of script that is automatically launched whenever a new song is openened, which would kind of "emulate" the behaviour of an always-running script.

Maybe someone has worked with similar scenarios in other applications and can give me some ideas? I currently prefer the first version, but maybe it's really stupid to do so.
Oh, and there's one more thing which I'm sure most people who have created bigger APIs must have thought before: Automating the process of generating the API bindings. Sure, I have to implement all API classes and functions on the C++ side of things, and then I also have to register the in my scripting language; I have to fill enums with values and whatnot.
If I do this manually, I essentially have to do the same work twice, rely on ugly X macros for setting up enums somewhat automatically... that's really a lot of error-prone work, so I'd prefer to automate this. I know there's stuff like tolua which would do the same thing for lua, but not for Squirrel (or other languages for that matter)... So, what do you use for automatically generating API registration code, and maybe even automatically generating API documenting? So far I've seen things like SWIG, flex or Bison, but that all seems to be rather complicated and over-engineered for my purpose. Maybe writing a (or using an exisinting) small C++ parser would work here? I mean, all I really want to parse are a few function and enum declarations inside classes...


Go to top