JUCE LV2 Plugin Wrapper

[jules]

Jules, would you consider adding this to the source tree when ready?

Sure, thanks very much!

[falkTX]

ok, things are advancing where I feel soon I'll be ready to show the lv2-wrapper to everyone.
with the juce-2.0 modules now is a great time to introduce new features I guess...

Jules, to make LV2 plugins work juce needs some small mods (check for Build_LV2 macro and 2 new Processor functions), here's a patch:

Code: Select all

diff --git a/modules/juce_audio_plugin_client/utility/juce_CheckSettingMacros.h b/modules/juce_audio_plugin_client/utility/juce_CheckSettingMacros.h
index a942404..fcb9e02 100644
--- a/modules/juce_audio_plugin_client/utility/juce_CheckSettingMacros.h
+++ b/modules/juce_audio_plugin_client/utility/juce_CheckSettingMacros.h
@@ -26,7 +26,7 @@
// The following checks should cause a compile error if you've forgotten to
// define all your plugin settings properly..

-#if ! (JucePlugin_Build_VST || JucePlugin_Build_AU || JucePlugin_Build_RTAS || JucePlugin_Build_Standalone)
+#if ! (JucePlugin_Build_VST || JucePlugin_Build_AU || JucePlugin_Build_RTAS || JucePlugin_Build_Standalone || JucePlugin_Build_LV2)
  #error "You need to enable at least one plugin format!"
#endif

diff --git a/modules/juce_audio_processors/processors/juce_AudioProcessor.cpp b/modules/juce_audio_processors/processors/juce_AudioProcessor.cpp
index 03e0ab4..4d797f5 100644
--- a/modules/juce_audio_processors/processors/juce_AudioProcessor.cpp
+++ b/modules/juce_audio_processors/processors/juce_AudioProcessor.cpp
@@ -246,6 +246,16 @@ void AudioProcessor::setCurrentProgramStateInformation (const void* data, int si
}

//==============================================================================
+String AudioProcessor::getStateInformationString ()
+{
+    return String::empty;
+}
+
+void AudioProcessor::setStateInformationString (const String& data)
+{
+}
+
+//==============================================================================
// magic number to identify memory blocks that we've stored as XML
const uint32 magicXmlNumber = 0x21324356;

diff --git a/modules/juce_audio_processors/processors/juce_AudioProcessor.h b/modules/juce_audio_processors/processors/juce_AudioProcessor.h
index f5c224c..37866c2 100644
--- a/modules/juce_audio_processors/processors/juce_AudioProcessor.h
+++ b/modules/juce_audio_processors/processors/juce_AudioProcessor.h
@@ -524,6 +524,12 @@ public:
     */
     virtual void setCurrentProgramStateInformation (const void* data, int sizeInBytes);

+    //==============================================================================
+    /** LV2 specific calls, saving/restore as string. */
+
+    virtual String getStateInformationString ();
+
+    virtual void setStateInformationString (const String& data);

     //==============================================================================
     /** Adds a listener that will be called when an aspect of this processor changes. */


plugins will have to add support for get/setStateInformationString if they want proper lv2 state save (otherwise only normal parameter values are saved in the host).

[jules]

Ok.. but I don't understand the need for the string methods. Can't you just convert the binary data to a base-64 string?

[falkTX]

Ok.. but I don't understand the need for the string methods. Can't you just convert the binary data to a base-64 string?

Yes, it's possible, but since LV2 uses strings by default it feels kinda awkward to go back to binary data (and thus we don't have to worry about alignment, endian size or any other "problem" when using blobs)

Note that lv2 presets are on the host-side, by using *.ttl files. This means the data can be read by users, so for this it also makes sense to have strings.

[jules]

But if you just convert the binary data to/from strings, then all existing plugins will work without modification, which is surely a good thing?

[falkTX]

But if you just convert the binary data to/from strings, then all existing plugins will work without modification, which is surely a good thing?

Yes, but not very much in the long term.
I understand that you may not want 2 new functions just for LV2, but afaik the 'transition' is very easy and fast. All the plugins code I saw so far used juce XML to save/load data, so it gets really easy to support the new calls.

I spoke to the lv2 developers about this, they will really appreciate if strings could be used instead of blobs. Main reason for this, quote: "Short answer, blobs are not portable"

[jules]

Short answer, blobs are not portable

Piffle! The reverse is true.

If you're concerned about your stored data being passed safely between different platforms or hosts, then you'd be *much* wiser to pass it around as a binary blob. The "black box" nature of binary data means that there's no danger of the host buggering things up by reformatting it, trimming whitespace off it, changing the newline characters, screwing up the character encoding, etc etc! As soon as you put the data into a format that the host might be able to tinker with, then you're just asking for trouble!

[drobilla]

LV2 developer here.

Um, that is obviously not true because you can put non-portable data in a blob. This isn't much of a stretch, since even an int is non-portable.

To be able to save portable state, the plugin API must know whether or not data is portable. Without any other information attached to it (like a type), an opaque binary blob is not portable data. An XML string is portable data. You can't assume a blob is portable when it might not be, since this will fail catastrophically.

If you state is already XML text, the obvious ideal solution is to simply save it as XML text. This is elegant, transparent, convenient, and portable.

Base64, while supported, doesn't actually help, because it doesn't make non-portable data portable, it just encodes it differently (and with overhead). The real world impact of this is that the plugins would not save portable state, which means hosts are actually pretty likely to just not save it at all in session files: trying to encode architecture is difficult and not really worth the effort, it is assumed plugins can save a portable version of their state. The option to save non-portable state (e.g. blobs) is mainly there for fast in-process snapshots and the like.

I am not at all familiar with JUCE and what guarantees are baked in to the API, but in general it all boils down to a simple requirement: the plugin must be able to tell the API with absolute certainty whether or not the state is portable. If you already have portable data in a nice standard textual format, converting it into binary to save on disk is insane.

[jules]

Hello!

I think we might be using the word "portable" to mean different things. Perhaps you can define "portable", and give an example of exactly what is being ported, and what the source source and destination are between which this porting is being performed..?

When I say "portable", all I mean is that if I write a plugin that runs as a VST on OSX and also as an RTAS on Windows and a LV2 on linux, I want to write just one portable piece of code that stores my data.

And you must admit that it's just silly to say "XML data is more portable than binary data" when you're talking about an opaque piece of content! The plugin gives the host a lump of data that only the plugin itself can understand, so as far as the host is concerned, it doesn't make the slightest bit of difference whether this data is encoded as binary, string, XML, or piles of sausages. The host's job is not to manipulate the data's content or magically make it "portable" (whatever that means). The host just is simply responsible for returning the same lump of data when the plugin asks for it.

In fact, most JUCE plugins actually do store their state using XML. The wrapper classes provide utility functions that convert the XML to the cross-platform binary format which gets passed to the host. But many plugins NEED to use binary because they're storing enormous data-sets (sample banks, etc), so binary is the only practical solution for the base storage format.

[drobilla]

Portable with respect to hardware architecture. The goal with saving state is to save the state in a way that a host can reload it later. The problems come from the fact that the host could be running on a different platform when it is restored.

For example, say I save state on an i386, throw that session on a USB key and restore it on a different x64 machine. If the state is just a binary blob, which is not portable (it may contain integers and pointers, for example) this is not only impossible to do correctly, it will fail catastrophically. It's not even possible to know you can't safely restore that state! The best you can do is just try and hope it doesn't get you corrupt data, or crash the entire process.

It's not a coincidence that most JUCE plugins (and VST in general) actually do save to a textual format like XML, because this is by far the easiest way of saving portable state. Doing endian conversion and making sure all your structs and whatnot are portable with respect to alignment and sizes and all the rest is a heck of a lot harder than just saving to portable formats (e.g. text-based stuff like XML). As you say, nobody wants to write a bunch of code to handle all those different situations, they just want to write one implementation that works everywhere. Text is the obvious solution, which is why pretty much everyone does it.

That said, sure, saving binary data is necessary some times - but it is absolutely crucial that the host knows which is which. It is not a matter of the host magically making anything portable, it's a matter of the host even knowing the data is portable at all. To be able to restore data, the host and/or plugin needs to know what that data is.

It is not at all silly to say XML is more portable than an abitrary binary blob. What an odd thing to say! XML (unless you encode non-portable binary data in it, anyway) is a completely portable text-based format that has nothing to do with architecture, you can save it on i386 and restore it on ARM or PPC or x64 or whatever. This is not true of an arbitrary binary blob. It seems you aren't actually considering the reality that multiple architectures exist - the integer 4 is not the same "lump of data" on different computers. If only life was so simple...

An API that saves state only to an opaque binary blob with no type/portability information associated with it is flawed because the saved data may be tied to the current architecture, which effectively means the entire session is tied to that architecture and can't be safely distributed or loaded elsewhere. LV2 does not duplicate this flaw, but supports portable state properly, so hosts can at the very least by know if it is safe to save/restore state. To support this, the plugin (and thus JUCE) must tell the host what type the data it is saving is (or at least whether or not it is portable). This is not really LV2 specific, it is inherent to providing portable state (it is why Apple uses property lists, which have typed values, for many things including AU state. Apple actually had to care about portability. VST is just plain broken in this respect.).

[jules]

For example, say I save state on an i386, throw that session on a USB key and restore it on a different x64 machine. If the state is just a binary blob, which is not portable (it may contain integers and pointers, for example) this is not only impossible to do correctly, it will fail catastrophically.

So it's literally IMPOSSIBLE to pass any binary data from one architecture to another? Wow! Good job you warned us.

Hopefully you're viewing this website on a CPU that has the same endianness as the one that created its (binary) image files, otherwise your browser may fail catastrophically at any moment!

[drobilla]

Of course it's possible - sometimes. However, if you don't know whether or not the data is indeed portable, then the only safe assumption is that it is not. Therefore, via such an interface, it is impossible. That is the problem.

My browser can read those files because it knows what format they are in, which is why the relevant protocols convey that information, exactly how LV2 state works, and exactly what the JUCE API needs. It is pretty telling that your own example illustrates this: the web could not possibly work if it was all binary blobs with no type information whatsoever. If not a type, then at least a flag to indicate whether or not the data is portable is necessary. Without that, yes, it is impossible.

P.S. Binary isn't necessarily non-portable, those images work because their format is well-defined, like wav files, and any implementation will load them on any platform (one of the many advantages of using standard formats over meaningless binary gunk). Binary vs. text is not the issue here, the issue here is knowing the type of data, or at the very least knowing if that data is portable.

[jules]

Of course it's possible - sometimes. However, if you don't know whether or not the data is indeed portable, then the only safe assumption is that it is not. Therefore, via such an interface, it is impossible. That is the problem.

You're right that it's impossible to guarantee that a plugin will behave safely. But it's not your job to try to rectify that situation.

A plugin shouldn't crash. End of story. If a plugin emits some data and then crashes when it re-loads it, then the plugin is breaking its contract, and that's not your fault. I'm not sure why you seem to think this is a problem that you need to solve?

[drobilla]

I don't intend to solve it. Plugins are free to save whatever, even if it's completely non-portable. That is certainly a stupid idea, but it's not my job to enforce it. It is simply a fact that the host needs to at least know whether or not the data is portable for things to work correctly. You can do this via rules (like "the data must always be portable", but you don't seem to agree with that either), or do it via allowing the plugin to specify which. You can't allow arbitrary non-portable binary data and not even have a mechanism to say so. How is a host to know the state isn't suitable for distribution (e.g. as part of a preset or example session) or loading on a different machine? It makes all plugins pay for the disadvantages of a few that make the dubious decision of storing non-portable state, and in this case it does so when the actual state is usually XML text, which instead of saving as plain readable text gets bloated out to 3/2 the size in base64 encoding (I pity anyone who has to debug such a thing). Plugins are using sane portable state (XML) with all the desired qualities, then it's being crammed through a broken API so that as far as the host knows it's a bunch of non-portable crap, and we're all just supposed to trust that the plugin is actually saving the current architecture along with this data (highly doubtful and extremely difficult to do precisely in general) and dealing with all cross-platform issues itself (unlikely). In a perfect world that might be wise, but not in this one.

I am a bit shocked I have to argue this one, to be honest. Why would you choose something that requires such lofty unrealistic things of plugins (all that binary portability stuff), when a simple superior alternative provides all the benefits while requiring almost nothing of plugins (specify a type or a flag) and allows the host to avoid any potential problems whatsoever? That is not a sensible decision. What is the advantage? What the heck could the disadvantage be of allowing the host to avoid problems by not restoring non-portable state on the wrong platform? What could the disadvantage possibly be of specifying a type (or some flags) along with data? It doesn't seem like I'm the one that needs to be doing the justifying here...

[jules]

You can't allow arbitrary non-portable binary data and not even have a mechanism to say so. How is a host to know the state isn't suitable for distribution (e.g. as part of a preset or example session) or loading on a different machine?

You can allow this. Because every VST/RTAS/AU host that has ever been created has allowed this. And that's all been working just fine for over a decade without your help, thanks very much!

Remember PPC/intel? Remember all the agony when people tried to load a Cubase Mac project on an intel machine and all the plugins crashed? No? Me neither, because that was never a big problem.

There's no reason for the host NOT to assume that state data is safe to load on another machine. Preferably the API would stipulate this requirement explicitly, but it's a bit of a no-brainer when you're writing a loadState() function that the damn thing shouldn't crash, for f**k's sake!

If you were writing an OS, would you stop users from transferring binary files between different machines, because one or two badly-written apps might have been written without taking endianness into consideration?

Why would you choose something that requires such lofty unrealistic things of plugins (all that binary portability stuff), when a simple superior alternative provides all the benefits while requiring almost nothing of plugins (specify a type or a flag) and allows the host to avoid any potential problems whatsoever? That is not a sensible decision.

Lofty and unrealistic?? Seriously? That's pretty patronising. People aren't complete idiots, you know.

What the heck could the disadvantage be of allowing the host to avoid problems by not restoring non-portable state on the wrong platform?

If you KNOW for sure that this state is non-portable, then that would be fine.

But the only way to KNOW whether a plugin can safely load something is to force it to make it implement a canStateBeSafelyLoaded() function. And if you're going to make the plugin do that, why not just tell them to do this check internally in their loadState() function so that it doesn't crash? Because that's what 99.9% of programmers have already been doing for years!!

Look, I've no time to argue any more on this thread, but the bottom line is this: It doesn't matter what you or I think. The reality is:

- Some functionality is impossible without a binary storage format. End of discussion. I've seen many VSTs which store 10s of megabytes of compressed data. If LV2 can't do binary, then some plugins simply won't be able to run on the LV2 format.
- Nobody in the real world gives a rat's-ass what LV2 does anyway. Even if you come up with the world's best, safest, mollycoddling API that gives seamless magical cross-architecture data, few people will care or even notice, because everyone has already written and tested all their storage functions to use binary blobs on VST or AU, or are using JUCE's XML format wrappers. There might a dozen people who only write LV2 plugins, and who don't port them to other platforms, so that's your audience.
- If LV2 (or any other plugin platform) provides a string storage interface, then of course JUCE's XML wrapper should use it directly, rather than unnecessarily transcoding XML->binary->string. That's trivial to do, and not in question. But I doubt if any of the users would notice.