VFLib: A collection of JUCE modules! (MIT Licensed)

[TheVinn]

I'm surprised to see it's GPL'd. Are you planning to offer a closed source compatible license?

At some point in the near future I will address that. But for now, until I'm done cleaning everything up, documenting all the important classes I'm going to leave it as GPL. I also need to get some legal opinions on the inclusion and dependence on the external modules that VFLib uses before I can make any claims to offering non-GPL licenses. Additionally, I'd like to get some more information on the utility of these classes: Are they helpful? Do they have any commercial value? What is the eventual popularity? Depending on all of this I may likely offer a dual license smiliar to how JUCE is presented.

Of course, your thoughts on the usefulness of these objects is welcome! Constructive criticism also equally welcomed - it seems I don't have quite the knack that Jules has for making things simple and clear.

[Graeme]

Cool, that's understandable. I'll keep an eye on it and if I end up using something I'll be sure to give some feedback. Keep up the good work Vinn!

[TomSwirly]

Just saw this, very exciting, TheVinn!

Regarding the GPL license or not - I believe that given what you have uploaded there, you can use any license you like or none at all. You aren't actually distributing any other libraries in your code as far as I can see - just talking about GPL'ed libraries does not GPL your code automatically.

[TheVinn]

Just saw this, very exciting, TheVinn!

Regarding the GPL license or not - I believe that given what you have uploaded there, you can use any license you like or none at all. You aren't actually distributing any other libraries in your code as far as I can see - just talking about GPL'ed libraries does not GPL your code automatically.

Yeah actually I am re-distributing all the libraries I mention with the exception of JUCE. That's one of the cool things about this library, it provides bZip2, sqlite, my modified Soci (tuned to work only with sqlite), FreeType, Lua, luabridge, and TagLib each as individual Juce-style modules.

Full sources for these external libraries are provided, some in amalgamated form (FreeType, sqlite, and TagLib). The project has no external dependencies with the exception of JUCE - everything compiles, and there's no need to fiddle around with compilation settings (just like JUCE). You do need to add a few magic lines to your AppConfig.h but these are documented in AppConfigTemplate.

On my to-do list is to improve these external modules so that they can use system-provided or user-provided libraries if they exist. For example using the native sqlite on iOS, or the native FreeType on GNU/Linux systems, instead of the copies that come with VFLib.

You don't have to use these external modules if you don't want to, just don't add those .c or .cpp files to your project. Some of my objects depend on them however. For example, FreeTypeFaces depends on FreeType but it gracefully compiles itself out via preprocessor directives if FreeType support is configured to off (VF_USE_FREETYPE in AppConfig.h)

None of the outside libraries that I've included are GPLed. And I don't plan on adding any GPLed externals, since I might want to move to either a more permissive license or a dual license (provide a commercial license like Juce) in the future.

[TheVinn]

Moved from viewtopic.php?f=2&t=8923

Code: Select all

thread::callAsync(this, &Foo:someFunction, 27, false);

SEXY!!! This looks exactly like CallQueue and ThreadWithCallQueue !!

I've encapsulated SafeBool in an easy to use template, and also put together a very robust Error object that uses it.

A significantly earlier version of this code is here. With the slightest provocation I'd be willing to put up the newest code...

Hmm...I just looked over all of your thread and callback code and it seems that we are addressing two completely different use cases (correct me if I'm wrong but you're creating a new thread to handle each callback).

The purpose of the objects in my library is to implement concurrent synchronization without using CriticalSection objects to protect mutable data, not to perform lengthy asynchronous operations (although I do have a facility for that, see the discussion of the idle function in ThreadWithCallQueue).

To this end I provide the GuiCallQueue for synchronizing shared data in the message thread, a ManualCallQueue for synchronizing shared data in the audioDeviceIOCallback (or any other system or library-provided thread), and a ThreadWithCallQueue for synchronizing your own thread that performs tasks in the background (for example, a separate thread which scans through a library of audio files and extracts meta-data).

I don't mean to sound boastful...it's just that I'm well, I guess you can say "highly competitive" lol

Tom thanks, upon consideration of your code and my targeted use-case I should document that functions placed into a CallQueue need to follow the same restrictions as code that executes in the audioDeviceIOCallback.

Although this should have been obvious to me after writing the example for ManualCallQueue

[Bruce Wheaton]

Good stuff Vinnie. I see a lot of generic solutions for problems I've had to deal with -AudioBufferPools, cross-thread object lifetimes with defined deletion threads, etc. I would probably have wanted to use them.

I don't see what you're doing with lua though. Lubridge implies you're mapping some classes to lua?

Bruce

[TheVinn]

Good stuff Vinnie. I see a lot of generic solutions for problems I've had to deal with -AudioBufferPools, cross-thread object lifetimes with defined deletion threads, etc. I would probably have wanted to use them.

Cool thanks. The library is tuned towards everything that an audio plugin or audio host author would want to do (including making cool user interfaces, see LabColour, FreeTypeFaces, and RadialImageConvolutionKernel). It consolidates all of the code that I've posted in the Juce forums over the years into a single unified library in the Juce module style.

I don't see what you're doing with lua though. Lubridge implies you're mapping some classes to lua?

Some of the external libraries in there are only being used by my private application code. I've included these external projects as individual modules so that someone can build a project that uses them in a way that is largely "idiot proof." I personally despise when I pull sources to an open source project, and then I have to go out and download a bunch of external dependencies. Ever have to deal with downloading and building libCurl, getting the Steinberg headers, getting some ODBC toolkit, etc.. yadda yadda? Yuck.

So with these externals I can build code on top of it and distribute the stuff in a way where the average JUCE user or plugin writer doesn't have to fiddle with downloading and configuring projects (or worse, install and run autotools, yikes!) Sometime soon the externals will be upgraded so you can use existing libraries when it makes sense (like the SQLite that comes with iOS).

As for luabridge, its a pretty nifty wrapper that lets you easily create Lua bindings for your own C++ classes. The author is no longer developing it so I am taking it over and cleaning it up since I use it in my own project.

On the other hand, some externals are needed for classes in VFLib. FreeTypeFaces requires FreeType. Rather than forcing the user to go out and download FreeType and configure include paths, I put the amalgamated version directly into VFLib, as its own module (vf_freetype). So you can get FreeTypeFaces working easily without any hassle.

The entire module vf_db (which is based on soci) provides a robust C++ front end for accessing a SQLite database. There is a vf_sqlite module which has the latest SQLite amalgamated sources included. So to use vf_db requires no additional effort just add the module and go! Now if you're on iOS, SQLite is built into the operating system and soon I will update my headers so that they use the operating system provided SQLite. vf_db is pretty cool, the following code actually works:

Code: Select all

using vf::db;
session sql ("mydatabase.db");
int id = ...;
string name;
int salary;

sl << "select name, salary from persons where id = " << id, into(name), into(salary);


I'm also planning on building some goodies on top of the externals. For example, a CompressedHeapBlock that extends the juce::HeapBlock to allow compressing and uncompressing of the data using the included bZip2 module.

VFLib requires that you add a few lines to your existing AppConfig.h, a template for doing so is provided in AppConfigTemplate.

[X-Ryl669]

Vincent, you should use @param in the template class declaration so the template parameter is documented.
This is useful to understand the role of each template argument in your library.

Anyway, I'm amazed by this work, and I'm wondering if you've written test vector for this code.
I know that writing test vectors for threading code is very boring/difficult, but from my own experience, until you do this, lock free code tends to fail in very specific way.

[TheVinn]

Vincent, you should use @param in the template class declaration so the template parameter is documented.
This is useful to understand the role of each template argument in your library.

Yep, I did that for a couple of the classes. The docs still have a way to go (its a work in progress).

Anyway, I'm amazed by this work

Thanks!

I'm wondering if you've written test vector for this code.

Yes, I did. I assume you're talking about CallQueue and Listeners. They are both rock solid (so is the ReadWriteMutex). The throughput on CallQueue is massive - both of the underlying memory allocators (with and without thread local storage) have been extensively tested, analyzed, and optimized.

I know that writing test vectors for threading code is very boring/difficult, but from my own experience, until you do this, lock free code tends to fail in very specific way.

Ain't that the truth! Solving the ABA problem without resorting to hazard pointers or 64-bit CAS was tricky, but it works. Rather well in fact, there's already a commercial app thats using it.

[X-Ryl669]

Solving the ABA problem without resorting to hazard pointers or 64-bit CAS was tricky, but it works

Can you explain in two lines how you did this ?
In my case, I've used deferred deletion with atomic ref counting (I know it's bad, but still, it works), so that the deletion of object always happens when the "userCount" of the container reaches 0.
I'm interested to figure out a better/faster way.

[TheVinn]

Solving the ABA problem without resorting to hazard pointers or 64-bit CAS was tricky, but it works

Can you explain in two lines how you did this ?
In my case, I've used deferred deletion with atomic ref counting (I know it's bad, but still, it works), so that the deletion of object always happens when the "userCount" of the container reaches 0.
I'm interested to figure out a better/faster way.

Actually, that's the same method I'm using. Except that instead of applying it to individual objects, I apply it to entire pages of memory (8 kilobyte blocks if I recall). I designed it specifically for the memory usage pattern of call queues and listeners, which tend to allocate and free in mostly FIFO order. There's a small percentage of wasted storage but its incredibly fast. When the reference count on a page drops to zero it gets set aside to "cool down" before being garbage collected once per second. The garbage collection executes in O(1).

This all happens in PagedFreeStore. The FIFO allocators build on the paged free store so they inherit the ABA-avoiding properties of the paged free store.

[X-Ryl669]

Clever.

I wonder if this scales up when the number of function to call overcome the page size (possible with large object pushed on the stack ?).

While reading your code, I've found 2 subtile bugs:

Code: Select all

    page = new (::malloc (m_pageBytes)) Page (this);
    if (!page)
        Throw ...
// and twice this:
   if (page)
   {
       page->~page();
       ::free(page);


The former will throw if malloc fails (in placement new), so the second Throw will never be reached.
The latter is exactly equivalent to "delete page;", except that it's not exception safe (if the destructor throws, you'll never free "page"). This is not really a bug, since you wrote the Page's desctructor, but it's an optimization.

[TheVinn]

I wonder if this scales up when the number of function to call overcome the page size (possible with large object pushed on the stack ?).

You could call it a feature / limitation. You're not supposed to pass big structures by value using the CallQueue. For that, create a ReferenceCountedObject and pass it by ReferenceCountedObjectPtr. I should mention that in the docs.

The former will throw if malloc fails (in placement new), so the second Throw will never be reached.

Something else I should probably mention in the docs. Exceptions thrown in VFLib are fatal, they indicate undefined behavior. There's no catching the exceptions thrown in the library, except to show an error dialog and exit.

[TheVinn]

While reading your code, I've found 2 subtile bugs:

Code: Select all

    page = new (::malloc (m_pageBytes)) Page (this);
    if (!page)
        Throw ...


Come to think of it, there probably is a bug in there! Nice find. Of course, that code path has never been tested (because malloc() never fails for me). It figures that the code path that was never tested has a bug (isn't that always how it goes?). I changed it:

Code: Select all

    void* storage = ::malloc (m_pageBytes);
    if (!storage)
      Throw (Error().fail (__FILE__, __LINE__,
        TRANS("a memory allocation failed")));
    page = new (storage) Page (this);


If you were to keep looking through the code and commenting, I certainly wouldn't complain!

[TheVinn]

Vincent, you should use @param in the template class declaration so the template parameter is documented.

Any assistance with documentation is GREATLY appreciated. I'm working on it now, trying to get everything down. If something is not clear or missing, now is the time to speak up because I would like to just get this over with and done and never come back to the docs again!