#1 2014-04-08 20:25:05

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Querying module properties without loading the module

Please correct me if I’m wrong, but it appears that a module must be loaded to examine its properties. This would imply that modules need to be loaded to determine if you need them. If there are a large number of plugins, this could be a significant performance issue. Is there or will there be an efficient way to examine module properties without loading them? For example, on Windows a DLL’s embedded resources can be queried without loading the DLL.

Offline

#2 2014-04-09 09:06:36

sascha
Administrator
Registered: 2012-05-04
Posts: 45

Re: Querying module properties without loading the module

Offline

#3 2014-04-09 15:52:06

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Re: Querying module properties without loading the module

I'm curious to know your thoughts about the two different implementations you mentioned. That is embedding meta-data into shared libraries in file readable form such as appending a zip file to the end of shared libraries, or a framework that’s built on top of shared libraries to add file readable meta-data. An example of the latter approach is the Poco OSP zip-based bundles. What you do see are the pros and cons of these two approaches, and why do you choose the former?

Offline

#4 2014-04-10 22:55:44

sascha
Administrator
Registered: 2012-05-04
Posts: 45

Re: Querying module properties without loading the module

The decision to embed meta-data in form of binary blobs (a zip file) into shared libraries is not set in stone and I am also curious about the opionions of other people/users. The CppMicroServices library is actually the fourth generation of a OSGi - inspired library and the design principles were derived from the previously collected experiences and user feedback. The most important design rule for CppMicroServices could be loosely summarized as:

"Do not mess with the dynamic linker and its responsibilities."

In other words, the development model when using CppMicroServics should be the same as in a traditional C/C++ project, meaning just shared/static libraries and let the dynamic linker do what it can do best. Trying to do module dependency resolution and loading of dependencies at runtime within the framework/library added a lot of complexity and in fact duplicates some of the dynamic linkers capabilities and responsibilities. The benefits of a slightly more dynamic resolution and update process of libraries/modules at runtime did not prove to be worth the efforts (let alone the fact that we did not come across a real use case for this yet).

Using zip files as the units of modularization also added complexity to our build system (we had such a system about 5 years ago) and developers needed to learn about yet another modularity concept. Unpacking each zip file also added some overhead (you need to unpack it on disk in order for the dynamic linker to find the embedded shard library). Because of a large legacy code base, we also had a mixture of zip files and normal shared libraries, which needed to work together - adding yet another level of complexity. A system like Poco OSP based on zip files is probably quite elegant when you create a new project and the complete code base is build upon "zip bundles". It's been a while that I looked at Poco OSP but when looking at advanced features like bundle/module signing with certificates (which Poco OSP seems to support), unpacking a signed zip file and making sure that the unpacked contents is not tempered with is probably also more complex than working with "just" shared libraries.

The other important feedback we got over the last years is based on the fact the our C/C++ developers are in general quite cautious when it comes to introducing yet another library, let alone a "framework". A system based on zip files and its own dynamic dependency resolution mechansim just scared people and it was also hard to integrate into existing code bases. With CppMicroServices, we tried to create a library (instead of a framework) which allows for an easier migration path to a service oriented programming model for existing systems. This also means sticking as close as possible to traditional work flows.

In the case of module meta-data, all this leads to the requirements that the system should still be based on traditional shared libraries but should also allow to query meta-data without loading the shared library and all its dependencies into memory (in a cross-platform way). Qt5 has some support for this but the implementation is quite different on each platform and far from trivial (at least the last time I checked). Appending a zip file to a shared library seems to combine the best properties of both worlds. No introduction of new concepts like "zip bundles", the dynamic linker of the operating system can do what it is supposed to do, and meta-data can be read from modules (loaded or not) by just using the shared library as if it was a zip file (the zip file signature is located at the end).

If you have worked with Poco OSP and/or similar systems, I would be very interested in learning about your experiences.

Offline

#5 2014-04-15 14:06:41

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Re: Querying module properties without loading the module

Generally I prefer your idea of appending module meta-data to the module’s shared library. As you point out, finding a suitable solution is not obvious. However, C++ Micro Services has one advantage over the typical use of embedded blobs. A module doesn't need to have direct access to its own metadata. A module can retrieve its properties from the framework that can cache them before loading the module.  That means you’re free to come up with a solution that doesn't use traditional schemes (e.g. global statics, linked data sections or embedded resources). Such a solution would also avoids the module needing to redundantly load its metadata and potential interference with the module’s own use of embedded resources and data.

A relatively naive approach is to simply append the blob in such a way that it doesn't interfere with the module’s binary format and cache at the end of the blob its offset from the start of the file. I know this is more easily said than done.

Offline

#6 2014-04-18 07:32:16

sascha
Administrator
Registered: 2012-05-04
Posts: 45

Re: Querying module properties without loading the module

Agreed. And treating a zip file as a "binary blob" and appending it to a module would automatically provide the offset information and allow for zip file querying and retrieval without loading the module. Integrating this into the current code does not look too complicated and it is what I will try to do over the next weeks.

Offline

#7 2014-04-18 12:34:24

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Re: Querying module properties without loading the module

Excellent idea. I believe this would also allow for encryption of metadata for those who have security concerns. I’m eager to try this out. Please keep me posted.

Offline

#8 2014-08-12 14:34:17

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Re: Querying module properties without loading the module

Much thanks for scheduling this feature for release 3.0. Is there a due date for this release?

Release 3.0 related topic:
The release notes for version 2.0 mentioned Config Admin and Declarative Services will be worked on first. However, they don't appear to be included in release 3.0. Both are quite useful in a production environment. Do you know when these features will be available?

Offline

#9 2014-08-12 20:11:44

sascha
Administrator
Registered: 2012-05-04
Posts: 45

Re: Querying module properties without loading the module

To be honest, there is no specific due date for version 3.0. Almost all issues except for the resource loading feature have been resolved for 3.0. I did work on the resources system but couldn't find time to finish it yet. I am still aiming for a late September release though, after we got through all the testing for our own product using 3.0 features.

Config Admin is still high on the priority list (we have a few internal projects with requirements for such a service). Declarative Services on the other hand is something I would personally love to have, but its development is not directly driven by any of our current requirements. I also was not planning to couple the service implementations to the 3.0 release to no longer defer it. The development will very likely be based on the 3.0 release and either take place within the CppMicroServices project or get their own repository.

However, I am happy to incorporate external priorities into the planning process as far as possible.

Offline

#10 2014-08-12 20:44:39

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Re: Querying module properties without loading the module

Please let me be clear. I'm in no way advocating for additional release 3.0 features. I'd like to see it as currently defined as soon as possible.

The requirements for my current project are quite modest, and release 2.1.1 is adequate. However, other projects coming on line will require release 3.0, and will (over time) generate a large number of plugins whose configurations will need to be managed. So our need for Config Admin is probably similar to yours. My fear is that without Declarative Services maintaining all those plugins due to framework changes will be a nightmare. I would think you’ll have the same problem.

Offline

#11 2014-08-12 20:54:44

sascha
Administrator
Registered: 2012-05-04
Posts: 45

Re: Querying module properties without loading the module

Okay, good to know.

In our case, we have a large code-base developed over more than 10 years which we are gradually moving to a more modular and service oriented architecture. So currently we mostly use module activators replacing static initialization code, the resources system, and at some places the service registry. Service usage is slowly increasing and I fully agree that after a certain threshold, managing service dependencies directly by using the framework API will become unmaintainable. We are just no there yet, but it needs definitely attention!

Offline

#12 2014-09-08 18:31:55

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Re: Querying module properties without loading the module

I would think appending binary blobs should work on ARM and iOS as well. Are you aware of any issues in this regard?

Offline

#13 2014-09-08 23:44:05

sascha
Administrator
Registered: 2012-05-04
Posts: 45

Re: Querying module properties without loading the module

I am not aware of any issues on iOS in this regard, but I also didn't try the system on this platform yet. The Mach-O object file format is used both on Mac OS X and iOS and by looking at its documentation, all sections and segments are specified in terms of offsets from the beginning of the file. The current development branch (containing the new resources system) also passes all tests on Mac OS X 10.7 (my test system). It should even work with universal binaries, by adding the blobs at the end of the final universal binary, effectively sharing the resources between all object files for different architectures.

Offline

#14 2014-09-25 15:37:23

sascha
Administrator
Registered: 2012-05-04
Posts: 45

Re: Querying module properties without loading the module

Offline

#15 2014-12-08 15:00:35

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Re: Querying module properties without loading the module

I'm eager to use this new installed state. When will 3.0 be available with this feature?

Offline

#16 2014-12-15 02:21:33

sascha
Administrator
Registered: 2012-05-04
Posts: 45

Re: Querying module properties without loading the module

Work is currently being done on the state machine for managing state transitions properly. There will also be a change in the initialization code for modules, making it more robust in terms of (static) initialization scenarios. There is not much left but 3.0 won't make it before Chirstmas, I'm afraid. I will post a status updated in mid-January about this topic.

Offline

#17 2014-12-15 15:10:49

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Re: Querying module properties without loading the module

A useful feature related to this one would be filtering auto-loaded modules based on module properties; as well as, location. Either the client code or the framework will need to do this filtering. Better it be implemented once within the framework than differently by every client. What do you think?

Offline

#18 2014-12-15 23:03:27

sascha
Administrator
Registered: 2012-05-04
Posts: 45

Re: Querying module properties without loading the module

I like the idea in general but need to think more about the issues involved. Do you have a special use case in mind?

In the recent OSGi specs, there is a generic ReqCap model (requirements / capabilities) which I have been looking at and I am wondering if that would also "solve" our use cases. With such a mechanism, modules (auto-loaded or normal modules) would require specific capabilities in order to be resolved and these requirements would be matched by the framework against other installed libraries (or the framework itself, which could be configured with a certain set of capabilities). If the reqs are not met, the module would not be resolved and not be loaded at all.

Looking forward to your comments.

Offline

#19 2014-12-16 15:17:54

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Re: Querying module properties without loading the module

Having a module’s properties accessible without loading the module, allows a client to access data with little computational overhead. Of course, this could be accomplished through the embedded resource feature provided by C++ Micro Services. However, the name-value pair interface provided by the concept of module properties makes accessing date more convenient and amenable to other property related features such as filtering.

Two possible uses are data only modules and determining if a module should be loaded. The ReqCap model addresses the latter. Capabilities are simply properties and requirements are simply filters on those properties. Of course, the ReqCap model may layer on additional semantic behaviors, but underneath it’s simply employing properties and filters. Bottom-line, it’s great to use the ReqCap model to stay consistent with OSGi, but it’s also nice to expose the underlying module properties and filtering of those properties for other uses.

Offline

#20 2014-12-19 00:52:51

sascha
Administrator
Registered: 2012-05-04
Posts: 45

Re: Querying module properties without loading the module

Offline

#21 2014-12-22 17:20:02

WayneBailey
Member
Registered: 2013-06-12
Posts: 15

Re: Querying module properties without loading the module

How would ResolverHooks be used to address the following situation?

Assume a client wants to access module property values to be used as data, and those values are to come only from certain modules based on other module property values. In addition, the client would like to do this without loading those modules.

Offline

Board footer

Powered by FluxBB