How to make users of a library depend on its plugins

In a project I work on, one of our dependencies, OpenSceneGraph, has a system where one component, osgDB, provides a uniform interface for loading lots of different types of file, and the loaders for individual files are implemented as plugins. At runtime, when an application using osgDB asks it to load a file, it searches its plugin directory for a shared object or DLL (depending on platform) for a plugin that can handle the file, then calls the appropriate function in that plugin. It’s got another system for when it’s statically linked, but that’s not important right now.

Users of osgDB link with osgDB, and the plugins link with osgDB, too, but the user and osgDB itself don’t link with the plugins as that’s handled at runtime.

I want to specify that when something linking with osgDB is built, if any of the plugins are out of date, they’re built, too, but there’s no need to build them in any particular order as there’s no dependency until runtime.

The best idea I’ve had so far is something along these lines:
add_library(OSGPluginsPseudoTarget INTERFACE)
target_link_libraries(osgDB LINK_INTERFACE_LIBRARIES OSGPluginsPseudoTarget)
add_dependencies(OSGPluginsPseudoTarget ${USED_OSG_PLUGINS})

i.e. add an interface dependency so that anything linking with osgDB transitively depends on the plugins, but osgDB itself doesn’t (as that would create a circular dependency as the plugins depend on osgDB). Unfortunately, this doesn’t resolve all the circular dependencies as the plugins depend on osgDB, so now also depend on themselves.

I’m aware I could specify the dependency separately on everything else that links with osgDB, but that would be several other targets and end up messy and less maintainable.

Hopefully I’m missing something obvious, but I’ve been unable to find another way to specify that one thing depends on another only at runtime.