Yes. I would expose the various backends as CMake options, but have some default logic to select a sensible backend in the event the user hasn’t selected one manually.
So you want to be able to build multiple backends simultaneously? You’re right, this strategy would not support that.