Generator : writing a new generator for AtmelStudio 7 IDE

Development
1

Hi there,
I would like to implement an extra generator for the AtmelStudio7 IDE (based on top of VisualStudio 14, with which it shares the Solution files structure).

As AS7 shares the VisualStudio base file structure, it would mostly be a specific port of VisualStudio generators : almost the same Solution files as VS14, lots of similarities in project files’ structure and content as well.

However as AtmelStudio is not likely to be developed further (due to Atmel’s acquisition by Microchip).
So, I’ll need to borrow most of the Sln files generation implementation, as well as the vcxproj generation (but with heavier modifications).

Toolchain testing as part of the tree generation

As AS7 toolchains (avr8/SAM/avr32) are meant to build bare-metal firmware (cross-compilation), I was wondering about implementing a testing phase where CMake checks whether the targeted toolchain is able to compile simple programs or not.
=> Does Cmake only check if the toolchain has successfully built the target or does it also perform some testing on the built executable ?

I’ve debugged cmake to inspect when this testing phase was done while generating Visual Studio build tree, I suppose I could borrow a part of it as well.

Is there a documentation to help writing Generators ?

I dove into VisualStudio generators a bit to understand their overall architecture ; however, do we have some piece of documentation which explains the philosophy behind generators architecture?
I’m asking this because we have cmGlobalXXGenerators, cmLocalXXGenerators, cmXXGeneratorOptions, cmXXTargetGenerators, and I’m honestly a bit lost with all those files!

Testing

I came accross the CmakeLibTests project, which I suppose packs tests for CmakeLib.
Do we have testing requirements for the Generators ?

As a conclusion, it seems there is still a lot of work ahead, I’ll be glad if I can get some pieces of advice to get started with it.

Regards

2

In general, the “extra generator” approach has been superseded by the file-api. The latter allows external IDEs (or plugins to them) to load information about a CMake-generated build tree regardless of what main generator is used.

Is your goal to get IDE integration, or to have the AtmelStudio7 IDE actually be the build system as the main generator?

3

The best option to me right now (because development of AtmelStudio has ceased some years ago, and its base VisualStudio toolset does not seem to have any CMake integration) is to make AtmelStudio load the generated files as if they were written natively by it in the first place.

So I’d like AtmelStudio7 to be the main build system.

On the other hand, the only real requirement in terms of toolchain selection is to select the right one depending on the final target (let’s say an Atmega328P, which is Arduino Uno’s main chip for instance, will require the toolchain to be set to avr-gcc which comes bundled with AS7 installation).

As a result, AS7 does not require to be provided with the toolchain’s path : it already does that kind of resolution job for us.

Here is a sample of what AS7 expects to find within its .cproj files:

<PropertyGroup>
    <SchemaVersion>2.0</SchemaVersion>
    <ProjectVersion>7.0</ProjectVersion>
    <ToolchainName>com.Atmel.AVRGCC8.C</ToolchainName>
    <ProjectGuid>dce6c7e3-ee26-4d79-826b-08594b9ad897</ProjectGuid>
    <avrdevice>ATmega328P</avrdevice>
    <avrdeviceseries>none</avrdeviceseries>
    <OutputType>Executable</OutputType>
    <Language>C</Language>
    <OutputFileName>$(MSBuildProjectName)</OutputFileName>
    <OutputFileExtension>.elf</OutputFileExtension>
    <OutputDirectory>$(MSBuildProjectDirectory)\$(Configuration)</OutputDirectory>
    <AssemblyName>GccApplication1</AssemblyName>
    <Name>GccApplication1</Name>
    <RootNamespace>GccApplication1</RootNamespace>
    <ToolchainFlavour>Native</ToolchainFlavour>
    <KeepTimersRunning>true</KeepTimersRunning>
    <OverrideVtor>false</OverrideVtor>
    <CacheFlash>true</CacheFlash>
    <ProgFlashFromRam>true</ProgFlashFromRam>
    <RamSnippetAddress />
    <UncachedRange />
    <preserveEEPROM>true</preserveEEPROM>
    <OverrideVtorValue />
    <BootSegment>2</BootSegment>
    <ResetRule>0</ResetRule>
    <eraseonlaunchrule>0</eraseonlaunchrule>
    <EraseKey />
    <AsfFrameworkConfig>
      <framework-data xmlns="">
        <options />
        <configurations />
        <files />
        <documentation help="" />
        <offline-documentation help="" />
        <dependencies>
          <content-extension eid="atmel.asf" uuidref="Atmel.ASF" version="3.40.0" />
        </dependencies>
      </framework-data>
    </AsfFrameworkConfig>
  </PropertyGroup>
4

I’d like AtmelStudio7 to be the main build system.

Okay. In that case you’d need to implement this as a full generator and not an “extra generator”.

development of AtmelStudio has ceased some years ago

I don’t think upstream CMake should add new support for an unmaintained environment.

That leaves you with some options:

(1) Implement the generator in your own local version of CMake. We can at least point you at the appropriate code to get started.

(2) Construct appropriate platform modules and toolchain files to use these compilers with an existing generator.

5

I don’t think upstream CMake should add new support for an unmaintained environment.

I agree on that point.
I don’t really know about the needs of the AtmelStudio community for a dedicated Generator, maybe that’s a feature that no one ever asked for, maybe I’m the only one who needs it (…).

However, I wonder if we can still make some kind of plugin for CMake, something that won’t be supported by CMake’s main development branch, but available for any user who wants to generate AtmelStudio7 configuration files.

Keeping that idea in mind, it might be interesting to have a plugin which allows to use Cmake for other embedded platforms such as STM32, Texas Instruments. But that’s another story !

Regarding the generator’s implementation, I have debugged Visual Studio generators quite extensively now and managed to get a fairly good overview of their inner workings.
I was planning to create :

  • cmLocalAtmelStudio7Generator → inherits from cmLocalGenerator
  • cmGlobalAtmelStudio7Generator → inherits from cmGlobalGenerator
  • cmLocalAtmelStudio7TargetGenerator → inherits from cmLocalTargetGenerator
    and rely on their base class to handle generic processing.

However, I saw that toolchains and compiler are resolved using Modules/CMake.*.cmake files, which will sooner or later require something like Windows-AVR-C.cmake (…) I suppose this is what you refer to in the (2) point.
I am not really confident with this part right now, any help appreciated :slight_smile:

6

embedded platforms such as STM32

FYI, there is another thread about that: Generator for eclipse plugin for stm32 microcontrollers

I saw that toolchains and compiler are resolved using Modules/CMake.*.cmake files

Yes, though much of that is for command-line generators. For Visual Studio generators those modules have dedicated code paths, such as this one. In that case the modules are detecting what VS selects as a compiler rather than finding a compiler from the PATH or something.

which will sooner or later require something like Windows-AVR-C.cmake

Yes. Those modules typically contain information specific to their platform/compiler/language combination. In your case if the toolchain can only be used with the new generator you may not need that much specific information in the corresponding modules and can instead hard-code it in the generator itself. You’ll have to choose which as things progress.

7

Yes, the STM32 thread was up and running few months ago, however no news for now …?

I have very little spare time to work on AtmelStudio generator, nonetheless I started to work on it, essentially reusing Global/LocalVisualStudio10Generator’s code (.atsln solution file is an almost exact copy of .sln files).

Using a third party library (MIT licensed) : PugiXML (or any other)

However project (i.e cmake targets) files are quite different between AtmelStudio’s implementation and VisualStudio’s one.
I saw a lot of custom code to generate XML elements within VisualStudio Target generators and as vcxproj and AtmelStudio .cproj files both use the same microsoft xml file scheme (basic stuff, a subset of XML specification is supported), I would have like to use a dedicated library for that, such as PugiXML which seems to implement what I need, using MIT license.

Could I add this library as a dependency of CMake build or should I use/reuse something else instead ?
As this lib is provided as plain c++ files only, we cannot rely on artifacts for it, we have to include it as part of the build process, compiled only for the targets which depend on it.

What about a plugin interface for more exotic IDEs?

In the meantime, assuming I manage to get something working, is there a solution for CMake to use plugins? I mean plugins as cmake extensions, not present by default in cmake source code but loadable under some conditions (using a fixed plugin interface) , which may ease community-driven developments for unmaintained IDEs such as Atmel Studio.

I am asking this because this does not seem to be a modular solution for me to fork the full CMake code and just add my own code to it as this code will just lag behind and someday become completely unusable.

8

We made the decision long ago to not constrain ourselves to a stable API internally. There will be no support for such plugins. You will have to maintain a fork.

9

I suppose I’ll go this way then. Thanks for your time.

10

Hi there, it’s been a long time since our last exchange.

As advised earlier, I’ve been working on a dedicated CMake fork for this project whose aim is to provide support for the not-dead-yet Atmel Studio 7 IDE (which has been rebranded to Microchip Studio 7, still receiving support from former Atmel and Microchip companies!).

I am glad to share with you this new generator implementation, which is available for Windows X64 platforms.
You’ll find release for this fork at bebenlebricolo/CMake-AtmelStudio7-compatibility with a working release (3.19.3) based on the latest release of CMake mainstream project.

At the moment, AVR GCC 8 bit projects are supported (C language), with all the good stuff CMake provides such as out of source build, project linking.
AVR-GCC C++ needs a bit more code (especially in CMake Modules/scripts) in the aim to support C++ language for AVR 8 cores.

As usual with this kind of development projects, there is still a lot of work to be done in the aim to enable support for AVR 32 cores and ARM cores, however I tried to provide a solid backbone to support further developments.

Many thanks to the CMake community and development teams.

Bebenlebricolo

11

Hi!
Now that Microchip rebranded Atmel Studio to Microchip Studio and there are some support for PIC seems that Microchip Studio is more alive than ever. Is there any possibility of integrating this branch in mainline CMake?

12

A merge request would need to be created (or resurrected) first for review. Anything from Jan '20 probably needs updates to account for changes between then and now.

13

Hi all,
I regularly update the fork with new CMake releases (3.20,3.21…) so as long as I keep the repo up-to-date compared to the CMake mainstream counterpart, I think it should go fine.

However I’m not sure I can do a MR from github to gitlab directly ?

Also, as in my mind it was excluded that this fork were one day to be merged with mainstream cmake, I made some breaking decisions while designing my components.
For instance, I introduced 2 submodules :

  • PugiXML : in order to generate the AtmelStudio/MicrochipStudio .atsln files
  • Googletest : in order to test the new components without using CTest, which I’m not really used, and because I generally tend to prefer the gtest interface.

There will be a time to look at those points in details for sure, but prefer to let you know before I actually try to get the code back to CMake’s repo.

14

I don’t think so. Please open an MR on gitlab.kitware.com for the change.

Why is it needed to generate the XML? I don’t think a dependency for this is necessary. Even if so, a submodule is not suitable and it would need to follow the same patterns for other third party libraries (git merge -Xsubtree).

I don’t think there’s any need for GoogleTest; please look at using CMake’s existing patterns for testing itself.

15

We need to generate the .atsln and .cproj/.cppproj/.asmproj using the same kind of xml template than what VisualStudio generators do.
In facts, AtmelStudio/MicrochipStudio is directly derived from a VisualStudio2015 ide, with few modifications on it ; so I reproduced what I saw in the VisualStudioGenerators.

Visual Studio generators write their own version of xml files (.sln, .vcxproj, .csproj and so on) using the cmXMLWriter, which is the custom-made CMake implementation of xml, which is fine.

I don’t remember exactly why I switched to an external lib to do this, maybe I wanted to try something new after the first exchanges in this ticket.

The same applies to GTests over CTests and vice-versa.
I was looking for a testing framework a little bit more feature-rich than what CTests provides (at least from what I saw in some CMake test files).
I don’t want to criticize how CTests works however, it just made sense to me at that time since the fork was meant to live its own life.

We might still revert (or adapt to existing) the changes I did that diverged from CMake’s philosophy, but I think that might be a substantial amount of work ahead.

16

GoogleTest is a test framework (that is, it lives in the code under test). CTest is a testing harness that executes tests of any kind (provided their primary interface is via a process execution). We’re definitely going to continue to use CTest, but what is GoogleTest gaining us here? We need to test CMake’s command line interface for the most part for generators, so the existing test suite should do, but additional tests for behaviors specific to this generator should be added if relevant. What are you unit testing that GoogleTest is useful versus the patterns used in Tests/CMakeLib? (I admit to not having perused the code at all so far.)

17

I understand the way it is used, and I saw it can even post back test results on the webserver (don’t remember exactly where, but I had to remove this from the codebase to prevent unecessary posts from the tests I’m running within the development pipelines). And that’s great !

Don’t take me wrong, I’m not even remotely asking to stop using CTest to the profit of GTest. There is no sensible reason to do so.
Gtest does not provide a whole load of features that CTests doesn’t : it simply provides a macro-based interface to instanciate test cases, test suites, and other test configuration.
I’m not saying that GTest (or any other framework) is better than CTest. I just felt at that time I was more comfortable with GTests than with CTests.
That choice of course is just a matter of taste (pun … intended ?) and could be reverted, it just requires some time to do so.

Regarding what I’m achieving with GTest is not really related to any framework, or harness, at all.
I’m using unit testing to check the components internally ; for instance I’m not testing how the whole generator workflow works (i.e. I’m not triggering cmake with the command line at all), but rather single
components and implementations.

If you need an in-code example :
Check out the CMakeLib/testStringAlgorithms.cxx
and this test : CMakeLib/testCmAtmelStudioTools/testAS7DeviceResolver.cpp
→ functionally both implementations are achieving the very same thing ; but the custom implementation of assert_ok is removed to the profit of the generic asserts and expects of GTest (and this removes the need to log the console manually, this is handled by the framework).

I’m really testing core components, their inputs and their outputs, without the whole orchestration of CMake.
For instance, I have a collection of components which are designed to parse AvrGcc flags (-O, -w, -l, …) and catergorize them under the right section (Warning, Optimization, Debug, …). As AtmelStudio/MicrochipStudio provides some levels of abstraction for some of the most common flags of avrgcc (-Wall, -Wextra, …).
The IDE itself provides a gui with checkboxes for a limited amount of flags, and those flags are encoded in the .cproj (xml file) using a convention like:

<avrgcc.warning.extra_flag>true</avrgcc.warning.extra_flag>

This test suite implements what I depicted above.
So I’m testing that the components responsible of parsing the flags from CMakeLists.txt are indeed able to convert the “-Wall -Wextra” flags found in CMAKE_CXX_FLAGS variable into the aforementioned xml.

So yes, this could be implemented in CTest as well, with more work in case we want to get the same level of information than the one we get out-of-the-box from GTest.
E.g. : the ASSERT_TRUE(condition) macro reports the actual value of the condition if false, plus some details about the location of the line of code in case of failure.
That alone motivated me to use GTest over any other framework.

As a bottom line, testing framework is (almost) essentially a matter of taste and conventions. CMake uses CTest in production. And I get it, that’s fine, no issue with that.
If we’re merging that code with CMake, we’ll need to strip the GTest out of the port. And so we’ll strip out the PugiXML out the port as well, that’s part of the deal.