I’m developing some plugins for an architecture desktop application called Solibri.
The plugins are some forms of geometry algorithms, architecture rules to output some correlations between architecture components, visualized in 3d view.
(jar file, called rule in this application) https://solibri.github.io/Developer-Platform/
The plugin’s code is written in Java and application data is bim data (component’s 3d vector data) provided in ifc type of file.
How should I build an automation unit test for these plugins instead of manual checking on the main Solibri application ?
I cant read the ifc data without opening the main software Solibri, so cant implement CICD without human checking by eyes. Anyone have experiences with bim data or testing architecture applications ?
I have a biased opinion because I am building my own software to do desktop application testing, so excuse me if this sounds like a self-promoting post.
I truly believe that the best way to build testing around a desktop automation is to start from Manual testing. Havine an excellent understanding of what needs to be tested is step 0
Identify appropriate test cases for automation: Not everything will require to be tested, but there are main pathways which will require to be tested. Going from manual to automation requires you finding these.
Select a suitable automation tool: This is a personal preference but there are definitely some industry’s best practices out there.
Create a testing framework: Lay down the rules to follow to handle different test cases, failing tests, etc
Write automation scripts: This is where you’ll actually have to do a lot of work - simulating actions like a user would
Test and debug the scripts: sounds like a joke, but you obviously have to test your tests and ensure everything runs smoothly
Integrate with the test management system: for best results and tracking, you need to orchestrate your tests into a system which lets you see and analyze your results after each run
Gradually transition to automation: Don’t go from zero to automation. Take a hybrid approach, where you can automate the most obvious things first!
That’s an interesting case thank you for sharing the details.
Consider how automation can support your testing activities - can you use tools and scripts to optimize parts of the setup of the scenario?
Virtual machines can set up test environments/desktops with various browsers.
you can use scripts to vary the bim input, if applicable.
I know low code RPA tools that can navigate the browser controls and do simple image recognition and screenshots. Alternatively, use a script to fire simple screenshot tools and compare the screenshots from the previous runs.
The whole idea is to more the actions of you testing from something handcrafted and unique to something repeatable. Happy tinkering - I hope this helps!
BIM = Building information management I take it, that huge next thing optimizing our built environments to save energy, increase safety and cut emissions, its must be so cool to be working on a system like that. I feel your pain, so I’m trying to go back in my mind about 17 years when I worked on a CAD app. Unfortunately we had no real test automation, at least not for the GUI. But if I got offered that job today, I would work around the problem. A bit like a lioness wanting to catch dinner at a watering hole, but without becoming dinner for the crocodiles.
So it’s really hard to tell someone else what they would be doing in their shoes, you are hopefully getting loads of ideas. @okdashcam has dropped a really level headed approach in already. And when I read it I almost gave up trying to improve on it. Just take Orlies approach and you will have at minimum a plan you can easily talk about, I think. Especially point 7, take aim at anything that moves.
My approach would be to communicate a 2-part strategy to the dev team.
50% energy spent on manual test, tech-support analysis of past bugs and getting bug counts and severity down by measuring them. Build test environment helper tools like Jesper suggests, build tools that can auto-install and provision VMs , with the product and test data to speed up manual testing immensely.
50% energy spent on automation tool investigations, prototypes and a longer term target which you can prove works because you have some bug count metrics to back it up. Once you have some automation-test tool.
Just starting with “assisted” testing tooling like a tool that sets up a clean VM, with the product pre-installed and some example data files will save you loads of time in the long run. Always be looking for things that will save you time and pain. But, if it was me, I would do this one thing. Speak to the dev team and have them walk you through the code, even if the code makes little sense. Ask them about unit testing, and how they create mocks, get a feel for how confident they are in their code. Get them on your side - then spring the crocodile on them! I assume your app has debug logging?
Ask the developers to add some test instrumentation code into the app for you. Basically a hidden configuration value or flag that turns on a bit of extra logging. If the app uses log4J this might be really easy for them to do, basically add an analytics and testing log line every time the code does something interesting. Examples are a logline to know when the app has finished loading a bim/ifc file. Add another logline with metadata about the file, such as number of shapes, number of data inputs and number of exports/outputs for example. You can then use these loglines as events to trigger computer-vision checks (have a look at the appium eyes tool for ideas.)
Does Solibri offer any scripting or automation interface? I know for example, Adobe tools like Illustrator, Photoshop have a scripting interface that allows you to script actions/tasks against it using Javascript and Applescript. There is even some runtime facilities in the script engine for things like making HTTP requests.
I used a bit of that a long while back for something, including the HTTP interaction, but don’t recall the details, and don’t use no more.
I know some other apps offer such as well, and sometimes the scripting language exposed is Lua.
So that would be something I’d look into if Solibri offers such. And expanding on that, if there is such support in the base app, I’d look into exposing your plugins to the scripting interface, so that you could control actions/features of your plugin via the scripting interface. Granted going this route isn’t same as manual use/testing of your plugin in Solibri, but it sure is a lot better than using generic GUI automation against the whole thing, which is a lot harder to do effectively, and maybe more costly if using commercial tools rather than open source.
If Solibri doesn’t offer such feature, it sure would be nice if they add scripting interface in their future roadmap.
Was jsut thikign that when Testing that your app can export data is always just one aspect of testing the app and will not actually really give you that “MVP” feeling as a user anyway. So I’m with David on having a API or scripting accessible interface as my test surface. @smolshark has never come back to us since asking this question. Are there barriers that we do not see here?