As part of a new project, to automate a new product the team came up with, I always have to write screeds of framework code. To do all of the basic stuff, like connecting to a resource, handling the resource being unavailable or slow. Authenticating, sometimes using more than one mechanism, and then, finally pulling and pushing large chunks of data over what are invariably slow links.
And every time I do this I find code written in a language I’m not using, or if it is, it’s code that is not really going to save me time in the long run because it’s either incomplete, pants or better-yet just plain does no longer work. I envy people using frameworks that seem to “have it all” and get their jobs done in a fraction of the time. Why do I choose the hard products and environments I ask myself? Well probably because I enjoy the challenge, but after a while, I feel guilty.
Stealing half-working code off of github, thinking this will be worth contributing back to on the project only to get no responses from the developer when you do submit patches, I wonder if they are just very busy, just like me, and is all of the world hard at work. Perhaps all inventing the same wheel? Or, is it just me.
I find “all” a very big term. Sounds more like marketing.
Also context matters. If you have little to do “all” is achieved easily. If not, like you, it may take a while.
I don’t judge this other people. Maybe they are right in their context, maybe they overlooked something and feel frightened somewhen later (I hope for their sanity, that is more unlikely).
I cannot make all the gaps unseen all tools and frameworks have. But, similar to life, even they are compromises and its our demanding task to fill those gaps.
Also I consider any framework or tool just a base from which I start, which enables me to do more demanding stuff than e.g. repetition. Its not a magic wand which makes my work gone and getting wage for free.
I spend half the day this morning hunting for a python language wrapper to Teamcity you see, I only found 2 projects, and one quite old and was not supporting token auth, and the other was just loads of hard to read semi-generated code with loads of non-working self-tests, bleegh. I’m not complaining, just observing, because sharing something you coded up at work is often not legal, and often too much work anyway because someone might have a different platform in mind. And target platforms do impact the way people code, a lot.
I’m celebrating, because now I’m writing an api wrapper, but using a tutorial on how to model/pattern it to be clean and tidy for a change, instead of my normal organic api wrapper messes. So I’m winning in that respect. I also have had to read through 2 other implementations of wrapper and learned a bit about the gotchas of the teamcity api in the process.
I’m with you on this whole context matters thing @sebastian_solidwork . Making sure that the tool I write does just as much as it needs to, without the fluff or functions that someone else maybe needed. For sanity sake I often go about deleting code paths I don’t use if I steal code anyway, because anything I delete will still be in my git history. But less is more to me.
Mostly, I hope that other testers who find a tool gap, can take courage, from the fact that this does happen often, and it’s not just you who has to spend (waste) a week writing nice tidy scaffold code, refactor it a few times and then only move on to the next task.
I think this is due to most projects being “simple”.
I’ve seen projects where you just have to automate a webapp. Very simple a few pages and yea basically any framework will get the job done.
Most apps I’ve tested didn’t require anything out of the box or special to automate. Maybe here and there a specific flow but not much more then that. Remember that it’s not the goal to “automate everything”
Can you give an example of this?
Currently one of my projects is a PAIN to automate. It’s the first time I’m seeing it due to bad development. But we decided not to automate due to these reasons, a single “E2E” flow in the UI of like 5-8 pages took 3 weeks lol.
Even with frameworks, you can’t always have it all, but framework helps out with boilerplate stuff and orchestration. You could at least opt for a framework that is flexible and multi-platform/language supported/adaptable.
For me when I went evaluating such over making my own, I came to conclusion that RobotFramework does most of the general framework stuff you would need, has lots of existing test/automation libraries to use/integrate with, and you can easily wrap/integrate your own custom tools or integrations across a set of languages - some first tier supported languages natively supported, and second tier ones via remote library server interface. I saw the benefit and need and helped build/foster some of the 3rd party remote library servers for a few languages.
those dealing with some aspect of hardware, niche industries or functionality likely don’t get the benefit of off the shelf solutions/framework to “have it all”
I suspect, that a few of us choose to work in industry sectors where things are not simple. That has one added bonus, you gain specialist and deep knowledge, and you can command a better salary. Your job is more challenging, but you thrive when the puzzles you get sent at work are not easy ones. I am a slow but steady solver, but sometimes things do feel a bit stacked against me. Industry does move slowly in my world anyway. I just love hardware and software together that you can physically see, sure Saas and Cloud are great, they are their own thing. But very little beats motors turning and actual goods rolling off of an assembly line. It’s much easier to visualise your app being used to produce physical goods there. For example an app that just sells stuff, is middleware, it adds value, but it’s not the actual “value” itself. Middleware can come and go, but mining and extracting ores, or planting and harvesting for food on the table, that’s my bag.
So yeah @kristof , my current project is a from-scratch CI/CD with all the bells and whistles, but in a much more hardware-centric business. And E2E automation can be done in C++ or C#, but compile-time is not as flexible as interpreted languages like Java and Python which can handle hardware interface changes in their sleep, and are suitable for proper environment and varied versioning integration tests. So yes, the product has too many apps, and has too many hardware components to really test all the angles, so I’m choosing a narrow vertical slice to start off. I need to write installer wrappers, pull artefacts from jenkins and from teamcity as well, then ensure that all the firmware and hardware talks and has it’s own separate LAN, and then work out how to control the hardware as well to reboot it and monitor it via special channels. It’s actually great fun and I’ve now done this in a few of my roles as a complete E2E.
The biggest PAIN is that the unit test code is in Pascal, C, C#, and one other language that I cannot even describe here. So using that in a joined up fashion is beyond the scope of any one off-the-shelf framework. And so I end up glueing code into a larger framework. I’ve been doing similar things for 18 years now, every “Framework” out there is usually missing a vital ingredient. My current strategy is to start really small, do lots of manual testing and then automate slices as you go. I find it’s so easy to automate something and then discover a fatal flaw in your approach a few months later that invalidates a lot of work you did. So Slow but steady wins the race. So for now it’s a smoke test only, the release cycle is very fast too, so no pressure, we love a bit of urgency
@daluu I once used Robot, (17 years ago) I assumed it had gone out of support ages ago. I found it was a bit simple for some things, is it still an active project?
Yes, it does help to use an existing framework, until you grow out of it, it’s more reliable for starters and will get you a lot more value than wasting time to build your own, while you are still learning the requirements and the real defect areas that need more excavator-power. Yeah, remoteing libraries, and parallel processing or async processing itself are big things missing an all of the basic frameworks. Events and “promises” are a basic human right, in any framework now.
I haven’t actually actively used RobotFramework (RF) when I first looked into it nor ever since then. But have been passively following the RF community and news. And as far as I can tell, things have improved since the early days.
For your needs of something cross language, cross platform, the remoting library/server support should offer an easier way to “glue” tools/code to the framework, by wrapping the tooling with the libray interface and serving/interfacing it remotely to RF. There is support for C#/.NET. I’m not aware of any for Pascal or native C. You could probably interface to C via cpython to RF, or interface to C via .NET and serve that to RF as remote library. Worth re-evaluating if you ever need a framework to use today.
And for those who can use off the shelf stuff, RF has a decent amount of libraries produced by the 3rd party community.
Side note, it was also fun and interesting trying to build the RF remote library server implementations in other languages to offer support to RF for those languages. Would be nice if someone in OSS community did a Pascal server or C server. I did the initial PoC implementations for Java, .NET, PHP, golang that others later on improved upon or superseded my work.
I did use the Robot Remoting framework, but that was 17 years ago, to drive some C++ code.
It worked really well and I was able to extend the template code to drive a USB tester rig called a CAT-C. Yeah also some IronPython I am sure was used, I was new to python at the time. Glad they never asked my to actually build the interface library myself, I’m much more confident with these things than I was at the time.
I believe the Chrome team used Robot Framework as well at some point, so that might be why it got some attention as a tool. For the python users out there, I found a great blog series by someone called Eric, it’s pretty pro level learning stuff around pytest mostly. https://pytest-with-eric.com/
I definitely understand that feeling of inventing the same wheel over and over again but putting just enough difference into it. Like your wheel has Zig-Zag patterns but I need Zag-Zig patterns. It drives me crazy.
I’m not sure of your ecosystem, but could you wrap the template into a library or some type of ‘boiler-plate’ code? That’s what we ended up doing and we now have created a ton of libraries that will solve for this. They’re abstract enough to where you don’t need all the bells and whistles, but still allow to pass in your specifics. Although, thinking it through as a I type, all our dependent libraries/products, we also built wrapper libraries around those to make it standardized and easier to work with.
So this time last year I was the copy/paste slightly modify, but working with the enterprise and getting a “Year of Quality” sponsored at the organization, I got a lot of prioritization and a group of people to help build out these libraries to make it so we can get into a more One Size Fits “All”.
These days, I try not to overthink it, but most of the time, when I write some test wrapper code, I’m asking myself. Will this code need touching in future? If so let me write in such a way that it is extensible.
Like right now I’m writing a wrapper to control a power supply. Most of us use a basic supply, which is the same as the test bench, but I worked out that if I write a facade/interface, and factory pattern, then, later when we want to use a power supply from a different maker, (the one we currently use is super old actually), all I need to is write the new implementation , register it with the factory, and then edit the test.INI file to specify the newer power supply. This might feel like work, but as you say, if you get into the habit, it’s not such a pain.
I’m going to use a module I found on the web, but like you say I need the module to actually also log what it does, so I have to modify their code anyway. A ZAG-ZIG so to speak. What this has taught me is to write code that is far more open for extension than I normally would. Now if only I was fully allowed to and had time, to publish it as well.
As for copy-paste, probably no way of getting very far away from that.
Yeah that was my biggest hurdle was getting the allowance for time on these aspects. I got really lucky we hired a new Chief Architect and he is ALL about quality and was blown away that we didn’t do anything around it here and really gave me a ton of support and had the fights with the C-levels to get this prioritized. I get I am so completely lucky with that, and I am counting my blessings, else I may still be struggling to get things as far as we have.
well, today, that’s my job @sharmon . I started a new job you see. After making devs do all of the testing, we have a lot of really good test setup, but much of it grown stale because there is nobody to keep it running. I have been given a lot of power to make changes, but also a lot of responsibility. Being the only tester, does still not mean it’s all my responsibility, but am loving learning the product incrementally and then exploring what will work best for what.
Which is the big reason I am shopping for new tools, at some point I might buy an ethernet simulator of some kind, but one that is more than just bandwidth throttling, so yeah, lots of decisions. But also lots of very tiny steps.
That’s awesome congrats on the new dig! I know I’m still a newbie compared to others, but I remember when I started the big quality push for my company it felt like a uphill battle everyday but the smallest wins are still victories and we should celebrate those and that really helped keep me motivated!
