How I decide if a new AI tool is worth using

Conor Smart · November 22, 2026 · · 4 min read
I subscribe to a few AI tools, but I don't just throw anything into production anymore.
We've been using AI at Arklavo for about 12 months for marketing, customer service, production and more, and I do have a couple of paid subscriptions. But I only use tools that I know I can trust.
When I switched the team to every new tool
Any new AI tool meant a new way of working for the team and a whole new set of instructions (and validation of the work already done, according to the last set of instructions). I would change processes because there was a shiny new tool, assuming it would help the team (when I knew nothing about how well the tool would actually do what we asked it to do).
In retrospect, I think those changes probably cost the team more time than they saved.
What costing more than it saved looked like
For example, when it comes to product updates, I implemented the use of AI to help edit our products and product information.
Unfortunately, I rushed this process and didn't give the AI good enough guidance and instructions on how to make these edits, resulting in incorrect changes being made, which we then had to go back and manually edit and review for weeks.
This isn't to say that AI is bad; it's just a tool like everything else, and if you don't give it good instructions, it can actually end up being the enemy.
How I test a new AI tool on a real task
Testing AI tools on real-life tasks helps me to figure out whether they're actually useful or not. For example, I'll test out product information changes on just a handful of products rather than all products at once so that I can closely monitor the changes and make sure everything's been updated properly before rolling it out to more items.
I'm finding that the better guidance and instructions I give the AI, the better results I get out of it. Otherwise, it'll return equally vague results, or worse, changes that seem definitive but are completely incorrect. So rather than treating instructions as an afterthought, I'm thinking about them as an integral component of whether a tool will be useful to me or not.
Ultimately, I've done enough product information editing by hand that I know what it takes, and I know that meticulously checking changes made to a handful of items is preferable to having to go back and fix mistakes made to a whole lot of products.
What has to be true before the team relies on it
If it's just about managing ads and content, I'm not sure there's much of a case for an AI app at all given the native options already available on many platforms.
I'm not saying there isn't value in using more than one AI-powered app. But I am saying that before I'd implement any AI-powered app for my team to use, it would need to be tested with actual tasks on some of our products first. And before anyone hits publish on anything AI-generated, there'd need to be a human review step in there.
And again, my test would be pretty basic: Does the guidance provide clear and concise instructions? Does it produce a result that doesn't require guessing about what exactly was being asked for?
Where this test does not apply
Is your business one where AI will be touching live data that customers will see (like product data)?
If not, then this test will be of little consequence to you. If so, read on...
Do you have a catalog of products?
If not, what is the analogous task in your business that AI will be performing? That's where I'd encourage you to start. Again, if AI isn't going to be touching customer-facing, live data in your business, this test probably won't mean much to you.
I say all this to make clear that I'm speaking to those of us whose businesses include using AI to update data that customers see.
I still use AI. I think it has its place. I just don't believe novelty is the same as readiness anymore.
If you're considering implementing AI, start small. Give it clear direction. Check its work before it reaches your customers' eyes. Remember, AI is just a tool, and a badly directed one can turn into your enemy.
This article was first published on Featured News.