about betahaus
our innovation agencyLOG IN
Veronica Jonsson
October 15, 2018

Ask an Expert: Why Should We Automate Our Chatbot Testing?

Based in the chatbot hotspot Vienna, Botium is the startup that has set out to introduce a new industry standard for chatbots with their unique framework for automated chatbot testing. We had a talk with them to take the temp on their business and get the scoop on what’s next in the chatbot industry.

If you attended our yearly startup event BETAPITCH Global 2017, you might remember them as TestMyBot, the startup that entertained the audience by bluntly answering the judges' question on what they needed to reach their goal with an “Oh we don’t need money, we already have customers; we are pretty much ready!” Press fast-forward one year, and it’s clear that they were right. In this time, they have not only made an important re-branding, but they are also getting ready to launch a new product.

What were your thoughts behind changing your name from TestMyBot to Botium, and how important has it been to your startup success?

We actually came up with the name Botium on our way home from BETAPITCH Global 2017. Guess it was a side-effect of the event. In the quality insurance industry, the two big players are Selenium and Appium; we decided to follow their naming scheme so that people in the industry immediately will know what we’re all about. It worked! When we were presenting at a big AI conference in Bratislava the other week, the audience just saw the name, and we could hear the “DING” going off in their heads.

Making money with bots has been hard for many companies, but do you feel like you have cracked the code?

The thing is, this stuff is changing, it's the same as we saw maybe 15 years ago when all of the big companies realised that they had to test their software to ensure quality. In the last years, there has been a lot of crap chatbots going out since they are pretty easy to develop and run. The problem, of course, is that these bots are failing which results in all kinds of shit storms, so big companies are realising now that chatbots are software and should, just like other software, be tested. This, of course, opens up demand for proper testing.

Our core product is open source and developers can get it for free. But the open source product is merely a framework, so the user needs to code it and write their own test cases to use it. Which is hard to do, time-consuming and costly. Therefore we are now working on the product we call Botium Box where everything is included, and this is the product we’ll be selling. Basically, it’s the core framework delivered within a nice user-interface, including thousands and thousands of test sets customised to a specific bot.

The Botium framework is open source, and you say that it always will be, why is that?

One of our goals as a company is to contribute to the open source software developing community. That sounds a bit weird, but we just love software, and this community has given us a lot of advantages in our careers. For this reason, the botium core will always be open source. On another note, in less than a year we already have big enterprises all over the world using our framework, this has been possible in such a short time thanks to the open source community.

Why should companies start automating the testing of their bots?

You can ask a chatbot anything; the number of test cases is unlimited. So getting a good test coverage is very hard. On top of that, it’s not uncommon for companies to have many versions of their chatbots active at the same time in different phases, and they have to test all of them. Manually that means they’d need at least 30 people sitting there for six weeks for each version. People make mistakes, that’s unavoidable under these circumstances. With automated testing, you can run the same 200 000 tests that the team of 30 people did in six weeks in two hours. But to be clear, automated testing does not replace manual testing completely. It does, however, make the life of the manual tester way easier.

What role will chatbots have in the future?

Right now we’re seeing the whole IT sector going into the direction of bots and AI. According to the Gartner Report banks will have automated 90% of their processes with a bot by 2022, it also says that 80 % of all companies will have some chatbot implementation by then. At first, this can seem like bots are ruining jobs. But the industry is also creating jobs in, for example, developing, testing and design. If you look what the big companies like Facebook, Amazon or even Zalando are investing in their chatbot development right now, it’s clear that this is a big deal and that it only will get bigger.

At BETAPITCH 2017 you won the local competition in Vienna, only three months after launching your startup, and were also successful at our in-house final, what are your best tips on how to deliver a good pitch?

I think the best tip is to be authentic. For example, we are just a couple of software nerds, and we don’t try to appear like anything else. We would not go on stage wearing a suit because that’s not us. I remember when we did the pitch training before competing at BETAPITCH Global, we knew that the audience would not be as informed on the quality insurance industry and bots as our normal audience, so we had prepared a story about Knight Rider which we got good critique on during the training. After that, some other teams tried to implement a story like that in their pitch, but since it wasn’t authentic, it became forced instead.

What was your experience of Investors Day & BETAPITCH Global 2017?

We have made tremendous progress in this past year, I enjoy looking back on what we had at times around BETAPITCH and compare it to now. The competition was a stepping stone; we found a lot of new contacts and are now even hiring some of them.

Our next edition of Investors Day & BETAPITCH Global is coming up on November 15th! Come by, get inspired and find your own stepping stone.

Get your free startup ticket here.