About a month ago, we announced the new Analyze section in the Tars Chatbot builder. As you might expect, we made the announcement through a gif-laden newsletter with a title so clickbait-y, that even Casey Neistat would disapprove of us.

Usually, when we write these newsletters, we try to convey in the simplest possible terms, the value we are offering to you, the bot creators who use our builder. This newsletter, however, was unwittingly different.

Little did we realize that our bold claim (of the new Analyze Section leading to higher conversion rates) was perhaps most true for none other than ourselves.

Some Background Story

The story begins well before the analyze section was ever a thing. When we launched our website in 2016, we added a chatbot to our landing page which served as part demo and part lead generation tool.

The flow was pretty basic, but it got the job done reasonably well. As a result, we barely touched the bot between that initial release and about two months ago. Two months ago we decided that maybe it was a time for a refresh. We were seeing much more traffic, parts of the old flow were looking a little dated, and most significantly, we had learned a lot about bot creation since 2016. So after about a week of drafting, redrafting and a whole bunch of procrastination, we had a new bot ready to go.

As it’s creator, I thought the flow was perfect. It was witty, complex and it had wonderful little idiosyncrasies which were sure to entertain users. I was confident that this bot was a dramatic improvement on what was there before!

I was wrong.

We deployed the bot, and over the next three weeks we saw the number of leads coming into our funnel fall off a cliff. Even without the Analyze Section, we could tell that situation was pretty dire. Seeing as none of our other lead magnets had changed, it was fairly obvious that the new bot was the reason for the decline.

Disconcerted, I went back to our bot builder to figure out where I had gone wrong. The best hypothesis I could come up with at the time was that flow was too complicated. Users were simply getting lost in the conversation and would exit before they reached the part of the conversation where they would provide their contact info. I decided the best course of action was to start from scratch. After about a day, I had created a new flow that was far leaner and far easier to navigate. We deployed the bot and that seemed to be the end of the drama.

Qualitatively, the number of leads coming in increased overnight and everything in the world seemed good

About a week later, when the dust had settled and I had stopped thinking about the new bot, Vinit (you’ll know him as the recurring character from the newsletters), revealed the new Analyze Section! My first thought on hearing this was that we needed to check how our own bots were doing. I rushed to the builder, opened up the new bot and the old bot and compared the stats. The results were pretty astounding.

The new bot had a conversion rate of about 14%, where the old one had a conversion rate of 0. Seriously!

For a good month, the bot on our website generated 0 leads and we didn’t realize it.

Ener the Analyze Section

The fact that the Analyze Section had put a quantifiable number on my failure made me hyper-vigilant about the effectiveness of my bots. I was scared that the 14% percent conversion while being decent, was not perfect. What if I was missing something as big as had been before?

Every morning for the next 2 weeks, after I got ready for the day and sat down for work, the first thing I would check was the Analyze section on the new bot I had created. I’d spend hours tinkering around with the goal settings, to get a better sense how users were behaving and I’d follow this by trawling through the thousands of individual data entries to confirm my findings.

Finally, on July 5th (earlier this month), I was ready for my first change.

Before the change, the introduction to the bot looked like this:

 

My experimentation over the past two weeks suggested that the fourth option had two glaring issues with it.

  • First, if users selected this option it took far longer to reach the lead generation sequence than if they selected any of the other options.
  • Second, the vagueness of the option, was far too attractive to users who had even the slightest doubt about whether they wanted to sign-up for the builder.

I figured that if I removed the option altogether and forced users to chose between the first three options (which lead into the lead generation sequence pretty quickly) they would be more likely to actually provide some valuable contact information or sign up for the builder. A tenuous hypothesis, but a hypothesis nonetheless.

The result was a new introductory gambit that looked like this:

 

I checked back in two days later and the conversion rate had gone up to 23%.

 

Seemingly the changes were working! Though as my 9th grade stats teacher would tell me, only time would tell if the change was an anomaly or permanent. It has been two weeks since the change and the conversion rate has held strong at around the same level:

The Takeaway

Analyze Section is truly one of those product updates that can help you enhance the performance of your chatbots.

The story above shows how just one basic experiment helped us increase our conversion rate by a whole 60%. We are surely going to be using it to run more experiments in the following weeks and will share the results with you (even if they are bad). If you build chatbots using Tars, we encourage you to do the same and tell us how it goes!

, , , ,