Improving the Bot for Sport Accessibility–Language Understanding

An Intent Beginning

The first thing we needed to do for the viaSport Accessibility Sport Hub (ASH) was to determine which basic questions we wanted to answer.  In bot terminology or more specifically the Microsoft Language Understanding Intelligence Service (LUIS) what “intents” were we trying to decipher and respond to.   The challenge is not be very specific but keep it simple.  Also, just identifying the “Intent” isn’t good enough.  Once we had that we had to think through the application development on how we would respond to a specific intent.  Microsoft Cognitive Services are great and the deciphering part but ultimately YOU, the developer have to sort out what to do with the information provided.

The first intent we decided on was “How To Coach”.  Basically the original mandate was to provide assistance to coaches who needed material that would guide them in coaching athletes with disabilities.  That information exists but is scattered and not necessarily readily available.  To make this work in our Bot application we had to do three things:

1. Use LUIS to define the “How to Coach” intent.  This means adding the intent, providing a number of examples of possible questions that equate to “How to Coach”, training and publishing the service.
2. Create some sort of database to hold the references that viaSport and reviewed and listed and make sure those references are tied to the new intent.  We wanted to have more intents later so allowance had to be made for this.
3. Modify our Bot to accept the “How to Coach” intent and handle it.  This means recognizing the intent, querying the database that contains references for ones that match the intent and displaying it.  This part, good old coding, is the part that was expected to take the longest and was the highest risk area.

I’m hoping to cover each of these problems over the next few blog entries beginning with LUIS.  There is a lot to consider when you begin defining LUIS intents that may not be obvious when starting.

Is An Intent Enough?

Now that we’ve decided on an initial intent it’s time to consider what we will do with it.  In an information retrieval bot you ideally want to provide very specific results.  If your inquiry returns 200 possible answers, what good is it really?  So before we go to https://luis.ai to define our intent, lets consider what we really want to provide. 

We really need to know more than just “How to Coach”.  LUIS provides for a way of defining more details in the intent.  So all we need to do is come up with key bits of information that will help us narrow down the search for relevant resources for our coaches.  First, it’s obvious that the sport selected will have a big impact on the coaching information needed.  So with the intent of “How to Coach” we will want LUIS to extract from the natural language query from the coach what sport they wish to know about.  Sport will narrow down the information available but we decided on one more factor… disability.  Within each sport there are different disabilities that require different coaching techniques.  By adding that factor, the coach asking the question can get pretty specific with what they want.

Now we have an intent and two parameters, or what LUIS calls entities, that we can extract from the natural language inquiry.  All we need to do is create our LUIS application.

Working with LUIS

Start off by going to https://luis.ai.  You will need to logon with a Microsoft Account like @outlook.com or @hotmail.com.  If this is your first time with LUIS, there is a “Login/Sign up” link right on the home page where you can sign up (for free) for the service.

There have been many improvements since we started the viaSport project.  It used to be that you had to really spend a lot of time training it to recognize relevant entities but now, with just a few hints, it will generate many of the terms you need and then be able to deduce many more during operation.

The following steps were used to create the LUIS app for the viaSport Accessibility Sport Hub (ASH). 

• It may seem really simple for the very good reason that it is very simple.  Begin by tapping the “Create new app” button on the “My Apps” page.  Give the app a name and description although the description is optional.  I pretty much only work with English bots at this point so select English for the language.  As of this writing there are 12 languages already available.
• Tap the “Create new intent” button to add our “How to Coach” intent.  Give the name of the intent when prompted.  This is the name we’ll see in our Bot app when we are seeing which intent LUIS interpreted from the original input.  You might want to make a note of it.
• Now all you need to do is give it some examples (what LUIS calls utterances) of the query that should be interpreted as “How to Coach”. Let’s do the first one with a sport and disability.  We’ll then have an example where we can identify our chosen entities (Sport and Disability).
• How do I coach Hockey to a person who is an amputee.
• Tap on “hockey” so you can mark it as an entity.  In the popup type “sport” then click “Create new entity”.  We want this one to be a “list” entity as we are going to limit this to specific sports.  Select List from the Entity Type drop down.  After selecting list you can enter synonyms for hockey.  I wish that this had been available when we built ASH.  We had to build our own synonym system instead.
• Do the same thing with “amputee” creating a List entity called “disability”.
• Now add 4 more utterances that contain a sport and disability.  You can use words other than coach, like teach or any variant you wish.
• if you indicate a sport or disability different than our original ones, you’ll need to tap on the sport or disability and create a new synonym.
• Lastly we want to add some more disabilities and sports to our lists of entities we’ll recognize.  In one of the utterance tap the word “sport” then “Manage Entity”
• Notice right away that at the top are listed a whole bunch of other sports.  Now, ideally we want to add sports that we will be providing information on but you can go ahead and add a bunch of sports for now.  In our case we needed to add para-specific sports like para-skiing, wheelchair basketball, etc…
• We’ll want to do the same thing for our disability entity.

One of the keys to a great app is preparation and planning around what you will recognize.  If you think that you’ll have a sport called Hockey but there will be lots of other ways to describe hockey but will, for our purposes, be the same, then add lots of synonyms like, in our case, “sledge hockey” and “para-hockey”.

Once you have done all of the above, all you need to do is train and publish your LUIS app and you’ll be ready to call it.  Just click the “Train” button then you can go ahead and test it.  Once you are happy, tap the “Publish” tab and publish the app.

You will need the Endpoint listed at the bottom of the publish page.  Make a separate note of the Key String (if it’s not there, you’ll need to create it) and the App Id (looks like a GUID and can be found in Settings tab).

What’s Next?

We are now ready to move to our bot app in Visual Studio to make it all talk to LUIS and start understanding natural language.

Building a Bot for Sport Accessibility

Earlier this year I had the opportunity to help a local non-profit sports organization (NPO) get information on sport disability and accessibility out to athletes, teachers, parents, coaches and sport leaders.  I was fortunate enough to work with some great experts from the NPO as well as Microsoft.  After some initial discussions we came to the conclusion that we would go with a natural language interface otherwise known as a bot.  We decided on using Microsoft’s Bot Framework and at least one Microsoft Cognitive Service, the Language Understanding Intelligent Service (LUIS).  This blog will be about some of the decision making that went into choosing these technologies instead of others and what benefits were gained by our choices.

Chat or Click

When we first sat down with viaSport British Columbia, the NPO to discuss their needs, they made it very clear they wanted something different.  There have been plenty of attempts in the past to provide information to their constituents and there were examples of other organizations, such as the Canadian Paralympic Committee, that have provided some of the information. 

One of the methods used in the past was to have a series of cascading combo boxes where you selected some property which would then set a selection list for the next combo drop down box.  You would work your way through these items until you came to the end.  One of the problems with this is it assumes a certain level of knowledge of what question you want answered.  You also have no flexibility in decisions made during the process.  To continue you must select something even when “your” option wasn’t available.  viaSport (our NPO) wanted something a bit more friendly and more in line with the way its members commonly communicated with each other.  Sending messages and typing on their phone or computer using natural language seemed to be a natural (pun unintended) fit.  We decided on chat instead of click and implement a chat bot using the Microsoft Bot Framework and natural language with LUIS and Microsoft Cognitive Services.  Later on, we added the use of some other cognitive services.

Getting Started

The initial goal was to simply allow their constituents to visit their web site and ask, in plain language, for information specified by sport, disability and person asking for the information.  For example, a coach might ask “How do I coach swimming to a paraplegic” and the goal would be to provide reference material helping that coach to provide guidance to the athlete specified.  To accomplish that goal we needed only two elements and a hosting service.  The Microsoft Bot Framework could nicely handle the conversation part of things (back and forth), with a C# code behind, doing the lookup of the information and Microsoft’s LUIS Cognitive Service to provide the natural language understanding of the questions being asked. 

Ultimately the thing that would make the project a success, as with most projects, is the quality of the data or information we could provide to the the clients.  The whole point of the exercise was to provide curated information to the people who needed it without the massive amounts of information you would get from a Google or Bing search.  So, combined with our bot and natural language processing we used Microsoft Azure SQL database to store the curated list of information and a Microsoft Azure App Service.  Setting of all Cognitive Services is done through Azure also, so signing up, initially, for the free trial let us move forward with a prototype almost immediately.

Starting Coding is Fun!

I chose C# as the language of choice for a couple of reasons.  I could have done the bot project in NodeJS but chose not to.  First is that I’m very familiar with C#.  I’ve written hundreds of applications using C# and it seemed the logical place to start.  Also, I realized very early that not only did I have to create the Bot but an administrative tool for viaSport to manage their curated links and content.  That tool would be best written in C# as a UWP app.

So, to begin we used a template provided by Visual Studio 2015.  It created a very simple template that didn’t include anything to do with LUIS and required that all the connectivity to LUIS had to be done by me.  The good news is, since we created our bot, there now an easier way.  Inside Azure, you can simply create a new “AI + Cognitive Service” service and one of the choices is “Bot Service (Preview)”.  This creates the whole framework for you to begin, including all the hooks you need into LUIS.  It’s like magic!

Once you create the Bot Service you are given a choice of 5 (currently) different templates for the bot using two different languages, C# or NodeJS.  The templates cover everything from a simple “echo bot” where it just echo’s back the text input to a Q&A type bot to a bot using Azure Functions to the one we want, a bot template that will automatically bake in Language Understanding (LUIS).  Having that template would have been very helpful in the early days of creating this bot.  Once you have chosen your template the template wizard will walk you through all the options you need to create that bot including assigning the App Id (needed for publishing) and password (take note of all the ids and passwords, you won’t be able to get them later).  Once you have created the App, it will provision the LUIS service for you and you will be ready to start defining intents (basically an intent is what the natural language will be interpreted as, for example “How to Coach”). 

All the starter code with all the keys already in place will be created for you.  You can easily hit the road running.  The Bot Service even provides an on-line editor that looks a lot like Visual Studio or you can download the source and use it with Visual Studio 2017 (even the free community edition) or lastly, have it uploaded to a source control provider like Visual Studio Team Services or GitHub.  You could literally deploy the Bot code as is and start using it (once you put some intents in).  The on-line editor is surprisingly useful and lets you get going right away.  You can even run and test your bot or debug it using the Bot Framework Emulator (they give you a link to download and install the emulator).

The End of the Beginning

At this point we had our prototype.  The next steps I’ll outline in a later blog involved building the back end database containing the content and an administrative tool for managing that content.  My goal was to hand off the project to viaSport without them needing to call me for every little thing.  I didn’t want to be a dependency for them moving forward.  I made myself available to help out but most of what they wanted to do they could easily do on their own, including manage the content, track usage and other telemetry and make improvements to the bot understanding, all without having any developers on staff or needing to contract one.

As I work through the building of this application over the next few blog entries, we’ll look into Windows UWP App development, other Cognitive Services, inclusivity considerations and telemetry.