Create Intelligent Bots using MS Bot Framework and Azure Cognitive Services
Contents
Introduction .................................................................................... xiii
Target Audience. .............................................................................. xv
Types of User Interfaces .............................................................................. 2
Drawbacks of Conventional UI ..................................................................... 3
Conversations as Platform ........................................................................... 5
Introduction to Microsoft Bot Framework ...................................................... 7
Meet a Few Bots ......................................................................................... 9
Summarize ................................................................................................. 9
Your Face .................................................................................................. 10
Azure Bot Service ...................................................................................... 11
LUIS Bot ................................................................................................... 13
QnA Bot ................................................................................................... 13
Proactive Bot ............................................................................................ 14
Direct Line .............................................................................................. 15
IOT and Bots .......................................................................................... 16
Other Bot Frameworks ............................................................................ 16
Bot Abuse .............................................................................................. 17
Summary ............................................................................................... 17
Designing Bot Applications ...................................................................... 20
Setting Up the Development Environment ............................................... 23
Testing the Bot .......................................................................................26
Debugging the Bot Application .................................................................27
Bot Application Life Cycle ....................................................................... 28
Bot Architecture ..................................................................................... 31
Bot Authentication ................................................................................. 32
Building a Bot ........................................................................................ 33
Deploy Bot to Azure ............................................................................... 38
Register the Bot ..................................................................................... 42
Configure Channels ................................................................................ 45
Configuring Skype Bot .............................................................................47
Configuring Web Chat ..............................................................................50
Summary ............................................................................................... 52
Setting Up a Development Environment ................................................. 54
Build Hello World Bot Using VS Code ...................................................... 54
Debugging Using VS Code ..................................................................... 60
Building Bots with Conversations ............................................................ 61
Dialogs ..................................................................................................61
Prompts .................................................................................................62
Messages ...............................................................................................67
State ......................................................................................................68
Deploying to Azure ................................................................................. 69
Summary ............................................................................................... 73
Channels and Channel Data .................................................................... 75
Channel Data ......................................................................................... 78
Build a Chat Bot Using an Email Client ..................................................... 80
Build a Chat Bot Using Slack Channel and API .......................................... 87
Multi-dialog Bot Using Slack and Slack Channel Data ................................ 89
Onboarding a Slack Bot .......................................................................... 95
Remote Debugging Slack Bot on Development Machine ............................ 96
Summary ............................................................................................... 97
Understanding Conversations .................................................................. 100
Messages ...............................................................................................100
Activity ...................................................................................................101
Relationship Between Channels, Conversation, User, and Bot .................... 102
Message Under the Hood .........................................................................103
Conversation Under the Hood ...................................................................104
Building Bots with Conversations .............................................................. 105
Attachments .............................................................................................105
Hero Card ................................................................................................ 111
Thumbnail Card .........................................................................................113
Carousal ...................................................................................................114
Buttons ....................................................................................................116
Prompts ...................................................................................................116
Summary ................................................................................................ 121
Introducing Skype Calling Bots ...................................................................... 124
Use Cases for Skype Calling Bots ................................................................... 124
Enabling Calling for Your Bot ......................................................................... 125
Building a Skype Calling Bot ........................................................................... 126
Sequence of Events ....................................................................................... 130
Debugging Skype Calling Locally Using Ngrok .................................................. 139
Speech-to-Text Using Bing Speech API ............................................................ 141
Summary ....................................................................................................... 149
Stores for Bot State ........................................................................................ 152
State Service .................................................................................................. 153
Storing and Retrieving State Using StateClient .................................................. 155
Storing and Retrieving State with Dialogs ......................................................... 158
More Control over State with Dialogs ............................................................... 162
Custom State Data Store ................................................................................. 165
Overview of Cosmos DB ...................................................................................166
Cosmos DB as Custom State Data Store ............................................................166
Table Storage as Custom State Data Store ........................................................173
Summary ....................................................................................................... 180
The Dialog Model ........................................................................................... 181
IBotData .........................................................................................................182
IBotTouser ......................................................................................................182
IDialogStack ....................................................................................................182
IBotContext .....................................................................................................183
Dialog Stack ................................................................................................... 183
Dialog Context ................................................................................................ 183
Root Dialog .................................................................................................... 183
Building a Simple Dialog Bot ............................................................................ 184
SimpleDialog.cs ...............................................................................................184
MessagesController.cs ......................................................................................188
Creating Multi-Dialog Bots ............................................................................... 189
Scenario ..........................................................................................................190
Solution ...........................................................................................................191
RootDialog.cs ...................................................................................................192
Synonym.cs ......................................................................................................194
Antonym.cs .......................................................................................................194
Support.cs ................................................................................................,,.......194
MessagesController.cs ........................................................................................195
FormFlow ......................................................................................................... 195
Building a Simple FormFlow Bot ..........................................................................196
FormBuilder ...................................................................................................... 199
Customizing the Prompts .....................................................................................199
Customizing the Order of Prompts ....................................................................... 200
Conditional Fields .................................................................................................200
Summary ........................................................................................................... 202
Cognitive Services .............................................................................................. 204
LUIS ................................................................................................................. 204
Intents ...............................................................................................................204
Entities ...............................................................................................................205
Utterances ..........................................................................................................205
Features .............................................................................................................206
LUIS Development Lifecycle .................................................................................206
Sample Application ............................................................................................. 211
Creating Intelligent Bots ..................................................................................... 215
Creating Intelligent Bots Without Dialogs .............................................................215
Summary ......................................................................................................... 232
Introduction to Microsoft Cognitive Services ....................................................... 234
Getting Started ................................................................................................ 238
Building Smart Bots with Bing Web Search ......................................................... 244
Query Parameters .............................................................................................247
Bing Search Request .........................................................................................249
Handling Errors ................................................................................................ 253
Optical Character Recognition with Computer Vision API .....................................256
Summary ........................................................................................................ 260
Application Insights .......................................................................................... 261
Getting Started ................................................................................................ 262
Enable Bot Analytics ......................................................................................... 271
Advanced Analytics .......................................................................................... 277
Summary ........................................................................................................ 278
Index .............................................................................................................. 279
Introduction
Bots are the new face of the user experience. Conversational user interfaces (CUI) provide a plethora of options to make the user experience richer, innovative, and appealing with Email, SMS, Image, Voice, or Video to communicate with the application. Modern web or desktop applications will soon be replace or augmented with intelligent bots that can be connected from anywhere using any device. Bots can use artificial intelligence and user data to provide richer insights and a personalized experience.
The Microsoft Bot framework has made the bot-building experience easy, and by using this framework we can build rich, scalable, and intelligent bots that can be connected from anywhere using an impressively vast list of platforms like Email, Skype, SMS, Facebook Chat and so on. This book explains how to develop intelligent bots using the Microsoft BOT framework, Visual Studio, Microsoft Azure, and Microsoft Cognitive Services. The preliminary chapters of the book deal with helping developers learn the basics of bot development using Visual Studio, .Net, C#, and Node.js. You will learn basic development and debugging skills, publishing to Azure, and configuring bots using the bot developer portal. The advanced section of the book deals with building intelligent bots using scalable storage, conversation flows, Microsoft Cognitive Services like LUIS, Bing Search, Vision and Voice API. This section also explains configuring analytics and other common Bot Operations.
The book is divided into the following sections:
Part 1 focuses on the need for a new communication platform and how conversation user interfaces (CUIs) break the barriers of building user interfaces; it also describes the current trends and future focus of this upcoming CUI and bot platform. This part also focuses on salient features of the MS Bot framework, available versions and features, and how the industry is embracing the change, and compares it with other competitive technologies and roadmap.
Part 2 teaches how to design and develop simple Skype bots on the Windows platform using the MS Bot framework, Skype, Visual Studio, .NET, and Azure. In this part, the readers will also learn how to build bots using open source platforms like Node JS and VS Code. Readers will be shown how to manage the complete lifecycle of a Skype bot, like design, development, testing, and pushing to production.
Part 3 goes into prospective features of the Microsoft Bot framework, like channels and channel data and using rich text, buttons, media, and actions in chat messages. It also explains building a bot using the Skype Calling API and speech-to-text conversion, managing user data using Bot State Service, and using different conversation flows, like dialog model and form-flow model.
Part 4 delves into the details of building bots by integrating with Azure Cognitive Services, like Bing Search, OCR, and LUIS. We also focus on how to perform common operations, analytics, and diagnostics on bots in production environments.
Target Audience
The target audience for this book is C#/Node.js developers and architects who design and build modern applications using a Microsoft stack like Azure Cloud, Visual Studio, and code. Developers who want to get up to speed by learning the cutting-edge technologies that enrich the user experience and cater to multiple form factors. Architects/developers who wish to learn to build and design scalable and reliable messaging platforms that offer rich conversation experiences with the use of attachments, rich text, and voice for communication with enterprise applications. Developers can learn to integrate bots with machine learning and Cognitive Services offered by Azure. Business analysts and UX specialists can also learn to design trendy user interfaces by using bots and Azure ML that can be connected using any device and provide an enriched user experience to end customers. The target audience of this book do not need any prior bot-building experience.
CHAPTER 1
Conversations as Platforms
Have you ever had the experience of ordering pizza using an application that remembers your favorite pizza and orders to your current location automatically? Or have you ever booked a cab just by typing in a chat window or by using voice inputs and had a cab show up at your door step? If you have seen either of those, what you have experienced is the new generation of smart applications called bots (a short form of robots). Bots provide richer and more personalized experiences in our day-to-day activities, thereby making our lives much better. If you have not experienced this firsthand, you have yet to witness the next revolution in IT after the worldwide web, mobile, and data. Bots are much smarter than mobile applications; in some cases they can be smarter than you. Bots are designed to perform human-like interactions and exhibit human-like intelligence. Chat bots are not new; we have had platforms that help build chat-based applications for quite a few years (like Skype SDK), but what makes the new generation of bots special is their integration with artificial intelligence.
Over the years, smart devices and smart phones have become such an integral part of our life that they now hold lots of useful personalized information, like your favorite color, calendar, contacts, favorite restaurants, and so on. New-generation bots are designed to use the data and context surrounding the data with machine-learning (ML) and deep-learning technologies to give you a richer experience. A few decades ago, using ML or deep-learning technologies in a commercial application was highly complicated because they involve lots of new learnings and come with heavy computing and memory requirements. With the advent of cloud and serverless computing, the use of machine learning, data analytics, and advanced algorithms like facial recognition, voice recognition, and search is just a click away. The focus of this chapter will be on introducing the benefits of building conversations as a platform for all kinds of business needs; the Microsoft Bot framework, one of the top-class, end-to-end suites for building smarter, richer bots; and the various bot intelligence services and platforms available.
The following topics will be discussed in this chapter:
• Types of user interfaces
• Drawbacks of conventional user interface
• Conversations as platform
• Introduction to Microsoft Bot framework
• Meet a few bots
• Azure Bot Service
• Direct Line
• IOT and bot scenarios
• Other bot frameworks
• Bot abuse
