Wednesday, February 25, 2015

One hard worker among many: Team Spotlight

In one of the earliest posts on this blog, James introduced everyone to our brand new kettle, which he thought might inspire nightmares in some.

More than six years later, the kettle is still chugging along. It seems less nightmarish now. Given the number of tea drinkers in the office, it may be one of the hardest working members of the team. 

As you can see, it does have the looming spectre of the Keurig ever present over its shoulder. Knowing that its job is in jeopardy should it ever falter, we expect to see further prodigious output from this kettle for years to come.

Tuesday, February 24, 2015

Is Anybody There? Logging Agents Out of Your Call Center Software

Agent call center ACD login
Don't resort to this
Agent behaviour isn't always perfectly aligned with the needs of the call center. An example of this is agent logout from the call center software. This is something that comes up from time to time. How do we make sure that agents who are logged in are actually available and working, and that agents who have gone for the day are logged out?

There are a few reasons we want to have agent sessions end appropriately. The obvious one is agent time tracking. If we are tracking agent time from login to logout, it's a meaningless metric if agents just get up from their seats and leave at the end of their shift. In that case we can usually look at the agent states and determine when the actual time of effective work ended within a reasonable time range.

Sometimes incorrect logout causes other issues. In the case of call center software where there is monthly usage licensing in place, or a licensing capacity constraint, it's important to make sure that those who are not working get logged out in favour of those who are actually present.

Unfortunately there's not a magic bullet for this issue. I once had to explain to a potential client that the software had no way to tell if an agent was actually in their seat. Short of a sensor in the chair (which could be confounded by a backpack full of books, undoubtedly), the system can only tell that the phone is ready, the agent has indicated their current status, and that a call may come in. Testing with automatic timeouts turned out not to work in slower inbound workflows. Having automatic pingbacks to make sure the agent screen was still open encountered timing issues with browsers executing javascript differently in browser tabs that did not have focus.

Ultimately, there are three methods that have stood the test of time for making sure that agents who are logged out are logged out correctly:

  1. Ensuring the agent logs out correctly. In any reasonable call center software, a successful logout should be no more than a couple of clicks away after successful wrapup of a call (or from an idle or wait state). We have found that newer agents haven't always gotten the same level of training as previous agents, and don't know the correct procedure. Agent training can help with this.
  2. Floor supervisors can help by looking around and logging out any agents who are not present and who are not expected to be working. Obviously this one can be very call center dependent. It's then incumbent on the software to make sure that agent names, current state, length of time in said state, etc., are all available to allow the supervisor to make the correct decision.
  3. Making sure that the previous session is closed the next time an agent logs in. If the agent has just logged in, we can presume they are done with their previous session. While this may seem a bit obvious to point out, in cases where agents are expected to log out between calling sessions during the same shift, sessions can begin to pile up if automatic logout did not occur.
It's also often the case that a simple set of business rules can be determined for a call center, allowing for an automated solution that matches the needs of the call center to be quickly constructed using existing APIs.

Agents can help. If they are aware of the proper procedures and the importance of following them, adherence to correct login and logout practices can rise dramatically. This helps reporting, licensing, and the overall management of your call center.

Tuesday, February 17, 2015

Where's My Data? Accessing External Data Sources in your Call Center Software

When you have a previous relationship with your client, it's important that this be reflected in the customer experience. Often, the data concerning the client is available in a separate system from your call center ACD software.

We've touched on the topic of using external data in previous blog posts, notably Populating Client Information to Agents in Your ACD - Part 1 and Part 2. In this post, the topic is solely external data.

Keeping data in a central location can be an effective way to ensure data stays in a consistent state, rather than being replicated and updated separately. It's important to make sure it's available when and where needed.

In the Dialplan

While the call is still in the dialplan, there are a few methods you might use to associate external data with the call in an Asterisk-based system:

Asterisk Gateway Interface (AGI)

The AGI method is really a combination of methods to pull data. Since an AGI can be any sort of executable program, an AGI can return data from an external database, pull results from a web service, or even calculate values based on other information already available in the call. This flexibility is the biggest advantage of this method, despite the potential complexity. The biggest drawback can be response time. In most cases, the call has to wait for the AGI to finish. When calling a web service or consulting a database, any delay in receiving a result delays the processing of the dialplan. If the web service being called takes 30 seconds to return, that's 30 seconds the client is waiting before they can enter a queue, hear a prompt, or do anything else.

AGIs are easy to set up, you must simply put your program in the Asterisk AGI directory. This is usually /var/lib/asterisk/agi-bin. Once it's there, it can be called in the dialplan using the AGI command and passing any parameters. The AGI can also access a wealth of values that are already set for the current call. For common tasks such as calling a web service, your dialplan builder may already have an AGI that can be used. Many providers now offer external APIs for their services. Functionality can range from a large database that provides multiple pieces of data in a single call, to something as simple as a Caller ID Name service. These are easy to access via a web service AGI.

Asterisk Manager Interface (AMI)

Using the Asterisk Manager Interface to push data is not for the faint hearted. Connecting to the AMI allows events to be monitored, and in theory if you note an event you can respond to it via the same interface. Unlike the AGI, if you want to have data set at a point in the call, the call won't stop and wait for it. You either have to guarantee that the proper AMI commands will be sent at the appropriate time, or you have to build something into the dialplan to wait until the correct values are set.

Using the AMI is not something call centers usually do for themselves, but sometimes system integrators will hook into it. The AMI is the most efficient way to actively monitor Asterisk and calls all the way through live on the system. Anything else would normally require customization of Asterisk or a lot of monitoring of files and processes at the operating system level. It does allow sophisticated software systems a window into your system, so it does require caution but provides powerful access.

DTMF From the User

Really, the most common method used is the third. Using the client as a data source is efficient to do, but does not improve the customer service experience in most cases. Asking the client to enter a 9 digit account number is prone to error, as well as memory lapses. If you are asked for such data in a phone call, it usually means you have had to look for old invoices or through your email. Very inconvenient. This is a method that is not recommended if other methods are available, but it can be a good fallback if another method returns an invalid, confusing, or non-result.

The biggest limitation to user-generated data is the interface. For the most part, the interface is limited to the common keys present on telephones - 0 through 9 and the * and # keys. Voice recognition is a possibility, but it can be difficult to use, and accuracy becomes a bit of a problem. In a voice call, you can't count on additional interface options being present.

At the Agent Screen

Once the data has been received, you still have to do something with it. When you're branching in the dialplan based on what data is received, it certainly makes sense to get it while the call is still there. However, you often need to get this data to the agent handling the call. In that case, it can make sense to access this data via the agent interface. The agent interface should offer a client interaction script that can pull data from external sources. The most common methods to do this would be:

  • An external web page
  • A database query
  • A web service
  • Asking the client directly

External Web Page

The simplest way to present external data to the agent is to incorporate a third party web page into the agent interaction script. This may be done in-line in the agent script itself, or as a pop-up or open in a new page or tab. Usually some identifying piece of information is used to start the process, such as an account number that was pulled out at the dialplan stage, or the client phone number. This method has the advantage of being easy to incorporate into a web-based interface. It can also allow the agent to update and save data directly to the external source, rather than complete the call then have the information synchronized at some other stage.

When using this method, testing has to be done to ensure that some property of the external page doesn't interfere with the calling page. For instance, we've seen instances where the external CRM being used changed the page to automatically go full screen. This completely disrupted the rest of the web interface, and meant the page could not be used inline. Other complications are certainly possible.

Database Query

Using an external database query to pull data lets you go straight to the source and get the most up-to-date version of the data. Most call center managers are not familiar with databases, so this is something that would have to be done in conjunction with more technical personnel. Also, there are a few ways to write the SQL and any number of possible return values, so this can be limited to one value per request.

One gotcha with using external databases is that the database and drivers often have to be defined at the OS level. This makes configuration an operating system level issue in order to make sure the drivers are present and configured. In cases where frequent accesses of information from a single database are being made, this can be a good solution, however.

Web Service

Just like with accessing data in the dialplan, web services are quickly becoming a preferred way to get external data into the agent screen. Providing an external API accessible via HTTP or HTTPS can be safer and more secure than providing external access to a database, and allows processing of the information as needed. There can be a few technical challenges to making this method work, but once it is, the flexibility and efficiency gains can be tremendous.

The biggest advantage of this method is that web services are much more easily customized. Implementation doesn't depend on specific configuration or drivers on the client servers. Hosting and implementation details are completely separate from the ACD solution being used. The Cloud can easily be utilized to provide such services.

Asking the Client

Having the agent ask the client for data is a much better option at the agent interaction level. The limitation is no longer the keypad, and agents naturally do speech recognition very well. The biggest disadvantage to this method is the time and annoyance factor. It can take time to go through all the information needed. Callers do find it annoying when they are asked for information that they assume the call center alragent interaction scripteady has. When that information already exists elsewhere, it's faster and more efficient to gather it in one of the other ways.

In the end...

A solid, comprehensive contact center software solution should offer many ways to pull data from external sources. The solution can be simple, it may be complex, but ultimately making sure the data is there when the caller is on the line is how you are going to ensure a smooth customer interaction.

Tuesday, February 10, 2015

Save the State: Data Synchronization Between Master and Backup Services

When the center is actively handling calls, minimizing any disruptions is key. You don't want to have your agents stop, log out, log back in, then attempt to continue as they were. Doing that is an efficiency killer. To avoid this, multiple copies of a single service can be set to run on different servers. However, data has to be kept current in order to avoid losing information when switching services from backup to master.  Synchronization of information is absolutely required when we are attempting to set up redundancy for high availability in the call center.

Two special cases where data must be available are call recordings and voicemail. The topic of recordings has been discussed in the past especially in the case of call recordings in the Cloud. Once recordings are in the Cloud they are available to redundant systems. In the case of voicemail, using database-backed voicemail allows the use of a single point of access for voicemails that were originated on multiple servers. The difference between these two services and others is that the data is available as a service that is (potentially) outside our system.

When there is a disruption, the three primary systems where backups need to be kept up to date are:
  •  Call and client data (the database)
  •  Telephony connections and sessions (for call survival/recovery)
  •  Agent sessions and states (the ACD software).

Database replication is a well understood problem. MySQL, for example, allows for easy synchronization of data as it comes into the system. A strict master/backup system is the simplest to set up. Circular replication, where each server of a pair acts both as master and backup, is also a fairly common configuration. The database should be configured so that automatic values such as IDs are unique. This is usually done by specifying an auto-increment that is at least the number of servers, and a unique offset for each. When a failure occurs and the system switches to the other database, the data is present and available. When the original master is repaired and replication restarted, both databases will again have a complete copy of the data.

Telephony connections (calls) have two different ways that data redundancy is required. If a telephony server itself goes down, the situation is similar to the database one above. If call survival is desired, then the Indosoft High Availability SIP Proxy (HAASIPP) must be used. The HAASIPP itself is responsible for maintaining the SIP session, and if it detects an issue, it can resume the same call by sending the call information and voice data to another Asterisk box and redialing the agent. This allows the call to stay live, with the client staying on the line the whole time.

In order for the HAASIPP to maintain redundancy, it must transmit information to the backup HAASIPP in real time. Every state must be synchronized between the master and backup service. This allows SIP sessions to continue even if the master HAASIPP goes down; the second HAASIPP server has been receiving live data all along and is ready to be promoted to master at any time without a loss of service.

Synchronization is also a requirement for redundancy in your call center automated call distributor (ACD). While the calls themselves continue on, the agent states must be maintained. This includes details such as:

  •  who is logged in
  •  which telephony device they are attached to
  •  who is ready to receive calls
  •  who is in an away state
  •  any other details that may be relevant

Of course this has to be maintained live. As soon as an agent is ready to receive a call, that fact must be noted on the backup ACD. That way the agent doesn't notice a disruption in call handling when the ACD server or service goes down.

Synchronization isn't needed only for failures, either. In situations where it's desirable to replace hardware or to do software upgrades, having the ability to move services to a different server and bring down a service or server while production is ongoing is very beneficial. Having this capability allows for maintenance to be done even in 24/7 contact center configurations.

Tuesday, February 3, 2015

Improve Your Agents: Customizing Your Call Center Software Part 2

You can make your agents better. At least you can help them perform better, and you probably don't care about the difference. The ability to adjust your incoming call flow improves operations from the client side and can make the system as a whole work better. Optimizing operations on the other end, the agent interface, can have the same effect once the call has been answered. The interface, especially the client interaction script, has to be well organized and present the correct information in an easy to read format in order to give your agent a chance to handle the call effectively.

The most basic way to customize the agent interface for better call handling is to update the client information script. Call center software that allows the script to be updated in the call based on script progress or based on information on or gathered in the call is ideal. Being able to present a different script based on the number dialed allows a great deal of flexibility. Additional features that can be used to improve agent interaction are:
  • Data access from web service. The web service could accept nearly any detail from the inbound call and process it using whichever business logic is needed. This allows the custom work to be applied away from the software itself.
  • Adding CSS or modifying the HTML of a web-based agent interface. Creating a custom theme allows highlighting of certain components, hiding confusing aspects, renaming, etc..
  • Adding custom JavaScript. Use of JavaScript allows custom logic to be inserted directly into the agent interface, changing the way it works. JavaScript has become a strong platform on its own, and allows for powerful changes to be made.
  • Calling an external page. Using another web page entirely, called by the screen with data from the call, can be a simple solution in cases where the level of customization would be impractical. This allows systems that would not normally be able to co-exist to work side-by-side. The most common examples are domain-specific CRMs.
All of these are available in Q-Suite via the customer interaction script builder.

The ultimate in customizing your interface is writing your own. Using the call center software APIs available with the system, it should be possible to create an interface to your own specification. This is not something that would be done by call center managers, of course, but a developer or two should be all that's required to create the interface you need if the APIs are up-to-date and documented.

Tuesday, January 27, 2015

Let It Flow: Customizing Your Call Center Software Part 1- Customizing Your Call Flow

A big driver of the adoption of Asterisk is the flexibility it brings. It has been called the Swiss Army Knife of telephony with good reason. As a result, there are a number of things that one can do to utilize that flexibility and add a bit of customization to their Asterisk-based call center software. In most cases, the actual amount of modification done is pretty small, but it's always nice to know the ability exists if needed.

One of the most common ways to influence your call flow/functionality is incorporating a web service. Web service APIs exist for many services, from doing caller ID lookups to sending SMS from the dialplan. This functionality is not directly available in the dialplan for the most part, although a good IVR builder should allow the option. It is fairly straightforward for a developer to wrap the web service call and return into an AGI that can be called (more on AGIs in a couple of paragraphs). This allows both the sending and receiving of data, and your dialplan can be set to branch one way or another based on the results returned.

Of course, if we are discussing call flow, being able to create your own dialplans is a must, and is a feature of even the most basic Asterisk-based systems. On some systems, you write the dialplan as a file included in extensions.conf. In more advanced systems, there is a visual dialplan builder that allows you to script complex IVRs as well as directly writing your own commands to the generated dialplan. For speed and ease of use, it's preferable to have both methods available, as well as a variety of methods of reaching the dialplan you've created.

Many multi-tenant PBX systems will have a few default functions that can be mapped to star (*) codes, such as *99 to Voice Mail. Asterisk does allow you to define your own, and tie your functionality to a star code of your choosing through its Features configuration. Being able to define your own functions and map them to a star code can be done by manually editing config files in Asterisk. It is, of course, much better if your software handles this in a multi-tenant safe way, so that tenants can define their own functions and map to their own star codes without the chance of function collision, just as the Q-Suite does.

Last but not least is the ability to write executable code and have it run by Asterisk. Through the Asterisk Gateway Interface (AGI), programs can be run directly by Asterisk that have access to call data, do some processing, and generate some output as a result. The Enhanced AGI (EAGI) can allow programs to process the channel itself, reading DTMF or manipulating audio. This requires a greater degree of technical knowledge, and is not something that can be done through the interface of your call center software. However, the requirements of the program itself are fairly straightforward, and making it available is simply a matter of placing the executable in the correct directory with the correct permissions. Once it's there, it can be called just like any other dialplan component. The complexity can range from a simple script that acts as a wrapper to a web service, to a full-fledged system that accesses and processes external data, manages the audio stream, etc.. If you have an external system that you need to interface with, you may need to use an AGI.

Tuesday, January 20, 2015

Call Center Load Balancing with Kamailio

Session Initiation Protocol (SIP) hasn't officially been crowned king of call center technologies, but it has become ubiquitous. The wide availability of SIP service providers and the way Asterisk is pushing Open Source technologies into the call center has made it undeniable. Especially now with the widespread adoption of Cloud-based call center software and remote agents, SIP is cementing its importance.

While Asterisk has been a major topic in this blog, there are other ways to serve up SIP. Q-Suite has its high availability SIP proxy which manages call recovery. HAASIPP development was driven primarily with call survival in mind, however. A less-focussed, more feature rich SIP product that is widely used is Kamailio (formerly OpenSER). Kamailio has a huge feature set, but a few of these are important for the Cloud-based multiple server install. The ones that come first to mind are:

  • SIP packet capture and logging
  • Smart SIP proxy for routing calls to specific servers
  • Single point of registration for multiple servers
The SIP packet capture feature is something that proves to be invaluable when trying to diagnose VoIP issues. SIP packets themselves aren't large; most of the bandwidth of VoIP is consumed in the Real-time Transport Protocol (RTP) packets which carry the media itself. SIP concerns itself with the connection, not the data. Issues with calls can often be diagnosed by examining these packets and looking for lapses in the protocol. Unanswered requests, dropped or ignored requests, and timeouts can be spotted when examining the SIP session. When compared with the other end of the connection, such as against the provider's own logs, a cause to the issue can often be confirmed or ruled out.

Kamailio can be used to store these packets, so that diagnosis of an issue can be carried out on first report, rather than having to set up monitoring and wait for the issue to arise again. Tools such as Homer can be used to produce graphs and more easy to read displays of SIP data. This can be invaluable and save a lot of time in diagnosis.

Balance is important in the Cloud
Kamailio is also used quite frequently as a SIP proxy. When using it as a proxy, it can be set to deliver calls from specific trunks to specific Asterisk servers. This is useful in a multi-tenant multiple Asterisk server configuration, when a tenant may be assigned to a subset of the Asterisk servers. Rather than having calls delivered to the "wrong" server and routed back to an agent over an additional channel, the overhead caused by the tenant can be limited to the specifically assigned servers.

What might be the most important use for Kamailio in the multiple Asterisk server configuration is the ability to act as a single point of registration. It can be set to handle agent phone registrations, while handing off the RTP part of the call. This gives one location for phones to be registered to, while allowing call load balancing over multiple servers. In a Cloud setting, this allows for instances to be brought up and down in order to manage changes in demand without having to reconfigure phones or registrations.

For these reasons, Indosoft Q-Suite is moving to increase integration of Kamailio with Asterisk and its own features. Smartly directing calls and handling registrations can reduce the amount of work the Asterisk servers are required to do and reduce the resources needed. This will reduce costs due to server load and bandwidth.