Tuesday, January 27, 2015

Let It Flow: Customizing Your Call Center Software Part 1- Customizing Your Call Flow

A big driver of the adoption of Asterisk is the flexibility it brings. It has been called the Swiss Army Knife of telephony with good reason. As a result, there are a number of things that one can do to utilize that flexibility and add a bit of customization to their Asterisk-based call center software. In most cases, the actual amount of modification done is pretty small, but it's always nice to know the ability exists if needed.

One of the most common ways to influence your call flow/functionality is incorporating a web service. Web service APIs exist for many services, from doing caller ID lookups to sending SMS from the dialplan. This functionality is not directly available in the dialplan for the most part, although a good IVR builder should allow the option. It is fairly straightforward for a developer to wrap the web service call and return into an AGI that can be called (more on AGIs in a couple of paragraphs). This allows both the sending and receiving of data, and your dialplan can be set to branch one way or another based on the results returned.

Of course, if we are discussing call flow, being able to create your own dialplans is a must, and is a feature of even the most basic Asterisk-based systems. On some systems, you write the dialplan as a file included in extensions.conf. In more advanced systems, there is a visual dialplan builder that allows you to script complex IVRs as well as directly writing your own commands to the generated dialplan. For speed and ease of use, it's preferable to have both methods available, as well as a variety of methods of reaching the dialplan you've created.

Many multi-tenant PBX systems will have a few default functions that can be mapped to star (*) codes, such as *99 to Voice Mail. Asterisk does allow you to define your own, and tie your functionality to a star code of your choosing through its Features configuration. Being able to define your own functions and map them to a star code can be done by manually editing config files in Asterisk. It is, of course, much better if your software handles this in a multi-tenant safe way, so that tenants can define their own functions and map to their own star codes without the chance of function collision, just as the Q-Suite does.

Last but not least is the ability to write executable code and have it run by Asterisk. Through the Asterisk Gateway Interface (AGI), programs can be run directly by Asterisk that have access to call data, do some processing, and generate some output as a result. The Enhanced AGI (EAGI) can allow programs to process the channel itself, reading DTMF or manipulating audio. This requires a greater degree of technical knowledge, and is not something that can be done through the interface of your call center software. However, the requirements of the program itself are fairly straightforward, and making it available is simply a matter of placing the executable in the correct directory with the correct permissions. Once it's there, it can be called just like any other dialplan component. The complexity can range from a simple script that acts as a wrapper to a web service, to a full-fledged system that accesses and processes external data, manages the audio stream, etc.. If you have an external system that you need to interface with, you may need to use an AGI.

Tuesday, January 20, 2015

Call Center Load Balancing with Kamailio

Session Initiation Protocol (SIP) hasn't officially been crowned king of call center technologies, but it has become ubiquitous. The wide availability of SIP service providers and the way Asterisk is pushing Open Source technologies into the call center has made it undeniable. Especially now with the widespread adoption of Cloud-based call center software and remote agents, SIP is cementing its importance.

While Asterisk has been a major topic in this blog, there are other ways to serve up SIP. Q-Suite has its high availability SIP proxy which manages call recovery. HAASIPP development was driven primarily with call survival in mind, however. A less-focussed, more feature rich SIP product that is widely used is Kamailio (formerly OpenSER). Kamailio has a huge feature set, but a few of these are important for the Cloud-based multiple server install. The ones that come first to mind are:

  • SIP packet capture and logging
  • Smart SIP proxy for routing calls to specific servers
  • Single point of registration for multiple servers
The SIP packet capture feature is something that proves to be invaluable when trying to diagnose VoIP issues. SIP packets themselves aren't large; most of the bandwidth of VoIP is consumed in the Real-time Transport Protocol (RTP) packets which carry the media itself. SIP concerns itself with the connection, not the data. Issues with calls can often be diagnosed by examining these packets and looking for lapses in the protocol. Unanswered requests, dropped or ignored requests, and timeouts can be spotted when examining the SIP session. When compared with the other end of the connection, such as against the provider's own logs, a cause to the issue can often be confirmed or ruled out.

Kamailio can be used to store these packets, so that diagnosis of an issue can be carried out on first report, rather than having to set up monitoring and wait for the issue to arise again. Tools such as Homer can be used to produce graphs and more easy to read displays of SIP data. This can be invaluable and save a lot of time in diagnosis.

Balance is important in the Cloud
Kamailio is also used quite frequently as a SIP proxy. When using it as a proxy, it can be set to deliver calls from specific trunks to specific Asterisk servers. This is useful in a multi-tenant multiple Asterisk server configuration, when a tenant may be assigned to a subset of the Asterisk servers. Rather than having calls delivered to the "wrong" server and routed back to an agent over an additional channel, the overhead caused by the tenant can be limited to the specifically assigned servers.

What might be the most important use for Kamailio in the multiple Asterisk server configuration is the ability to act as a single point of registration. It can be set to handle agent phone registrations, while handing off the RTP part of the call. This gives one location for phones to be registered to, while allowing call load balancing over multiple servers. In a Cloud setting, this allows for instances to be brought up and down in order to manage changes in demand without having to reconfigure phones or registrations.

For these reasons, Indosoft Q-Suite is moving to increase integration of Kamailio with Asterisk and its own features. Smartly directing calls and handling registrations can reduce the amount of work the Asterisk servers are required to do and reduce the resources needed. This will reduce costs due to server load and bandwidth.

Thursday, January 15, 2015

Call Center Recordings in Amazon S3

As has been discussed before, call centers sometimes have issues with recordings piling up. Amazon S3 has been the solution for several Q-Suite clients.

One tool that comes in handy for managing files with S3 is s3cmd. If you're an Ubuntu user, you can even use apt (Advanced Package Toolset) to install. Other distributions also likely offer support.

The logic for putting files to the Cloud is pretty simple. After configuring s3cmd you can use the "put" option to send the file to its final resting place. In Ubuntu, s3cmd gets installed to /usr/bin, so my command to put the file big_recording_file.wav to the recording folder on my bucket would look something like this:

/usr/bin/s3cmd put big_recording_file.wav s3://stephensBucket/recordings/

That's fine for moving a single file. To move a number of files, you could write a simple shell script to find recordings, copy them to S3, then move the file to a "done" directory so that it doesn't get sent again by accident, wasting time. Here's a minimally viable shell script typical of the sort clients have used to move files:


#!/bin/bash

# This is information used to place the files in S3
S3BUCKETNAME=myBucket    # change this to whatever your bucket name is
FOLDER=myFolder          # the folder in the bucket 

COUNT=1000               # The maximum number of files to move each run

# These are defaults, and should not normally be changed
S3DIR=/usr/bin
TEMP=/tmp
DATE=`date +%F`
WAV_LOC=/var/spool/asterisk/monitor
FINAL_DEST=$WAV_LOC/done

mkdir -p $FINAL_DEST     # creates the directory if it doesn't exist

# Find the first $COUNT files in the target directory that aren't:
#    * In the final destination directory
#    * unmixed
#     * newer than a day
echo "$DATE :: Find and upload WAV files older than 1 day.." 
for i in `find $WAV_LOC -mtime +1 -iname "*.wav" \
    | grep -v "\-in\|\-out\|$FINAL_DEST" | head -n $COUNT`

        do    # put the file in S3
                $S3DIR/s3cmd put $i s3://$S3BUCKETNAME/$FOLDER/$i     
        
        # check for an error placing the file
                if [ $? != 0 ]              
        then    
                        echo "$DATE :: Command failed! See output for errors. Exiting." 
                        exit 255
                else
                        echo -n "$DATE ::  Removing backed up files... " 
            # you can do other things here, too
            mv $i $FINAL_DEST && echo "Moved $i to $FINAL_DEST"
            fi
done


The COUNT is where we determine the maximum number of files to move at a time. Note that in this example, there is no locking, we're not checking to make sure all the relevant directories exist, and the recording is simply being moved to /var/spool/asterisk/monitor/done. Ultimately, something else will have to be done periodically to remove the recordings from the server itself.

It should also be noted that s3cmd isn't highly efficient. If you have a lot of files and a lot of bandwidth, you will want to find a way to use concurrency to send multiple files simultaneously. If running a script like this, you should also make sure to capture the output by redirecting output to a log file.

If you or your software is keeping track of the recording location, you can add statements to update that location in the "# you can do other things here, too" area. For instance, Indosoft Q-Suite is call center software that keeps track of the location of the recording because it allows the use of multiple telephony servers, and presents recordings via the admin screens. Therefore Q-Suite installs sending files to S3 normally have to update the location in the database before finishing, so that the S3 recordings can be presented to the QA team.

If you're interested in what a script with locking and multiple processes would look like, you can see an example on this page.

Edit: I changed the s3cmd line slightly to specifically name the file at the end. I found that older versions of s3cmd don't behave as we would like without the file name specified, and newer versions can accept the filename without an issue.

Tuesday, January 13, 2015

Czy mówisz po polsku? Labels and Languages in Call Center Software

I can't vouch for the accuracy of these, either.

Czy mówisz po polsku? I'm hoping I can trust the Internet enough that this is a reasonable question to ask. Google confirms it is, and if you can't trust Google for good information, then who can you trust?

In most cases, the default names for things that are used by your call center software are reasonable, and designed to avoid confusion. Sometimes, however, labels wind up causing confusion when the label matches a term that is already in common use in the call center in another context. In other cases, you might find that some agents are simply confused, and a name change can ease the confusion. For instance, you might wish that the "Hold" button said "Put Caller On Hold" or something similar. Depending on the interface the agents are using, this might be something that can be done manually. However, your software should allow you to identify a label and change the name that is presented to agents. Your software provider doesn't work daily with your agents, so you are a better choice as an arbiter of names of things.

Taken a step further, it makes sense that languages should also be configurable by the call center. Often local variations in dialect will mean that a standard translation is a bit confusing. Call center software should allow the administrators to define their own translations for elements of the screens. Many languages use a right -to-left format rather than the left-to-right that is most common in the western world, and such support is also a requirement of any software that has to support multiple languages.

Q-Suite has long supported editing of labels and the specification of languages. Furthermore, agents are able to select from all the languages that are defined when they log in, so that the interface they're using can be set to match the language they will be using. Not having to stumble through translating the interface, or having to check their notes to find the "Transfer" button saves time and eases the burden on the agent, making the experience better for the client.


Tuesday, January 6, 2015

Decision 2015: Call Center in the Cloud?

In Decision 2015: On-Premise Call Center? we looked at some of the trade-offs with on-premise call center solutions. At one time on-premise was the only viable solution, but in the last few years the march to Cloud solutions has had the momentum.

When considering a Cloud call center implementation, one of the first advantages that presents itself is the ease of getting started. Most providers give you a menu of options, and once your choice is made, you're presented with a system to match. No ordering and waiting for hardware to arrive. Then there's the cost structure. You are only asked to pay for what you use when you use it. This is often on a monthly basis, allowing companies with to get started quickly at a low initial outlay.

  • No waiting for hardware to arrive. Systems should be available within hours if not sooner.
  • Low/no initial cost to get setup. You don't have to buy a set of servers.
  • Easier migration to bigger or smaller systems. You can often migrate a snapshot to more powerful systems when needed, or if it turns out the load is lower than expected, you can switch to something with fewer cores/less RAM.
  • Easier backup/snapshot options. Options to image and restore drives are often available.
  • You pay only for what you use. If you use 10 servers this month, you pay for 10. If you need 15 servers in 3 months, you don't have to pay for them now to have them availability later.
  • No responsibility for hardware. You don't have to dispose of old hardware or replace faulty components. The cost is covered by the provider.
  • No server room. You don't have to allocate space, power or resources to a server room.
These advantages mean that if you're just getting started, you can have a lot of your infrastructure in place and ready to go before you've acquired any space for your call center.

Obviously, if there were no disadvantages, adoption of the Cloud would be nigh universal. In moving to the Cloud, you may encounter these issues:

  • Higher ongoing cost. The monthly cost includes provisions for some of the advantages above, as well as a profit margin.
  • Difficulty incorporating add-ons like local storage, Orecx. Anything that requires direct access to the hardware is probably out.
  • Legacy systems may not be suitable. If you've got a special phone switch that your system depends on, for instance, you may have difficulty incorporating the Cloud.
  • Distance from users (latency). Local users are no longer local to your system, so you wind up increasing the number of connections out of the system over the Internet. Issues with latency can cause audio or data issues.
  • Inability to be hands-on with the software/hardware. If you've got an extra-special IT team or particular requirements, you can't manage your systems like you would with on-premise.
  • Increased bandwidth requirements. Now your data is flowing from the Cloud to your agents over the Internet, whereas before it may have been over the local LAN. You will need to acquire more bandwidth.
  • Many telephony interface options aren't available.
Whether you ultimately decide to go with an on-premise or cloud-based call center, you will want to make sure that you're using world-class call center software for your contact center. A system that supports either helps make the decision easier.

Tuesday, December 30, 2014

Decision 2015: On-Premise Call Center?

The old gatekeepers are stumbling. The momentum behind the change to the Cloud has caught them off-guard. That's not to say that the Cloud is the right solution to every problem, but it is increasingly a solution for call centers that must decide how to allocate their resources in the future. Having said that, decision-makers must weight the advantages and disadvantages of Cloud-based contact centers and on-premise solutions.


On-premise call center software has been the default solution for years. It certainly does have its advantages. For organizations with an experienced and equipped IT department, the ability to manage the hardware and software directly can be a distinct asset. In cases where users are all in or near a central location, the proximity of the hardware to the users can mitigate issues with network connectivity and latency, which can eliminate some problems with voice quality. Finally, it's easier to take an on-premise solution and combine it with other on-premise resources. These could include local storage, legacy systems, or solutions that require a direct connection to the hardware or network, such as Orecx.

If on-premise has these advantages, then there must be some disadvantages, too, otherwise the Cloud wouldn't be getting such traction. It turns out there are:

  • High initial cost. Hardware and software must be paid for up front, while many jurisdictions require the costs be amortized over a period of time. This is especially painful if the capacity is only required for certain periods, requiring you to have excess capacity sitting idle the rest of the time.
  • Responsible for failing hardware/software. When hardware fails, as it does, you will be responsible for fixing/replacing the hardware, at your expense.
  • Managing old hardware. As systems age, they are no longer able to keep up with performance and reliability standards. The hardware must be replaced. Once it is, disposal or recycling is your responsibility.
  • Maintaining server room. In addition to the space required, cooling and power must be supplied in sufficient quantities.
  • Managing power/cabling, other IT overhead. This is not usually an expense, but an additional headache that your IT staff must cope with from time to time as hardware is moved into and out of the server room.
 Now the picture is getting a little clearer. Next time we'll examine the Cloud in the same manner.


Tuesday, December 23, 2014

Populating Client Information - Part 3: Getting the Data Back Out

Data that can't be extracted or reported is worth less than data that can be reported. In "Populating Client Information to Agents In Your ACD" Part 1 and Part 2 discussed getting data into the system and to agents. Once the agent has gotten their hands on it, there can be updates, changes or even new data that needs to be reported back. Details about the call usually also falls under the rubric of exportable/reportable data.

If the data was accessed via a web service, database or API in the agent script, then it's reasonable to anticipate it can be sent back in the same manner. This method suffers from the same limitations as pulling the data in the first place (speed of external server, bandwidth, etc.) but does have the advantage of pushing responsibility for reporting to another server that may already have custom reports built in, or a more direct interface with the call center's client.

Another method is pushing the data via a web service post at the time the call is dispositioned. At that point, the agent should have completed the wrap-
up phase, and the data will be as up-to-date as it is going to be. It should be possible to submit any of the collected data to the web service, in a customizable manner. The script builder that comes with Q-Suite allows you to rename fields as necessary to match the expectations of the receiving service. Any number of fields can be accommodated, to the limits specified by the HTTP protocols. This has the same advantage of immediacy as the external database method, for those clients who want instant gratification.

The most manual of the extraction methods is a lead or call extract report. It's important to be able to define the fields, rename and reorder them as necessary, and also be able to define whether the extract will be done on a per-lead or per-contact basis:
  • Per lead: There is only one record per lead. The most recent data is returned
  • Per contact: There is one record per contact (call, usually), and custom fields are reported with the data as saved at the end of the contact.
It should be possible in the call center software to specify multiple templates, for multiple purposes, and have these saved and available whenever a report needs to be generated. This requires a bit of extra work in the beginning, and can easily be updated as needed, but makes extracting the data in a way that's convenient for the client much more convenient for the person extracting the data.