MediaGateway SDK Flash chat example

Download:

05_ChatExample.zip

This page demonstrates how to build a text/audio/video chat program between more Flash clients using the Ozeki VoIP SIP SDK. After reading this article, you will be completely familiar with all terms concerning chat programs and you will be able to build your own solution using Ozeki VoIP SIP SDK.

Introduction

One of the most popular Internet communication solutions is the chat that allows the users to communicate with each other directly. However, the Internet chat programs a are mainly using text messaging, you can create more sophisticated solutions that also can use voice and video communication (Figure 1).

If you want to implement a communication-related software, especially if it is capable of sending and receiving audio and video data, you will need to do a lot of background work for supporting audio and video codecs, streaming, displaying, etc. or you can use an SDK that provides you all of this. Ozeki VoIP SIP SDK is an extraordinary tool that gives you all the support you would need when building your chat program.

The chat application is a client-server solution as the client programs cannot communicate directly with each other by sending audio and video data. The server is used for establishing the connection between the clients and for transferring the audio and video streams to and from the clients.

If you want to use your chat solution within a web browser, you will need to use a multimedia environment that was designed for this purpose. One of the most widespread multimedia environments is Flash. Flash is the product of Adobe (formerly Macromedia) that uses a special language and can play multimedia data within a web browser by using the free Adobe Flash Player plugin that is available for most web browsers.

The sample program this article introduces uses Flash as the client-side technology. This means that the program's graphical user interface (GUI) will be built and controlled within Flash.

The server of the sample program is a console application written in C#. As the server itself is only a connection provider and supporter, it does not need a GUI. It is written in Visual Studio 2010 and uses the extraordinary background support of Ozeki VoIP SIP SDK so basically, the server only contains some method calls and setting for the communication.

Before you download the sample program, you will make sure you have all the requirements the program needs. The following list will show you all the software and hardware requirements of the program.

System requirements for the sample program

Operating systems:

Microsoft Windows XP
Microsoft Windows Server 2003
Microsoft Windows Server 2008
Microsoft Windows Vista
Microsoft Windows 7

Developer environment:

Microsoft Visual Studio 2010
Microsoft .Net Framework 3.5 or 4.0
Flash builder 4.5 or other equivalent IDE
Flash player 10 +

Internet connection

Hardware requirements:

1 GHz or faster processor
1 GB RAM (32-bit) +

How does an Ozeki VoIP SIP SDK supported chat server build up?

You should create the server as a console application in Visual Studio. For this purpose you can use the File->New Project.. menu from the menu bar (Figure 2).

You will need to choose the Console Application from the list on the New Project Window. You can set the project name and the location folder on this window. When you are ready, press the OK button and your server will be created.

In order to get the extraordinary background support of Ozeki VoIP SIP SDK, you will need to register it to your server project. This can be done on the Solution Explorer panel that is usually on the right side of the Visual Studio Window.

You will need to right click on the Resources label on the Solution Explorer panel, choose Add Reference... (Figure 4) and browse for the Ozeki VoIP SIP SDK on your hard drive. If you use the sample program of this article, the MediaGateway SDK will be in the downloaded project folder.

In C# programming language you define the names (classes, interfaces) in namespaces. If you want to use a name outside the namespace, you need to label it with the namespace path or you need to use using lines that allow you to use the names from different namespaces without labeling.

The tools Ozeki VoIP SIP SDK provides are in the Ozeki.MediaGateway namespace and its sub-namespaces. In order to use these without labeling, the using lines shown in Code 1 should be added in the using section of your server class.

using Ozeki.MediaGateway;
using Ozeki.MediaGateway.Config;

You will need to set the MediaGateway class as parent for your server class. This ensures that your server can use the methods defined in MediaGateway class without any override or change.

The server has to keep track of the connected clients. For this purpose it uses a C# collection type that is a Dictionary object. The Dictionary is basically a Hash table that stores the elements like key-value pairs. In this case the key will be a nickname the client chooses and the client object itself will be the value.

The Dictionary is a generic type that can be declared like it is seen in Code 2. The connected clients will be added to this Dictionary object and when disconnected, they will be deleted from it.

private Dictionary<string,IClient> chatClients;

The server functionality can be divided into different function groups. One of the main functions of the server is the client handling that means the server receives connection requests from the clients, establishes the connection and handles the data stream playing between the clients.

The methods needed for handling the clients from server-side

When a client sends a connection request to the server the OnClientConnect method will run. This method gets the client object as the first parameter and a nickname as the second. The method simply checks if the nickname is set and if the client table (Dictionary object) doesn't contain the nickname it adds the client to the table and notifies the client about the registration (Code 3).

The server is a console application, so it writes the messages on the console. The Dictionary is a table that can contain every key only once, so if the nickname (as it is the key) is already used by a client, the new client will not be able to connect with it.

If the client's nickname isn't in the chatClients table, the server adds the client to the pool and sends message about the succeeded connection to all connected clients by calling the ConnectedClientChanged method that invokes a client-side method that notifies the client about the connection success.

public override void OnClientConnect(IClient client,  object[] parameters)
{
    string nickname = parameters[0] as string;

    if (String.IsNullOrEmpty(nickname))
        return;
    Console.WriteLine("New client '{0}' is trying connect.", nickname);

    if (!chatClients.ContainsKey(nickname))
    {
        chatClients.Add(nickname,client);
        Console.WriteLine("Client '{0}' connected successfully.", nickname);
        ConnectedClientChanged(client);
        base.OnClientConnect(client, parameters);
    }
    else
    {
        Console.WriteLine("Nickname: '{0}' already has been used.",nickname);
    }
}

The disconnection request of a client is handled in the OnClientDisconnect method (Code 4). When a client sends a disconnection request to the server, it will be deleted from the chatClients Dictionary.

The server runs through the whole table and when it finds the right client, it removes it from the Dictionary. It also sends message to the connected clients about the succeeding disconnection by calling the ConnectedClientChanged method.

if (chatClients.ContainsValue(client))
{
   foreach (KeyValuePair<string, IClient> keyValuePair in chatClients)
{
    if (keyValuePair.Value==client)
    {
        Console.WriteLine("'{0}' client disconnected.", keyValuePair.Key);
        chatClients.Remove(keyValuePair.Key);
        break;
    }
}
}
ConnectedClientChanged(client);
base.OnClientDisconnect(client);

The server also needs to be able to enlist all connected clients that is made in the GetConnectedClients method. This is essential when the server wants to notify all connected clients about some change like a new connected client or a client's disconnection. This action is implemented in the ConnectedClientChanged method.

The server uses the OnStreamPublishStart method when a clients starts to send media data to another. This method only writes a notification message on the console and calls the OnStreamPublishStart method that is defined in the MediaGateway class.

The chat programs are usually capable of sending text messages. The server has to support this functionality too.

The server support for text message sending

Text messages could be sent between to clients directly without the server support, however the connection must be established by the server in that case too. When you want to implement a chat program, however, it is easier to use the server for transferring the text messages too, as in this case the server will handle the target client for you.

The SendText method that is shown in Code 5 is the server-side support for text message sending in the chat application. It receives the client, the sender and the target clients' nickname and the message as parameter.

The server tries to choose the target client from the connected clients and if the nickname is in the clients Dictionary, it sends the message to the target client with invoking the ReceiveMessage function that is defined in the client-side code.

if (chatClients.ContainsKey(target))
    {
        IClient cl;
        chatClients.TryGetValue(target,out cl);
        cl.InvokeMethod("ReceiveMessage", owner, msg);
    }

Video stream sending is not a usual function of a chat program, but Ozeki VoIP SIP SDK also provides support for this task, so you can have video-chat too. This will also needs some basic server-side support.

Camera handling with Ozeki VoIP SIP SDK

When a client wants to start or finish video chat with another, it has to send a camera request to the other side. The server transfers the camera request of the client to the target by running the SendCameraRequest method (Code 6).

This method gets the sender client object, the sender and the target cilents' nicknames and a boolean parameter that indicates if the request id for starting or stopping the camera.

if (chatClients.ContainsKey(target))
    {
        IClient cl;
        chatClients.TryGetValue(target, out cl);
        cl.InvokeMethod("CameraRequestReceived", owner, isEnable);
    }

The camera request is answered by a camera response that is implemented on server-side in the SendCameraResponse method. This method invokes the CameraResponseReceived client-side function that will inform the client about the answer of the remote client.

The SendCameraResponse has four parameters. It gets the sender client object, the sender and the target clients' nicknames (these are in reverse order as they were in the SendCameraRequest method parameter list)(and a boolean parameter that indicates if the remote client accepted or rejected the camera request.

if (chatClients.ContainsKey(target))
    {
        IClient cl;
        chatClients.TryGetValue(target, out cl);
        cl.InvokeMethod("CameraResponseReceived", owner, response);
    }

Ozeki VoIP SIP SDK also gives you support for voice chats between the clients. You will need to define this function both client- and server-side like in the case of text messages and video streaming.

Server-side support for voice chats

The audio stream sending or finishing also needs some setting on both clients' side. When a client wants to send audio data, it has to send an audio request first in order to be able to use the remote speaker for the voice playing.

The audio request is sent to the remote client by running the SendAudioRequest method on server-side. This method invokes the AudioRequestReceived client-side method of the target client.

The SendAudioRequest method gets four parameters. The first parameter is the sender client object. The second and third parameters are the nicknames of the sender and the target clients' and the last parameter is a boolean that indicates if the sender wants to start or finish the voice chat.

The server checks if the target nickname is in the connected clients Dictionary, and if it is there it sends the request to it by invoking the AudioRequestReceived function.

 if (chatClients.ContainsKey(target))
    {
        IClient cl;
        chatClients.TryGetValue(target, out cl);
        cl.InvokeMethod("AudioRequestReceived", owner, isEnable);
    }

The audio request is also answered with a response message that is implemented in the SendAudioResponse server-side method. It calls the AudioResponseReceived client-side function if the target client is connected.

The SendAudioResponse gets the client object, the sender and receiver clients' nickname and a boolean value as parameter. The last boolean parameter indicates if the remote client accepted or rejected the request.

if (chatClients.ContainsKey(target))
    {
        IClient cl;
        chatClients.TryGetValue(target, out cl);
        cl.InvokeMethod("AudioResponseReceived", owner, response);
    }

At this point your server only needs to specify what kind of clients it will accept. The Ozeki.MediaGateway.Confic namespace has a class called MediaConfig that can define the acceptable client types for a server.

The above introduced server application can accept both Flash and Silverlight clients. You only need to write some code line shown in Code 10 into the Main method to indicate the acceptance.

MediaGatewayConfig mediaConfig = new MediaGatewayConfig();
mediaConfig.AddConfigElement(new SilverlightConfig("0.0.0.0", 4502));
mediaConfig.AddConfigElement(new FlashConfig());

The given mediaConfig object will be the parameter of the ChatGateway that will be the fundament of the whole server. The ChatGateway is a class provided by Ozeki VoIP SIP SDK for chat programs.

In order to make your server work, you will need to add the two lines shown in Code 11 to the Main method and when you run the server project, the ChatGateway will start functioning.

var mediaGateway = new ChatGateway(mediaConfig);
mediaGateway.Start();

Your server application is ready for working. Now, you can write the client-side solution that will be used within a web browser. This server can accept both Silverlight and Flash clients. This article shows how you can build the Flash version of the client.

The client-side application uses Flash technology

The client-side solution of this chat application is written in Flash. As the client is the program the customer will meet, it needs an impressive GUI. Adobe Flash Builder provides built-in support for GUI building, so you will be able to create the GUI with your mouse and some property settings.

The GUI has to contain the tools for all of the functions the chat client has. The sample program that is introduced in this article works with three GUI panels. There is a main panel that contains the basic functions and two pop-up windows. One of these will show notifications and the other is for setting the client's nickname.

Figure 5 shows a potential main panel for the chat client. Let's see how it builds up. It mainly contains essential GUI elements but as for the video displays it will contain a special element that is provided by Ozeki VoIP SIP SDK.

If you register Ozeki VoIP SIP SDK to the client project, you can use the tools it provides, even the GUI elements it defines for video stream display.

The GUI in Figure 5 can be divided into 3 columns. The left-side column contains the names of the connected clients (including the local client nickname on a separate label). The right-side column contains the video support, aka the display for the local and the remote video stream, and the central column contains the basic text messaging functions and the function buttons.

The left-side of the GUI contains two panels with the titles "Your nickname" and "Connected users". The Your nickname panel has a single label on it that shows the nickname the client has set before connecting to the server.

The Connected user's panel contains a TextArea GUI element that contains all the connected clients' nicknames. When a client connects or disconnects, this TextArea is updated from a server-side method invocation.

The central section of the main GUI is set of a TextArea that displays the chat log with all the clients you have chatted in text messages and a Group object that contains the TextArea on that you type in your messages and the function buttons.

The Flash client contains three function buttons, one for each message types (text, audio and video). The Send button is for sending the text messages to the selected client (you can select a client from the list on the left-side panel).

You can send audio and video requests with the Enable audio and Enable video buttons. These buttons' texts modify when the audio or video is enabled. In those cases their texts will be Disable audio and Disable video and you can send audio stop or video stop media requests with them to the selected client.

The right side of the main window consists of two panels with the titles "Local camera" and "Remote camera". These panels contain a special display object that is for displaying video streams. This GUI element is a VideoDisplayEx that is an Ozeki VoIP SIP SDK provided tool with all the support for video playing.

This sample chat program starts with a pop-up window that asks for a nickname for the client to connect with. This nickname will be sent to the server that registers the connection request and puts the client's data into the client's pool that is a Dictionary object.

Figure 6 shows the window that pops up when the client is started. It is simple window with some labels and a TextArea where you can type in your nickname. If you gave the nickname you can press the Connect button and send your connection request to the server.

This pop-up window is defined as a separate class in the sample Flash client project and its only function is to connect to the server.

The client GUI needs a third separate Flash object that will be an informing pop-up window about the incoming media requests (Figure 7). This window only contains a label that informs the client about the media request and two buttons, with what you can accept or reject the request.

After having your impressive GUI built, you can start to write the background code. As you use Ozeki VoIP SIP SDK, this will only means some function calls and settings.

First of all, you will need to import the namespaces of the Ozeki VoIP SIP SDK in order to use the provided tools without labeling with the path (Code 12).

import ozeki.mediaGateway.MediaConnection;
import ozeki.mediaGateway.MediaStreamReceiver;
import ozeki.mediaGateway.MediaStreamSender;

As the client starts with the appearance of a pop-up window, you will need to write the eventHandler that runs in case of the project running start. This is the creationCompleteHandler function that should contain the instructions that are shown in Code 13.

The creationCompleteHandler function will run right after all the presettings are done in connection with the project start. It will initialize the window that asks for the nickname and makes it appear.

connectedUsers = new ArrayCollection();
listConnectedUsers.dataProvider = connectedUsers;

var window:WindowConnection=new WindowConnection();
window.addEventListener(EventConnectedSuccessfully.CONNECTED_SUCCESSFULLY, window_ConnectedSuccessfully);
PopUpManager.addPopUp(window,this,true);
PopUpManager.centerPopUp(window);

Text message sending to a selected client

There are three functions related to the text message sending. There is one for sending the message, one for receiving the messages from the remote clients and one for listening the status change of the typing TextArea.

The ReceiveMessage (Code 14) is a simple function that puts the received text message onto the message log TextArea. This message is invoked from the server-side when a remote client sends a text message to your client.

txtChatLog.text += StringUtil.substitute("{0}:\n{1}\n", owner, message);

When you type in a message, you can send it by pressing the Send button on the GUI. This will invoke the onclick event of the button and the btnSend_clickHandler function (that is the EventHandler for the onclick event) will run (Code 15).

This method appends your message onto the message log TextArea and also sends it to the selected client through the server. The server will receive the message and will invoke the remote client's ReceiveMessage function. This method also empties the typing area when the message has been sent.

if (listConnectedUsers.selectedItem != null)
        {
                txtChatLog.text += StringUtil.substitute("{0}:\n{1}\n", lblNickName.text, txtMsgInput.text);
                connection.invokeOnConnection("SendText", lblNickName.text, listConnectedUsers.selectedItem, txtMsgInput.text);
                txtMsgInput.text = "";
        }
        else
                Alert.show("You must select a connected user to send message.");

The GUI has to work properly and that's why the client has to be notified if you have typed something info the typing area. The txtMsgInput_keyUpHandler function does this trick for you (Code 16).

if (event.keyCode == Keyboard.ENTER)
btnSend_clickHandler(null);

The text message sending is only one of the three functions of this sample chat client. Let's see how you can make it capable for voice chatting.

Audio chat function in the client solution

Voice chatting is the second function of this sample chat program. You will need to handle the incoming and the outgoing voice streams. Voice chat support will need three specific functions, one for the audio request sending, one for audio request receiving and one for audio response receiving.

If you want to start a voice chat, you will need to press the Enable audio button and it will send an audio request to the selected client, you need to press the same button for finishing voice chats too for that matter.

The onclick event of the Enable audio button is the btnAudio_clickHandler function that is shown in Code 17. This method sends a start or a stop audio request to the remote client according to the current audio status.

If a voice chat is established, the button's text will be "Disable audio" and it will send a stop request to the remote party. In this case the audio stream and the microphone will stop and the button will get back the original "Enable audio" text.

If there is no audio stream set, the button will send a start audio message and if the remote client accepts it, the voice chat can be started.

if (audioIsEnable)
        {
                streamSender.attachMicrophone(null);
                connection.invokeOnConnection("SendAudioRequest", lblNickName.text, audioEnableWith, false);
                btnAudio.content = "Enable audio";
                audioIsEnable = false;
                if (!cameraIsEnable)
                        DestroyStreamReceiver();

                return;
        }
        if (listConnectedUsers.selectedItem != null )
        {
                audioIsEnable = !audioIsEnable;
                txtChatLog.text += "Please wait for other person response.\n";
                connection.invokeOnConnection("SendAudioRequest", lblNickName.text, listConnectedUsers.selectedItem, audioIsEnable);
        }
        else
                Alert.show("You must select a connected user, to enable audio chat.");

When a client sends an audio request to you, the server invokes the AudioRequestReceived function that informs you about the request. This will make the media request window appear, where you can accept or reject an audio start request. In case of a stop request the audio stream will stop automatically (Code 18).

if (isEnable)
        {
                winMediaReq = new WindowMediaRequest();
                winMediaReq.initDependency(owner, MediaType.AUDIO);
                winMediaReq.addEventListener(EventResponseSelected.RESPONSE_SELECTED, winMediaReq_ResponseSelected);
                PopUpManager.addPopUp(winMediaReq,this,true);
                PopUpManager.centerPopUp(winMediaReq);
        }
        else//close audio conversation
        {
                streamSender.attachMicrophone(null);
                if (!cameraIsEnable)
                        DestroyStreamReceiver();
                txtChatLog.text += StringUtil.substitute("{0} disabled his/her microphone.\n", owner);
                btnAudio.label = "Enable audio";
                audioIsEnable = false;
        }

When you send an audio request to a selected client it can accept or reject it on the pop-up window and the answer will be sent back by the server invoking the AudioResponseReceived function (Code 19). If the client accepts the start audio request, the voice chat can be started, otherwise nothing happens except you are informed about the rejection.

if (response)
        {
                createAndSetupStreamReceiver(owner, MediaType.AUDIO);
                streamSender.attachMicrophone(Microphone.getMicrophone());
                audioEnableWith = owner;
                txtChatLog.text += StringUtil.substitute("{0} accepted your audio request.\n", owner);
                btnAudio.label = "Disable audio";
                audioIsEnable = true;
        }
        else
        {
                txtChatLog.text += StringUtil.substitute("{0} rejected your audio request.", owner);
        }

The voice chat function of the client is ready to use, there is only one step left for the video chat and a simple function for the media answer settings.

Video phoning from the chat client

The video chat function seems to be the most complex one in this chat project, however, Ozeki VoIP SIP SDK and Flash provides all the background, so you only need to call some functions.

The camera handling and video chat functions will be built up in the same way as the voice chat functions. The video chat functions are btnCamera_clickHandler, CameraRequestReceived and CameraResponseReceived.

These functions work the same way as the audio versions, except for instead of the microphone, the camera will be turned on and off by the functions.

The last function to be written is the media answer implementation that will send the client's answer to the server that transfers it to the remote client.

Sending response to the remote client's media request

The winMediaReq_ResponseSelected (Code 20) function will send your media answer to the remote client. The answer will be formed according to the media request you have received and the answer you chose on the pop-up window.

switch (event.mediaType)
{
        case MediaType.VIDEO:
                txtChatLog.text += "Camera request accepted\n";
                createAndSetupStreamReceiver(event.owner, MediaType.VIDEO);
                var cam:Camera = Camera.getCamera();
                streamSender.attachCamera(cam);
                dispLocal.attachCamera(cam);
                dispRemote.attachNetStream(streamReceiver);
                btnCamera.label = "Disable video";
                cameraIsEnable = true;
                cameraEnableWith = event.owner;
                break;
        case MediaType.AUDIO:
                txtChatLog.text += "Audio request accepted\n";
                streamSender.attachMicrophone(Microphone.getMicrophone());
                createAndSetupStreamReceiver(event.owner, MediaType.AUDIO);
                btnAudio.label = "Disable audio";
                audioIsEnable = true;
                audioEnableWith = event.owner;
                break;
}

Now, your chat program is ready to use and fully capable of working with text messaging, voice and video chats. Now it is time to explore more of the extraordinary support of Ozeki VoIP SIP SDK.

Summary

The main purpose of this sample program was to demonstrate how to transfer media data by using the Ozeki VoIP SIP SDK. It includes cases when some events are not handled effectively (because that was not the purpose), but these cases do not affect the fact that the use of Ozeki VoIP SIP SDK you can develop the requested application faster and in a more efficient way.