Creating a Facebook Messenger BOT with Laravel 5

Keeping in constant communication with your customers is a necessity for most businesses now days. Most companies do this via email and SMS messaging, but keeping in contact via Facebook Messenger can also be a great tool in increasing sales and customer satisfaction.

Even better, having a Facebook Messaging BOT to automate your customer’s initial incoming messages can cut down on costs, and help your company be more productive and efficient.

Over the last few months I was tasked with integrating Facebook’s Messenger API into a popular SaaS platform, so this tutorial will get your feet wet with the Messenger API in general, and then show you how to get started integrating the Messenger API with your company’s platform using Laravel 5.


** NOTE: For this tutorial, you will need a Facebook Developer account


First create a Facebook developer account (if you don’t already have one), then create an app, a page, and then add the messenger product.

This Quick Start Guide will guide you through the Facebook app procedure quickly.


Alright… Now let’s get started!


First we need to create our routes..
Create a route group in your routing file and add the following two routes.

app/Http/routes.php


Route::group(['prefix' => 'facebook'], function () {
    Route::get('message', 'FacebookController@verify');
    Route::post('message', 'FacebookController@message');
});

Next up, just like any Laravel route, we need to create our Facebook Controller with the following two methods that the above routes are referencing.

After our dependencies are injected in our constructor, we have our facebookVerify method. This method is used in order for our Facebook Messenger app’s webhook to verify that our route is alive and returning
a 200 response.


** NOTE: To understand why this method is needed, refer to the Facebook Messenger API docs.


Our second method, facebookMessage is the entry point for all incoming messages from our Facebook User.

First, we get our Request, process the object’s JSON and then pass our payload off to our message handler. Our message handler is located in in our Facebook Manager service.

Let’s take a look.

app/Http/Facebook/FacebookController.php


namespace App\Http\Controllers\Facebook;

use App\Http\Controllers\Controller;
use App\Services\Facebook\FacebookManagerIncomingMessageService;

use Illuminate\Http\Request;
use Illuminate\Http\Response;
use Illuminate\Routing\ResponseFactory;

class FacebookController extends Controller
{
    private $facebookManagerIncomingMessageService;
    private $responseFactory;

    public function __construct(
        FacebookManagerIncomingMessageService 
            $facebookManagerIncomingMessageService,
        ResponseFactory $responseFactory)
    {
      $this->facebookManagerIncomingMessageService = 
          $facebookManagerIncomingMessageService;
      $this->responseFactory = $responseFactory;
    }

    /**
     * @param Request $request
     * @return \Illuminate\Http\JsonResponse
     */
    public function facebookVerify(Request $request)
    {
        $verified = $this
            ->facebookManagerIncomingMessageService
            ->incomingMessageVerify(
                $request->hub_challenge, 
                  $request->hub_verify_token);

        if (!empty($verified)) {
            echo $verified;
        } else {
            return $this
                ->responseFactory
                ->json(Response::HTTP_FORBIDDEN);
        }
    }

    /**
     * @param Request $request
     * @return \Illuminate\Http\JsonResponse
     */
    public function facebookMessage(Request $request)
    {
        $message = $request->json()->all();

        try {
            $this
              ->facebookManagerIncomingMessageService
              ->incomingMessageHandler($message);
        } catch (\Exception $e) {
            return $e;
        }

        // we want to always send a 200 back 
        // to the messenger api - even with 
        // an issue on our side or messages 
        // start backing up and everything 
        // comes to a halt really quick
        return $this
            ->responseFactory
            ->json(Response::HTTP_OK);
    }
}

Now let’s take a look at the FacebookManagerIncomingMessageService.

This Service is used to process our incoming messages FROM the Facebook Messenger API, along with any handling (i.e. saving the message, creating a customer, etc).

Our entry point to our Facebook Manager service is the method incomingMessageHandler. This method contains a switch statement that is used to parse our JSON that is passed to it from the controller for the sender ID, the recipient ID, the actual message, and any other metadata that is needed for our application. For this tutorial all we care about is the sender and recipient IDS and the message. For more information on all the available fields reference the Facebook Messenger API docs.


Our next method setConversationParticipants is pretty self explanatory. It is used to set the IDs of both the sender and the recipient.


** NOTE: The sender ID and recipient ID reverse when we are parsing an ECHO message. To learn more about echo messages, and their purpose reference the Messenger API.


The next methods are our setFacebookPageByPageId and setFacebookUserBySenderId. These methods are used to set the recipient ID (non echo message), and sender ID (non echo message) in our class to use when needed. This let’s us know which message came from which party.


Code tracing through the code brings us to the createOrUpdateCustomer and getFacebookUserBasicInfo methods. These methods are used to check our database if our customer is already in our DB, and if so retrieve that model, if not we save this Facebook User as a new customer. Our Facebook User is referenced by the Facebook Sender ID that we parse out in the incoming JSON.

If this Facebook User is new and not a customer in our database, we then use our basic info method together with our Facebook User’s Sender ID to retrieve the user’s first name, last name, and avatar image from the Facebook Graph API.

Next is our setFacebookPageByPageId method. This method is how we keep track and reference the Facebook Page (your company’s page) that the Facebook User is talking to. In our Customer’s table we save the Facebook User’s senderID, and in our Page table, we save the Facebook Page’s page ID (usually the recipient ID if not an echo message).


OK… Enough talking… Let’s take a look at the actual code..

This is our FacebookManagerIncomingMessageService.

app/Services/Facebook/FacebookManagerIncomingMessageService.php


namespace App\Services\Facebook;

use App\Services\Facebook\FacebookManagerOutgoingMessageService;

use App\Contracts\Repositories\FacebookRepositoryInterface;
use App\Contracts\Repositories\CustomerRepositoryInterface;

use Illuminate\Contracts\Logging\Log;
use Illuminate\Database\Eloquent\ModelNotFoundException;

class FacebookManagerIncomingMessageService
{
    private $facebookManagerOutgoingMessageService;

    private $facebookPageRepository;
    private $customerRepository;
    
    private $facebookPage;
    private $facebookPageByPageId;

    private $customer;
    private $facebookUserBySenderId;
   
    private $message;

    private $log;

    public function __construct(
        FacebookManagerOutgoingMessageService $facebookManagerOutgoingMessageService,
        FacebookRepositoryInterface $facebookRepository,
        CustomerRepositoryInterface $customerRepository,
        Log $log
    ) {
        $this->facebookManagerOutgoingMessageService = $facebookManagerOutgoingMessageService;
        $this->facebookRepository = $facebookRepository;
        $this->customerRepository = $customerRepository;
        $this->log = $log;
    }

    /**
     * Verify the Facebook challenge.
     *
     * @param $challenge
     * @param $verifyToken
     * @return null
     */
    public function incomingMessageVerify($challenge, $verifyToken)
    {
        if ($verifyToken === $this
                ->facebookManagerOutgoingMessageService
                ->getVerificationToken()) {
            return $challenge;
        } else {
            return null;
        }
    }

    /**
     * Entry point for handling any incoming messages
     * from the Facebook Messenger Webhook
     *
     * @param array $message
     * @throws FacebookManagerIncomingMessageException
     */
    public function incomingMessageHandler(array $message)
    {
        $this->setMessage($message);

        if (!empty($message['object']) && 
          $message['object'] === 'page') {

          foreach ($message['entry'] as $pageEntry) {

            // iterate through each message and process
            foreach ($pageEntry['messaging'] as $message) {

             // set the correct sender & recipient
             $this->setConversationParticipants($message);

             switch ($message) {
                       
               // incoming facebook user message
               case (
                 !empty($message['message']) &&
                 empty($message['message']['is_echo']))):
                   $this->processFacebookUserMessage($message);
                   break;

               // incoming facebook echo message
               // echo messages are sent 
               // FROM 
               // the API when messages are sent 
               // TO
               // the API. Echo messages contain 
               // the message that was
               // (along with other metadata) 
               // to the API in order "echo" 
               // what was sent so you can 
               // process it (i.e. store the
               // message in your database)
               case (
                 !empty($message['message']) && 
                 !empty($message['message']['is_echo']):
                   $this->processFacebookPageMessage($message);
                   break;
                        
               default:
                   break;
            }
          }
        }
        return true;
    } else {
      throw new FacebookManagerIncomingMessageException(
        'Problem processing the incoming facebook message.');
    }
  }

   /**
    * Set our Facebook User's incoming message.
    * ** NOTE: Echo messages have the  
    * sender & recipient IDs reversed.
    */
    private function setMessage(
        $message)
    {
        $this->message = $message;
    }

    private function setConversationParticipants(
        $message)
    {
        // incoming facebook user visitor message
        if (empty($message['message']['is_echo'])) {
            $this->isEcho = false;
            $this->setFacebookPageByPageId($message['recipient']['id']);
            $this->setFacebookUserBySenderId($message['sender']['id']);
        } else { // this is an echo message - swap sender and recipient
            $this->isEcho = true;
            $this->setFacebookPageByPageId($message['sender']['id']);
            $this->setFacebookUserBySenderId($message['recipient']['id']);
        }
    }

    /**
     * Process the Facebook User's incoming message
     *
     * @param $message
     */
    private function processFacebookUserMessage($message)
    {
        return $this->saveMessageFromFacebookUser($message['message']);
    }
    
    /**
     * Check to see if there is a Facebook Page by this
     * Facebook Page ID. 
     * ** NOTE:This is the ID that is
     * associated with the Facebook Page. The Facebook
     * Page is the page that the user is sending messages to.
     *
     * @param $facebookPageId
     * @return mixed
     * @throws ModelNotFoundException
     */
    private function setFacebookPageByPageId($facebookPageId)
    {
  
        try {
            $facebookPage = $this->facebookPageRepository
                ->findByFacebookPageIdOrFail($facebookPageId);
        } catch (
            ModelNotFoundException $e) {
            throw $e;
        }

        return $facebookPage;
    }

    /**
     *  Check to see if there is an already existing
     *  customer for this Facebook customer
     *
     * @param $facebookSenderId
     * @return $this|null
     */
    private function setFacebookUserBySenderId(
        $facebookUserSenderId)
    {
        // check to see if we have a record of this customer already
        $this->customer = $this->customerRepository
            ->getFirstBy('facebook_sender_id', $facebookUserSenderId);

        // This Facebook user is not a customer yet
        // Create a new customer
        if (empty($this->customer)) {
            $this->customer = $this->createOrUpdateCustomer();
        }
    }

    /**
     * Create a new customer based on this Facebook User
     *
     * @return \App\Models\Customer
     */
    private function createOrUpdateCustomer()
    {
      $customerData = $this
        ->getFacebookUserBasicInfo();
       
      $customerData['first_name'] = 
        (!empty($customerFacebookData['first_name']))
          ? $customerFacebookData['first_name']
          : '';
      $customerData['last_name'] = 
        (!empty($customerFacebookData['last_name']))
          ? $customerFacebookData['last_name']
          : '';
      $customerData['avatar'] = 
        (!empty($customerFacebookData['profile_pic']))
          ? $customerFacebookData['profile_pic']
          : '';
      $customerData['facebook_sender_id'] = 
        (!empty($this->customerByFacebookSenderId))
          ? $this->customerByFacebookSenderId
          : '';

      if (empty($this->customer)) {
        $this->customer = $this
          ->customerRepository'
          ->create($customerData);
      } else {
        $this->customer->first_name = $customerData['first_name'];
        $this->customer->last_name = $customerData['last_name'];
        $this->customer->avatar = $customerData['avatar'];
      }

      $this->customer->save();

      return $this->customer;
  }

    /**
    * Use our Facebook Manager Outgoing Message
    * Service to send a message to the Facebook
    * Messenger API to query basic user info for
    * our Facebook User (first name, last name,
    * avatar, etc) and return this info so we
    * can use it to create our new customer.
    */
    private function getFacebookUserBasicInfo()
    {
        try {
            $response = $this
                ->facebookManagerOutgoingMessageService
                ->getFacebookUserBasicInfo(
                    $this->facebookUserSenderId, 
                    $this->facebookPage
                      ->facebook_page_access_token
                );

            $responseBody = $response->getBody();
            $facebookUserBasicInfo = json_decode($responseBody, true);

        } catch (\Exception $e) {
            $facebookUserBasicInfo = [];
        }

        return $facebookUserBasicInfo;
    }

   /**
    * Save or process the Facebook User's message
    */
    private function saveFacebookUserMessage($message) 
    {
      // ...
      // ...
      // Save or do something cool with your Facebook User's message.
      // ...
      // ...
    }

   /**
    * Save or process the Facebook Page's message
    */
    private function saveFacebookPageMessage($message) 
    {
      // ...
      // ...
      // Save or do something cool with the message that your Employee sent
      // FROM your Facebook Page to the 
      // Facebook User.
      // ...
      // ...
    }



** Note: This tutorial already assumes you have a Customer and FacebookPage Model created and that your database is in sync with them.



Now let’s take a look at our model retrieval method inside our Facebook Repository. This method is used to lookup our FacebookPage information (so the Facebook User (customer) is communicating with the correct Facebook Page (your employee)).

app/Repositories/EloquentFacebookRepository.php


namespace App\Repositories;

use App\Models\FacebookPage;

class EloquentFacebookRepository extends AbstractEloquentRepository
{
    // ...
    // ...

    public function findByFacebookPageIdOrFail($facebookPageId)
    {
        $facebookPage = $this
            ->model
            ->where(
                'facebook_page_id', $facebookPageId)
            ->first();
        if (is_null($facebookPage)) {
            $e = new ModelNotFoundException;
            $e->setModel(FacebookPage::class);
            throw $e;
        }

        return $location;
    }

    // ...
    // ...
}


Now that we know how to process incoming messages from our Facebook User (customer), let’s see how we can use our FacebookManagerOutgoingMessageService to send a message to the Facebook Messenger API from our Facebook Page (employee) and ultimately to our Facebook User (customer).

For HTTP requests we will be using Guzzle.


The first method getFacebookUserBasicInfo is how we query the Facebook Messenger API for basic Facebook User information. Reference the API docs to see all of the available information that you can retrieve with the Facebook User’s Sender ID.

The next method, our sendMessageToFacebookUser is used to send the actual message FROM our Facebook Page (employee) TO our Facebook User (customer). First, we retrieve our Page API key that is stored in our config file, our Page token (stored also in our FacebookPage table), and our Facebook User’s Sender ID, and then concatenate them along with the URI to send a message to our Facebook User.


Let’s take a look at the code…

app/Services/Facebook/FacebookManagerOutgoingMessageService.php


namespace App\Services\Facebook;

use App\Models\Customer;
use App\Models\;
use GuzzleHttp\Client as GuzzleClient;

class FacebookManagerOutgoingMessageService
{
 
    /** @var array facebook configuration */
    private $facebookConfig;
    private $guzzleClient;

    public function __construct(
        array $facebookConfig,
        GuzzleClient $guzzleClient)
    {
        $this->facebookConfig = $facebookConfig;
        $this->guzzleClient = $guzzleClient;
    }

    /**
     * Get the Facebook verification 
     * token from our config
     *
     * @return mixed
     */
    public function getVerificationToken()
    {
        return $this->facebookConfig['app_verification_token'];
    }

   /**
     * Get the Facebook User basic profile info
     *
     * @param $visitorByFacebookSenderId
     * @param $locationByFacebookPageAccessToken
     * @return mixed|\Psr\Http\Message\ResponseInterface
     * @throws Exception
     */
    public function getFacebookUserBasicInfo(
      $facebookUserBySenderId, $facebookPageAccessToken)
    {
        $apiUrl =
            $this->facebookConfig['graph_user_profile_api_uri'].
            $facebookUserBySenderId. '?access_token='.
            $facebookPageAccessToken;

        try {
            $response = $this
                ->guzzleClient
                ->request('GET', $apiUrl);
        } catch (\Exception $e) {
            throw $e;
        }

        return $response;
    }

    /**
     * Send the message to our the Facebook messenger API
     * and ultimately the Facebook User we're communicating
     * with.
     *
     * @param FacebookPage $facebookPage
     * @param array $requestArguments
     * @return mixed|\Psr\Http\Message\ResponseInterface|void
     * @throws Exception
     */
    private function sendMessageToFacebookUser(
        FacebookPage $facebookPage, array $requestArguments)
    {
        $apiUrl =
            $this->facebookConfig['graph_messenger_api_uri'].
            $facebookPage->facebook_page_access_token;

        try {
            return $this->guzzleClient->request('POST', $apiUrl, $requestArguments);
        } catch (\Exception $e) {
            throw $e;
        }
    }


And finally a quick look at your Facebook Configuration. Add the following (update with your values) the config tokens in your .env file.


// ...
// ...
// ...
FACEBOOK_APP_VERIFICATION_TOKEN = my_voice_is_my_password_verify_me
FACEBOOK_GRAPH_MESSENGER_API_URI = https://graph.facebook.com/v2.8/me/messages?access_token=
// ...
// ...
// ...


Now the only value that will be pulled from your database will be the Page Access Token, the Page ID, for your Facebook Page, and the Facebook User’s Sender ID for your Customer.


And that should be it! This was a long tutorial, so if you see any mistakes or have any questions on something that’s not clear and confusing… Just ping me! I’ll be glad to help.. 😀


Have fun coding! 😀

Adding Symfony 2 Routing to your legacy PHP application

Recently I’ve had to rewrite portions of a PHP legacy application that was written in a company’s proprietary legacy framework, in order for it to be test friendly and much more maintainable.

In order to accomplish this, I’ve added a few key Symfony components to their framework. The result was that the application was easier to test, easier to cache, and much much easier to maintain than it was previously.


Let’s take a look.


Their front controller was stripped and completely rewritten to implement Symfony’s HTTP Foundation and Symfony’s Routing.

Let’s take a look at a snippet of the front controller.
New Front Controller


include '../vendor/autoload.php';

use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\RouteCollection;
use LtApp\LtAppRoute;
use LtApp\LtAppKernel;

$routeCollection = new RouteCollection();
// inject the Symfony 
// route collection to our 
// App Route class (shown below)
$ltAppRoute = new LtAppRoute($routeCollection); 
$routes = $ltAppRoute->getRoutes();

// using a dependency injection container
// will be covered in another tutorial
$diContainer = include __DIR__.'/../app/LtAppContainer.php';

// create our Request object
$request = Request::createFromGlobals();

// Retrieving the LtAppKernel 
// service from my DI container.
// If you're not using a DI container, 
// you can just instantiate your version 
// of AppKernel (shown below) in place of this.
// Our response object is then returned 
// to the client.
$response = $diContainer
    ->get('ltAppKernel')
    ->handle($request);

// enable caching for our response
//$response = $diContainer
    ->get('ltAppKernelWithCache')
    ->handle($request);

// phone home
$response->send();

exit;


Now let’s take a look at our App Kernel.

Our App Kernel is the engine that will dynamically instantiate our application’s Routing and Controller objects. It is responsible for asking our matcher if the incoming Request object contains a URL string with a valid route, and if so, resolving the correct controller for that valid route.

Let’s take a look at a snippet of our App Kernel.

App Kernel


namespace LtApp;

use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\HttpKernelInterface;
use Symfony\Component\Routing\Matcher\UrlMatcherInterface;
use Symfony\Component\Routing\Exception\ResourceNotFoundException;
use Symfony\Component\HttpKernel\Controller\ControllerResolverInterface;

class LtAppKernel implements HttpKernelInterface
{
    private $matcher;
    private $resolver;
    private $dispatcher;

    // Injecting in our Url Matcher
    // and Controller Resolver.
    // Again this injected from my DI container. 
    // If you aren't using a DI container, 
    // just pass the interface objects as 
    // arguments when you instantiate 
    // your App Kernel from inside
    // your front controller.
    public function __construct
    (UrlMatcherInterface $matcher,
     ControllerResolverInterface $resolver)
    {
        $this->matcher = $matcher;
        $this->resolver = $resolver;
    }
    
    // Our handle method takes an 
    // HTTP Request object, 
    // the Symfony Http Kernel
    // so we have access 
    // to the master request, 
    // and a catch flag 
    // (we'll see why below)
    public function handle(
        Request $request, 
        $type = HttpKernelInterface::MASTER_REQUEST, 
        $catch = true)
    {
        // Next we take our HTTP request object and 
        // see if our Request contains a 
        // routing match (see our routes class 
        // below for a match)
        try {
            $request
            ->attributes
            ->add(
              $this
                ->matcher
                ->match(
                  $request
                    ->getPathInfo()
            ));

            // Our request found a match 
            // so let's use the Controller
            // Resolver to resolve our 
            // controller.
            $controller = $this
                           ->resolver
                           ->getController($request);
            
            // Pass our request arguments 
            // as an argument to our resolved 
            // controller (see controller below).
            // If you have form data, the resolver's
            // 'getArguments' method's functionality
            // will parse that data for you and 
            // then pass it as an array to your 
            // controller.
            $arguments = $this
                          ->resolver
                          ->getArguments(
                             $request, $controller);

            // Invoke the name 
            // of the controller that
            // is resolved from a match
            // in our routing class
            $response = call_user_func_array(
                          $controller, $arguments);

        } catch (ResourceNotFoundException $e) {
            // No such route exception 
            // return a 404 response
            $response = new Response(
                'Not Found', 404);
        } catch (\Exception $e) {
            // Something blew up exception
            // return a 500 response
            $response = new Response(
                'An error occurred', 500);
        }
        
        //**Note:
        // If you need any event listeners for specific
        // actions to take on your response object 
        // after returned from your business logic 
        // / controller and before you return 
        // to the client you can invoke them here.
        
        return $response;
    }
}

Now let’s take a look at the routing class that our App Kernel was
calling.
Routing


namespace LtApp;

use Symfony\Component\Routing\RouteCollection;
use Symfony\Component\Routing\Route;

class LtAppRoute
{
    private $routes;

    // pass in Symfony's 
    // Route Collection object
    public function __construct(
      RouteCollection $routeCollection)
    {
        $this->routes = $routeCollection;
        // ignore routeConfig
        // this I'm using for
        // something else
        $this->routeConfig(); // decouple later
    }

   
   // We register a route by
   // invoking the add method
   // with specific arguments.
   // The first argument is a 
   // unique name for this route.
   // Next we instantiate a Route
   // object with our route.
   // Our route is /docs/{ID}
   // Our associative array contains 
   // the controller that we
   // want to resolve with this route,
   // DocumentController and the 
   // method on that controller
   // that this particular route 
   // invokes viewDocumentAction.
   // separated by the :: 
   // token.
   // We can also pass in arguments
   // that contain a default value
   // if needed, and they will be
   // available in our method's 
   // parameters.
    private function routeConfig()
    {
     $this->routes
     ->add(
      'document_view', 
      new Route('/docs/{documentId}', 
      array(
      '_controller'=>
        'Lt\\DocumentPackage
           \\Controller
           \\DocumentController::
             viewDocumentAction',
      'action' => 
        'documents', // default value
      'documentId' => 
        '1' // default value
     )));
}


DocumentController (our resolved controller)


class DocumentController
{
  // ...
  // ...
  public function viewDocumentAction(
    Request $request, 
            $action, 
            $documentId)
  {
     // ... Grab services and models and
     // ... perform cool business logic
     $response = new Response();
     
     // return response object
     // back to App Kernel
     return $response;
  }
}


And there you have it. As you can see, this design decouples a lot of functionality from the application, which in this case, makes writing tests actually possible, where before automated testing wasn’t really possible.

And.. If you combine this with a Dependency Injection container, the result is even more flexibility and a code base that is much more maintainable.

Have fun coding! 😀

Supercharged PHP development with Vagrant, VMware, and IntelliJ IDEA

In my development environment I’ve been using Vagrant with Virutalbox for a while now; however, even with NFS, with the VM properly provisioned with enough resources, and running under Linux, the environment’s performance was still just barely tolerable. It’s performance compared to Ubuntu natively on a Samsung 850 pro SSD left me wishing for more.. A lot more.. 🙂


This tutorial will show you how to create a high performance PHP (or any language really) development environment using the JetBrains IntelliJ Idea IDE (with the PHP plugin), Vagrant 1.8.1 (latest version as of this post), VMware Workstation Pro 12, and good ole Ubuntu Trusty (14.04.3 with the vivid HWE stack). 😀 We will be using the vivid 3.19.43 kernel.


We also will be using native PHP 5.5.9 (but php-fpm, PHP 7, or HHVM would be even more awesome 😀 ) and XDebug as our remote debugger.


**Note: This tutorial assumes you have Vagrant, VMware, and Jetbrains IntelliJ Idea already installed and functioning on your host OS. It also assumes you already know how to install a guest OS in VMware.


This tutorial will be structured:

  1. First update/install all necessary Ubuntu packages in the VM.
  2. Configure our VM according to Vagrant standards.
  3. Configure Xdebug for Apache (although Nginx is a similar process).
  4. Turn our newly configured Ubuntu Virtual Machine into our own Vagrant base box with VMware as the provider.
  5. Configure our JetBrains IntelliJ Idea IDE (PHPstorm will work also) to work with our remote PHP interpreter and remote Xdebug debugger running in the VM.
  6. Lastly we will benchmark our VM development environment’s performance using wrk.

We have a lot to do… So… Let’s get started! 😀


**Note: This tutorial will use VMware Workstation 12 (but other VMware versions will probably work as well).


**Note: This tutorial also assumes you have a valid Vagrant VMware plugin license in order to interface Vagrant with VMware Workstation.


First do a minimal install of Ubuntu Trusty as a guest OS Virtual Machine using VMware Workstation.

If you don’t know how to do that, there are many tutorials available.

Pick one, install Ubuntu 14.04, and then come back. 🙂


**Note: When you install Ubuntu, since we’re creating a Vagrant base box, it is usually best to follow the Vagrant standard.

If you plan to share your base box with the public, using the Vagrant insecure settings is the standard way to configure it. This means the user should be vagrant with password vagrant, and the root password should be set to vagrant as well.

However, if this base box is only for your team and you don’t want it to be shared with anyone outside of your organization, then obviously the standard settings should be set to something much more secure.



OK….. So….. 🙂

At this point you should now have a minimal install of Ubuntu 14.04.3 (14.04.2 is fine also).


We’re now going to update the system, install the Ubuntu HWE stack, and then install all necessary packages for our LAMP stack and Vagrant Base Box.

Update repos and install/upgrade Packages:

vagrant@trusty:~$ sudo apt-get update

vagrant@trusty:~$ sudo apt-get install --install-recommends

vagrant@trusty:~$ sudo apt-get install build-essential libssl-dev

Upgrade Trusty to the LTS Enablement Stack:


// upgrade for better hardware support
// assumes using packages that are Multi-Architecture (both 32 and 64 bit)
vagrant@trusty:~$ sudo apt-get install --install-recommends linux-generic-lts-vivid xserver-xorg-core-lts-vivid xserver-xorg-lts-vivid xserver-xorg-video-all-lts-vivid xserver-xorg-input-all-lts-vivid libwayland-egl1-mesa-lts-vivid libgl1-mesa-glx-lts-vivid libgl1-mesa-glx-lts-vivid:i386 libglapi-mesa-lts-vivid:i386
 

At this point, if everything went well, you should be on the 3.19.0-43-vivid kernel.

Let’s check…

Check the kernel:

vagrant@trusty:~$ uname -r

The output should be…

Output:

vagrant@trusty:~$ 3.19.0-43-generic

Next, let’s install our LAMP stack.

Install the LAMP stack


vagrant@trusty:~$ sudo apt-get install apache2

vagrant@trusty:~$ sudo apt-get install mysql-server php5-mysql

// install php and any other php modules you need
vagrant@trusty:~$ sudo apt-get install php5-common php5-dev php5-cgi php5-cli php5-curl php5-xdebug php5-dbg php5-gd php5-mcrypt php5-oauth php5-memcache php5-redis php5-mongo libapache2-mod-php5


Next let’s make sure our NFS client is installed since we want Vagrant to spin up our VM using NFS (just like in Virtual Box) for greater performance.

Install NFS Client

vagrant@trusty:~$ sudo apt-get install nfs-common

Next, make sure you install any other packages you need in your stack.

Maybe you need Node? Mongo? Full MEAN stack? Redis? Java? Python? Ruby? Puppet/Chef?

Install any language, RDBMS, NoSQL DB, framework, or any platform that you need for you tech stack. Since our base box will be our template VM, we don’t want to have to install anything after it is created.

Lastly let’s install wrk so we can benchmark our HTTP requests. But first, let’s make sure git is installed and if it isn’t, we want to install it.

Install Git

vagrant@trusty:~$ sudo apt-get install git

Now let’s install wrk.

Install Wrk


vagrant@trusty:~$ cd /opt

// clone the github repo
vagrant@trusty:/opt$ git clone https://github/wg/wrk.git

vagrant@trusty:/opt/wrk$ cd ./wrk

vagrant@trusty:/opt/wrk$ make

// lastly add the exe to our env var PATH
vagrant@trusty:~$ cp wrk /usr/local/bin


Now we want to start configuring everything…

Let’s first configure our VM according to the Vagrant Base Box standard.

Since we’ve already created a user with a user/password combo as vagrant/vagrant and root has the password vagrant, the first step is done.


Next, we don’t want to be asked for a password when using sudo.

Let’s fix that.

Password-less Sudo


vagrant@trusty:~$ visudo

// append to the end and comment out any line containing "requiretty"
vagrant ALL=(ALL) NOPASSWD: ALL


Now we want SSH to be fast, so let’s make sure “UseDNS” is set to “no”

Check UseDNS


vagrant@trusty:~$ cd /etc/ssh

vagrant@trusty:/etc/ssh$ sudo vim ssh_config

// append "UseDNS no" to the file if its not already contained
UseDNS no


Next up, let’s create the insecure public key the Vagrant will look for so Vagrant can log us in with our key instead.

Set our insecure public key for the user Vagrant


// make sure we're in home
vagrant@trusty:~$ cd ~

vagrant@trusty:~$ ls -la

// create .ssh directory if .ssh does not exist in /home/vagrant
vagrant@trusty:~$ mkdir .ssh

// change .ssh to permission 0700
vagrant@trusty:~$ sudo chmod -rf 0700 .ssh

// create the insecure vagrant public key
vagrant@trusty:~$ cd .ssh

vagrant@trusty:~/.ssh$ vim authorized_keys

// paste into our authorized_keys file
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key


Now we need to make sure our authorized_keys file has the correct permissions.

Set Permissions


vagrant@trusty:~/.ssh$ sudo chmod 0600 ./authorized_keys


Next, up let’s enable and configure Xdebug and while we’re at it, let’s also enable mod_rewrite.

Enable Rewrite Module


// enable mod_rewrite
vagrant@trusty:~$ sudo a2enmod rewrite


Now let’s configure Xdebug.

Configure XDebug Module


vagrant@trusty:~$ cd /etc/php5/apache2/conf.d


**Note: If the symlink for xdebug.ini isn’t in conf.d, symlink the config file xdebug.ini from /etc/php5/mods-available/20-xdebug.ini to /etc/php5/apache2/conf.d/20-xdebug.ini



// configure xdebug
vagrant@trusty:/etc/php5/apache2/conf.d$ sudo vim 20-xdebug.ini

// add to xdebug.ini
[XDebug]
zend_extension=location_to_whatever_version_you_are_using.so
xdebug.remote_enable=1
xdebug.remote_host=192.168.33.1
xdebug.remote_port=9009
xdebug.remote_log="/var/log/debug/xdebug.log"


Next up, let’s configure a Vhost to be used as a template. Our Vhost will point to our NFS shared directory between our host and guest OS.

Create Vhost template


vagrant@trusty:~$ cd /etc/apache2/sites-available

vagrant@trusty:/etc/apache2/sites-available$ sudo vim vagrant-dev.conf

 
// Add these contents
< VirtualHost *:80>
    ServerName vagrant.dev
    DocumentRoot /var/www/html
    ErrorLog ${APACHE_LOG_DIR}/vagrant_error.log
    CustomLog ${APACHE_LOG_DIR}/vagrant_access.log common

    # uncomment if using php-fpm
    #ProxyPassMatch ^/(.+\.(php)(/.*)?)$ fcgi://127.0.0.1:9000/var/www/html/$1

    # don't use these settings in prod
    < Directory /var/www/html>
        AllowOverride all
        Options MultiViews Indexes FollowSymlinks
        Require all granted

        # not necessary if elsewhere or in .htaccess
        < IfModule mod_rewrite.c>
            RewriteEngine On
        < /IfModule>
    < /Directory>
< /VirtualHost>

Now we want to enable our vagrant-dev.conf vhost.

vagrant@trusty:~$ sudo a2ensite vagrant-dev.conf

Now let’s symlink our “html” directory to our soon to be Vagrant’s default shared directory.


**Note: You can point the vhost’s document root directly to the shared directory also.


Symlink to Shared Directory


vagrant@trusty:/etc/apache2/sites-available$ cd /var/www

vagrant@trusty:/var/www$ sudo ln -s /vagrant_data html


Next, since we don’t want to run a DNS server for a development environment, let’s manually point to this vhost.

Edit /etc/hosts


vagrant@trusty:/var/www$ cd /etc
vagrant@trusty:/etc$sudo vim hosts

// Add to your hosts file
127.0.0.1 vagrant.dev


That should be it for our VM configuration!

Now shutdown the VM, and let’s turn it into a base box with VMware Workstation as the provider.


Now we’re back on our host OS.

Navigate on the host OS, to the directory where your Virtual Machine is stored.

Navigate to Base box directory


dlaroche@laptop-dev:~$ cd baseboxes/Ubuntu-trusty

There should be files with extensions .nvram, .vmdk, .vmsd, .vmx, .vmxf, and some log files.

We need to first create a metadata.json file that Vagrant will read when creating the base box. This file can be detailed, containing provider and versioning info, but we are only going to use the bare minimum for this tutorial. You can learn more on the Vagrant site for more info on base box meta data, since this probably doesn’t meet your organization’s needs.

Create Vagrant metadata.json


dlaroche@laptop-dev:~/baseboxes/Ubuntu-trusty$sudo vim metadata.json

// Add to the file
{
  "provider": "vmware_workstation"
}


Now lets finally create the base box! 😀

We want to tar all of the files in the VM directory into a .box archive.

Create VMware Base Box


dlaroche@laptop-dev:~/baseboxes/Ubuntu-trusty$ tar cvzf ubuntu-trusty.box ./*


Now that we have our VM as a base box, let’s now add it to Vagrant.

Add Base Box to Vagrant


dlaroche@laptop-dev:~/baseboxes/Ubuntu-trusty$vagrant box add our-organization/ubuntu-trusty ubuntu-trusty.box


Now let’s take a look at our list of base boxes.
List Base Boxes added to Vagrant


dlaroche@laptop-dev:~/baseboxes/Ubuntu-trusty$vagrant box list


Now let’s configure our Vagrant init file, by going to the root directory of our code.

Create Vagrantfile


dlaroche@laptop-dev:~$cd workspace/test

dlaroche@laptop-dev:~/workspace/test$vagrant init 


Now add to your code’s Vagrantfile inside “Vagrant.configure” the following.

Add to Vagrantfile


config.vm.box = "larochetech/ubuntu-14.04.3_64"
config.ssh.insert_key = false
config.vm.provider "vmware_workstation"

config.vm.network "forwarded_port", guest: 80, host: 8080

// we configured xdebug to communicate to this network
config.vm.network "private_network", ip: "192.168.33.10"

// use nfs for greater performance
config.vm.synced_folder ".", "/vagrant_data", nfs: true

// Give the vm all of your cores and half of your ram.
// Setting the gui to true allows us to see the VM running
// in VMware. If you don't care about that, you can leave 
// it out.
config.vm.provider "vmware_workstation" do |v|
    v.gui = true
    v.memory = 8192
    v.cpus = 4
end


Now let’s edit our host OS hosts file to point to Vagrant.

Edit /etc/hosts


dlaroche@laptop-dev:~/workspace/test$ sudo vim /etc/hosts

// Add to your hosts file the private network of your VM
192.168.33.10 vagrant.dev


Now let’s spin up our VM using vagrant! 😀

Start VM through Vagrant


dlaroche@laptop-dev:~/workspace/test$ vagrant up

Since this is the first time the VM is spun up, Vagrant should provision it according to the VagrantFile settings, and if everything goes according to plan, you should have the VM running in VMware! 😀


Now that the VM is running under Vagrant… Finally… We’re ready to configure our IDE! 😀


Launch JetBrains IntelliJ Idea and install the PHP and Vagrant Plugins. You can search for them under “Settings -> Plugins”.

After they’re installed, restart the IDE and go to “Settings -> Languages & Frameworks -> PHP -> Servers” and create a new server.

For the “host” add “vagrant.dev”

Next, click “Use path mappings” and for “File/Directory” set it to the root directory for your code base (on your host OS).

For “Absolute path on the server”, set it to the shared Vagrant directory “/vagrant_data”.

Now click “Apply”.


Now go to “Settings -> Languages & Frameworks -> PHP” and click the “…” and create a new “Remote Interpreter”.

In the Vagrant plugin tab, enter the SSH credentials for the user “vagrant” we created in our VM, and under “PHP executable”, click the “Reload/Refresh” button.

This should search for the remote PHP interpreter running inside the VM and load it. After it is found click “Apply” to save and exit.


Next, we need to configure XDebug in the IDE. Go to “Settings -> Languages & Frameworks -> Debug” and make sure “Can accept external connections” is checked, and change the “Debug port” to “9009” (which is what we set Xdebug to inside the VM).

Click “Apply” to save.


That should be it! 😀

Now go to vagrant.dev

If everything is correct, you should be running your code through the VM’s PHP interpreter and XDebug should be communicating back to your IDE from the VM on port 9009. 😀


Now all thats left to do is to test our NFS performance in the VM with http requests using wrk.



Benchmark NFS Performance


dlaroche@laptop-dev:~/workspace/test$ vagrant ssh

// run a 60 second test
vagrant@trusty:~$ wrk -d60s http://vagrant.dev


Output – Vagrant NFS with VMware Provider:


Running 1m test @ http://vagrant.dev
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
  Latency    11.31ms   17.66ms 569.96ms   91.76%
  Req/Sec    650.24    245.14     1.01k    75.72%
  
77209 requests in 1.00m, 0.97GB read
Requests/sec:  1286.56
Transfer/sec:  16.49MB


Output – Native Ubuntu:


Running 1m test @ http://test.dev
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
  Latency    11.20ms   21.44ms 426.39ms   92.76%
  Req/Sec   786.50    374.96    2.00k    60.73%

93149 requests in 1.00m, 1.44GB read
Requests/sec:   1550.07
Transfer/sec:   24.60MB


And that’s it! Everything should be working..

If you run into any problems, either comment below, or ping me at info@daryllaroche.com

Happy coding! 😀

Functional programming in PHP using Map Reduce

This tutorial shows you how to leverage some of PHP’s functional programming abilities.

This is just a simple solution to Problem 22 from ProjectEuler.net, but it’s a good example of showing how to use a hybrid of OOP and Functional Programming to arrive at the correct solution.

We’ll use basic OOP for the structure and for encapsulation, and then we’ll implement the actual solution using the infamous Map Reduce pattern.

Here’s a look at the full implementation, then we’ll go over each piece individually.

ProjectEuler.net Problem 22

namespace ProjectEuler\Problems;

class Problem22
{
 private $inputData = array();

 function __construct()
 {
  $filename = "./p022_names.txt";
  $this->readInCsv($filename);

  // Would never do this. Only invoking from the
  // constructor for brevity since not a real app.
  //
  // Invoking methods on this class should
  // come from your controllers.
  // And a class with this type of logic would be classified
  // as domain logic, aka *business logic*.
  //
  // This "business logic" should be structured as
  // Domain Objects (or a service depending 
  // on reusability) in your application as
  // part of your Model *layer*.

  $result = $this->mapReduce();
  $this->displayResult($result);
 }

 private function readInCsv($filename)
 {
  $handle = fopen($filename, "r");
  $this->inputData = fgetcsv($handle, 0, ",");
  fclose($handle);

  sort($this->inputData);
 }

 private function mapReduce()
 {
  $nameTotal = array_map(function($name, $index) {

   $charsTotal = array_reduce(str_split($name),
    function($runningCharTotal, $char) {

     // calculate value by offsetting from capital 
     // ascii char values to get sequential values
     $runningCharTotal += ord(strtoupper($char)) - 64;

     return $runningCharTotal;
    });

   // return the name position
   // multiplied by the name's char total
   return ($index + 1) * $charsTotal;

  }, $this->inputData, array_keys($this->inputData));

  $total = array_reduce($nameTotal, 
   function($runningNameTotal, $item) {

    $runningNameTotal += $item;

    return $runningNameTotal;
   });

  return $total;
 }

 private function displayResult($result)
 {
  echo 'The total is: ';
  print_r($result);
 }
}




First we’ll take a look at how to read in data from our .txt file.

The readInCsv method uses fopen to open the file, bind it to a data stream, and then fgetcsv parses each line (using commas as the delimiter) into an array.

This array is then sorted alphabetically using one of the native PHP sorting algorithms (I believe it’s an implementation of QuickSort for those wondering)

 private function readInCsv($filename)
 {
  $handle = fopen($filename, "r");
  $this->inputData = fgetcsv($handle, 0, ",");
  fclose($handle);

  sort($this->inputData);
 }




Next, after we have our input file data read into an array, we do some functional programming magic using array_map, array_reduce, and anonymous functions (lambdas).



Let’s look at array_map

In functional programming a map function takes an array and a *callback function* as input (and other *optional* arguments, but we’ll leave that out for brevity) and then performs a transformation (functionality/algorithm) specified in the callback function on every element in the array. The new values calculated for each element are added to a new array.



Now, for our heavy hitter… array_reduce

In functional programming a reduce function starts with an initial value and passes this initial value to the callback function. The initial value plus the first element in the array is passed to the callback function and the callback function’s functionality is performed.

After this callback functionality completes, the data returned is now passed to the callback function again. The functionality is performed again with the data return from the previous call, and the *second* element in the array. The data accumulates with each call.

This continues through each element in the array until the last element completes. When done a single value/object/etc (meaning the array was reduced to its last element) is returned.



Confused?

Simple pseudo code example:

 array intExample = [1, 2, 3, 4]

 result = reduce(intExample, "callback")

 function callback(previous, current)
 {
  previous *= current
  return previous
 }


The output of the above looks like:
2 = (1 * 2)
6 = (2 * 3)
24 = (6 * 4)

So the array [1, 2, 3, 4] was reduced to 24.

**NOTE: If no initial value is given, NULL is passed, and the reduce starts with the array’s first element as the initial value (like above).



Using both map and reduce together:

 $nameTotal = array_map(function($name, $index) {

   $charsTotal = array_reduce(str_split($name),
    function($runningCharTotal, $char) {

     // calculate value by offsetting from capital 
     // ascii char values to get sequential values
     $runningCharTotal += ord(strtoupper($char)) - 64;

     return $runningCharTotal;
    });

  // calculate a name total, which is
  // (name position * name's char total)
  return ($index + 1) * $charsTotal;

 }, $this->inputData, array_keys($this->inputData));



**NOTE: The callback function does not have to be defined inline, nor does it have to be an anonymous function. We could have just as easily defined the function:

Callback Function

function callBackAction($name, $index)
{
  // Do stuff...
}


And then passed the callback like:

$names = array("Name", "AnotherName");

array_map("callBackAction", $names);


However, it’s much easier to read (and cleaner) using an anonymous function instead.



**NOTE: Our parameters used in our callback functions are implicitly passed as arguments from the previously returned result (previous) from the callback.


Now… Why str_split?

Since array_reduce only takes an array as the datatype, we create one on the fly using PHP’s str_split.

str_split is just a quick (and dirty?) way of taking a string and storing each character as an element in an array, which is exactly what we want to do, in order to reduce on it.



Now… I know what you’re probably thinking..

WTF does the following mean? 🙂

$runningCharTotal += ord(strtoupper($char)) - 64;


Well….
Since we want the sequential value of each character in a name, we need a quick and non super hacky way of doing this.

To do this, we just use ord() to get the ASCII value of a character. However, before passing the value to ord() we first want to convert (or make sure) our character is uppercase using strtoupper(), and then offset it by 64.

Offsetting by 64 gives us the sequential value for that character.

For example, suppose we have the lowercase character ‘a’.

First we convert ‘a’ to the capital character ‘A’ using strtoupper(), and then pass ‘A’ to ord().

ord() will return ‘A’ in ASCII (under base 10), which equals 65.

Then if we offset this by 64, the character ‘A’ will give us decimal 1, and ‘B’ will give us decimal 2, and ‘C’ decimal 3, and so on…



Now let’s look at our return.
The return is straight forward.

We need an easy way to calculate our row value in our rowValue * nameValue = totalNameValue calculation. In order to do this, we pass our array to PHP’s array_keys(), which will give us access to each indice in our array during the reduce.



**NOTE: Just a heads up if you didn’t notice. PHP has yet another annoying quirk with it’s naming consistency of the parameters for the array_map and array_reduce signatures.

With array_map the callback is the first parameter, with the second the array, where as in array_reduce the array is supplied first, and then the callback second…. grrrrr….

>> I know.. I know..
>> *insert non PHP developers mocking tone of voice here*
>> Somethin.. Somethin.. PHP is the *worst* language.. Somethin.. Somethin..

>> *now insert PHP developers mocking tone of voice here*
>> Somethin.. Somethin.. Don’t care about the language.. I just use the right tool for the job.. Somethin.. Somethin..
😀



Calculating the Overall Total

  $total = array_reduce($nameTotal, 
   function($runningNameTotal, $item) {

    $runningNameTotal += $item;

    return $runningNameTotal;
   });

  return $total;
 }


Our final reduce is straight forward as well. The accumulator and each element perform a summation and the total returned is the solution.

Output the solution and that’s it! 😀