HTTP get request with C++ Boost::Asio

So the other day I found myself needing to make several networking calls in a C++ application I was developing in my spare time.

As you can expect there is not much networking wise built into C++ out of the box. As I was developing On Windows I could have gone down the tedious winsock route and practically reinvented the wheel however I decided my time was better spend using a pre-built framework to achieve what I wanted.

I settled on using the Asio library which is a part of the Boost framework, essentially an addon to the standard library which can be downloaded at http://www.boost.org. From my experience, while Boost is generally well documented, it is designed to be a relatively low level framework and as such I ended up implementing a basic HTTP Get helper class. 

What follows is how I achieved a basic HTTP get request using C++/Boost.

The fundamental thing to remember with TCP is that while data is guaranteed to arrive correctly due to CRC checks and re-transmission, we do not know when the data will arrive. Thus it is important to perform our request asynchronously in order to not block the rest of our program from executing while the data is transmitted. Therefore I will demonstrate how to achieve this asynchronous behaviour using boost.

First off lets look at what is involved with getting a socket connected to an ip address.

First we need to include the relevant header files and namespaces.

#include <boost/asio.hpp>
using boost::asio::ip::tcp;

Before we can connect a socket we need to setup a Boost IO Service within our application. This runs an event loop which processes messages posted to and from various Boost components. Many of the examples you will find on the web run this on the Main Thread for the sake of brevity, as will mine. However for more complex programs you may wish to develop I would strongly advise moving this to a background thread so as to not block on your applications main thread.

Running a basic io_service is done as follows:

boost::asio::io_service io_service;

// queue some work to be done here

// io service will exit when queued jobs are completed
io_service.run();

Now that we have the basic setup we can begin the steps required to obtain the data from the web server. These go as follows:

1. Connect to a socket on the web server
2.  Send a HTTP Request header
3.  Read the header response (up to you what to do here, in this simple example I throw it away)
4. Read the HTML content

For performance reasons we use the async methods in the Asio library. These allow us to call a function such as connect and have the library call us back when that operation has completed. This is the preferred means of doing things predominantly for performance. Reading the response back in this async manner allows us to read and process chunks of data. This means we can start processing the start of the document while the rest is still on the wire being transferred and is a trick browsers use to improve performance.

So once we combine all those steps we end up with something like this:

// m_host should be in format such as "www.google.co.nz"
// m_url should be in format such as "/index.html"
void HTTPGetRequest::sendRequest()
{
 tcp::resolver::query query(m_host, "http");

 m_resolver.async_resolve(query, 
  [this](const boost::system::error_code& ec, tcp::resolver::iterator endpoint_iterator)
  {
   boost::asio::async_connect(m_socket, endpoint_iterator,
    [this](boost::system::error_code ec, tcp::resolver::iterator)
   {
    if (!ec)
    {
     std::ostream request_stream(&m_request);
     request_stream << "GET " << m_relativeURL << " HTTP/1.1\r\n";
     request_stream << "Host: " << m_host << "\r\n";
     request_stream << "Accept: */*\r\n";
     request_stream << "Connection: close\r\n\r\n";

     boost::asio::async_write(m_socket, m_request,
      [this](boost::system::error_code ec, std::size_t /*length*/)
     {
      boost::asio::async_read_until(m_socket, m_response, "\r\n\r\n",
       [this](boost::system::error_code ec, std::size_t length)
       {
        ReadData();
       });
     });
    }
   });
  });
}

void HTTPGetRequest::ReadData()
{
 boost::asio::async_read(m_socket, m_response, boost::asio::transfer_at_least(1),
  [this](boost::system::error_code ec, std::size_t /*length*/)
 {
  size_t size = m_response.size();
  if (size > 0)
  {
   std::unique_ptr buf(new char[size]);
   m_response.sgetn(buf.get(), size);

   m_receivedCB(buf.get(), size);
  }

  if (ec != boost::asio::error::eof)
  {
   ReadData();
   return;
  }

  m_socket.close();

  m_completeCB();
 });
}

Hopefully it is clear that chaining the async callbacks together carries out the steps listed earlier. Now that we have the bare bones of our network call we can add some meat and form it into a basic class. Note that while this class should be functional it is by no means designed to be totally complete. There are still error cases to consider for example.

Hopefully the following will provide you with a working sample to base your project on.

HTTPGetRequest.h

#pragma once

#include <string>

#include <boost/asio.hpp>
using boost::asio::ip::tcp;

typedef void(*HTTPRequestDataReceived)(char*, size_t);
typedef void(*HTTPRequestComplete)();

class HTTPGetRequest
{
public:
 HTTPGetRequest(
  boost::asio::io_service& io_service, 
  std::string host, 
  std::string relativeURL, 
  HTTPRequestDataReceived receivedCB,
  HTTPRequestComplete completeCB);
 
 ~HTTPGetRequest();

public:
 void sendRequest();

private:
 HTTPRequestDataReceived m_receivedCB;
 HTTPRequestComplete m_completeCB;

 std::string m_host;
 std::string m_relativeURL;

 tcp::socket m_socket;
 boost::asio::io_service &m_io_service;
 tcp::resolver m_resolver;

 boost::asio::streambuf m_request;
 boost::asio::streambuf m_response;

 void ReadData();
};

HTTPGetRequest.cpp

#include "stdafx.h"

#include "HTTPGetRequest.h"

HTTPGetRequest::HTTPGetRequest(boost::asio::io_service& io_service, std::string host, std::string clipURL, HTTPRequestDataReceived receivedCB, HTTPRequestComplete completeCB) :
 m_host(host),
 m_relativeURL(clipURL),
 m_io_service(io_service),
 m_socket(io_service),
 m_resolver(m_io_service),
 m_receivedCB(receivedCB),
 m_completeCB(completeCB)
{

}

HTTPGetRequest::~HTTPGetRequest()
{

}

// host should be in format such as "www.google.co.nz"
// url should be in format such as "/index.html"
void HTTPGetRequest::sendRequest()
{
 tcp::resolver::query query(m_host, "http");

 m_resolver.async_resolve(query, 
  [this](const boost::system::error_code& ec, tcp::resolver::iterator endpoint_iterator)
  {
   boost::asio::async_connect(m_socket, endpoint_iterator,
    [this](boost::system::error_code ec, tcp::resolver::iterator)
   {
    if (!ec)
    {
     std::ostream request_stream(&m_request);
     request_stream << "GET " << m_relativeURL << " HTTP/1.1\r\n";
     request_stream << "Host: " << m_host << "\r\n";
     request_stream << "Accept: */*\r\n";
     request_stream << "Connection: close\r\n\r\n";

     boost::asio::async_write(m_socket, m_request,
      [this](boost::system::error_code ec, std::size_t /*length*/)
     {
      boost::asio::async_read_until(m_socket, m_response, "\r\n\r\n",
       [this](boost::system::error_code ec, std::size_t length)
       {
        ReadData();
       });
     });
    }
   });
  });
}

void HTTPGetRequest::ReadData()
{
 boost::asio::async_read(m_socket, m_response, boost::asio::transfer_at_least(1),
  [this](boost::system::error_code ec, std::size_t /*length*/)
 {
  size_t size = m_response.size();
  if (size > 0)
  {
   std::unique_ptr<char> buf(new char[size]);
   m_response.sgetn(buf.get(), size);

   m_receivedCB(buf.get(), size);
  }

  if (ec != boost::asio::error::eof)
  {
   ReadData();
   return;
  }

  m_socket.close();

  m_completeCB();
 });
}

And finally Main.cpp to show it all in use...

// ConsoleApplication1.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"

#include <vector>

#include <boost/asio.hpp>
#include <boost/bind.hpp>

using boost::asio::ip::tcp;

#include "HTTPGetRequest.h"

std::vector<char> g_data;

void OnDataReceived(char* data, size_t dataLen)
{
 // store data in vector for sake of demo...

 unsigned int oldSize = g_data.size();
 g_data.resize(oldSize + dataLen);
 memcpy(&g_data[oldSize], data, dataLen);
}

void OnRequestCompleted()
{
 // print contents of data we received back...

 g_data.push_back('\0');
 printf(&g_data[0]);
}

int _tmain(int argc, _TCHAR* argv[])
{
 boost::asio::io_service io_service;

 HTTPGetRequest req(
  io_service,
  "www.google.co.nz",
  "/index.html",
  OnDataReceived,
  OnRequestCompleted);

 req.sendRequest();

 io_service.run();

 // so we dont exit too early...
 std::string line;
 std::getline(std::cin, line);

 return 0;
}

After compilation that simple example sends an asynchronous GET request to google New Zealand and prints the output to the console window once it arrives. As I mentioned earlier for more complex situations, you may need to move the io_service to a background thread to avoid the main thread blocking.

I hope you find the following useful, as I mentioned earlier please bear in mind it does not handle every situation but should provide you with a basic starting point.

Feel free to comment with any questions or feedback you may have! Thanks for reading.

Read More