Service development for Threat Manager

a corporate network security system

CLIENT

3 000
organizations use company solutions.
0
billion threats are prevented annually.

Alert Logic develops, supplies, and supports hardware-software complexes for the control over enterprise network security. The company headquarters are located in Houston (the state of Texas), USA.

CHALLENGE

Alert Logic and Reksoft have been cooperating since 2007. Previously, the services for Threat Manager and Log Manager – the corporate network security systems have been developed. They analysed the incoming traffic and user host logs respectively. Based on it, security threats were detected.

Over time, Threat Manager has no longer been able to satisfy growing customer needs because it was not designed for long-time processing and analysis of big data. Therefore, in 2012, the decision on performance tuning and upgrading the system has been made.

Alert Logic has formulated the following requirements to the upgraded product:

А 24/7 real-time handling of thousands of data packets

from corporate networks of the customers occurring every minute.

Finding dependencies (correlations)

between the incoming traffic and the occurrence of vulnerabilities.

The possibility to obtain detailed statistics on the incoming traffic

for building new attack progression models and analysing the existing ones.

SOLUTION

The new service was named Next Generation Expert System (NGX). The Erlang programming language has been used as the main development tool. It allows creating distributed systems for processing big data and ensures high availability.

The development of the Next Generation Expert System core by the specialists of Reksoft and Alert Logic has started in 2012. The developers had to:

Create a reliable packet handling and storage system that would enable to set up a parallel data processing at a minimal cost.

The Riak distributed NoSQL database has been selected as its core. It ensured high availability and reliable storing of structured data.

Set up a unified platform for the development of lightweight agent applications.

The applications had to collect the network traffic data, convert them to an analysable format, and ensure their reliable transfer to the server.

Build an infrastructure for packet handling.

Due to infrastructure limitations, the speed of data processing was not high enough. To solve this problem, the developers have created a packet handling infrastructure. It allowed scaling the solutions to customers’ needs.

The new service development did not entail any significant changes to the server infrastructure which had to ensure the uninterrupted running of other services. To achieve this, the Reksoft specialists have carried out work on the performance optimization at different levels, including the redesign of some components. As a result, the company has obtained the order for further support and optimization of the entire backend platform.

NGX allowed automating customer infrastructure monitoring services. Thanks to the improvements made by Reksoft, the number of mistakes during the threat risks assessment was considerably reduced, the security analysts at Alert Logic were enabled to make decisions faster and prevent attacks based on the existing analytical data. The performance tuning of the service did not require the involvement of IT resources.

By 2014, the system load has grown more than three times and reached 9 thousand packets of incoming data a minute. The Reksoft experts have additionally optimized the performance and speed of data processing. It allowed maintaining the performance with a 20% buffer comparing to the production load without spending substantial sums on the hardware.

The Reksoft specialists have accomplished the following tasks:

the NGX service software core was developed;
the solutions for real-time analytics and data processing were designed;
a flexible interface for setting filers was introduced;
the system was upgraded and configured without being reloaded and the subsequent involvement of programmers was not needed;
the system performance was optimized.

RESULTS

The upgraded Threat Manager operates 24/7 and processes up to 250 GB of incoming data per client connection. The current customer base is continually expanding and accounts now for about 4,000 subscribers. The volume of information stored on the server reaches a petabyte. The handling time of each incoming data packet to detect a network attack is under 15 minutes.

Reksoft has obtained the following results:

a new service available online 24/7 for processing the customer data was created;
a threefold increase versus 2012 in the performance of IT security services of the customer was achieved;
the risk of data loss when using the service was minimized.

ТЕХНОЛОГИИ

  • Linux,
  • C/C++,
  • Erlang,
  • Riak,
  • MySQL.

УСЛУГИ

  • the software development and support,
  • the setup of a Training Centre for developers programming in the Erlang language and a Centre for Advanced Training for developers to build/supplement the team.
Menu