The customer engaged with Cloudreach to build a Minimum Viable Model (MVM) to begin leveraging Machine Learning for their Event & Threat Detection Platform. This would strengthen their ability to automate their response. The first challenge was to begin training the Machine Learning model to correctly identify anomalous events.
Constructing a new, cloud-native “Security Event and Information Management (SIEM) data pipeline that could log, aggregate and aggregate event streams from over 40,000 devices. Over time, the goal is for the system to leverage AI capabilities to detect and respond to cybersecurity threats in real-time for their city’s systems. There was also a need for a data lake for forensic and proactive analysis, monitoring of system networks in real time, and cold storage of data retention and audibility.
We created a managed services data pipeline and implemented an Infrastructure-as-Code for repeatable and reliable infrastructure deployments across platforms. We created the CI/CD pipeline for data workflow, and designed event architecture and data modeling for ingesting heterogenous client data.
It is also possible for the customer’s team to create alerts, visualizations and enable analytics using Google Cloud Platform’s native services.
The environment is also highly resilient and available, as it is built implementing the BeyondCorp architecture, implementing the principle of “zero trust networking” to ensure the utmost security standards are achieved.
The key technologies used are BigQuery, BigTable, Dataflow, Cloud ML (Planned), and Pub/Sub.
In addition, over a series of months, we simulated a cyber-event that the city is accustomed to experiencing. From here, we analysed and parsed the logs, identifying anomalous events and leveraged a neural network to begin training the ML Model.
The key GCP technologies used are TensorFlow.
This solution allows the city’s Cyber Command to keep their city safe from cyber security threats, as well as monitoring events in real time using BigQuery. The platform aggregates data from multiple environments, which provides a unique view into cyberattacks, to actuate on intelligence and audit incidents.
The average processing time is less than 10 milliseconds per event and over a billion events are processed per day.
The data lake allowed third-party companies and internal tools to ingest data from blob storage and data warehouses. The data pipeline enabled the internal team to build machine learning capabilities by providing data at scale. Through the flexible and modular architecture, the platform has therefore been created to take advantage of GCP’s functionality for machine learning and automation.
Finally, the customer now has the initial foundations of a Minimum Viable Model leveraging Machine Learning. As the Model continues to evolve, this will enable the city to automatically react to cyber-security threats.
US City Cyber Command
The city is one of the largest in the world, with over 8.6 million residents. They launched a Cyber Command with the mandate of defending and safeguarding the city, and its systems and data, from cyberattacks and other online threats. They recognized that achieving cyber security required an innovative and forward-thinking approach that could achieve security at scale.