At this week’s AWS Re:Invent conference, Amazon Web Services (AWS) announced Security Lake to help users act on security data more quickly.

The service automatically centralises an organisation’s security data from cloud and on-premises sources into a purpose-built data lake in a user’s AWS account.

Security Lake manages data throughout its lifecycle with customisable data retention settings, converts incoming security data to the Apache Parquet format, and conforms them to the Open Cyber-security Schema Framework (OCSF) open standard to make it easier to normalise security data automatically from AWS and combine them with dozens of pre-integrated third-party enterprise security data sources.

Security analysts and engineers can use Security Lake to aggregate, manage and optimise large volumes of disparate log and event data to enable faster threat detection, investigation and incident response to address potential issues quickly, while continuing to use their preferred analytics tools.

Those who want greater visibility into security activity across their entire organisations can proactively identify potential threats and vulnerabilities, assess security alerts, respond accordingly, and help prevent future security events.

To do this, most organisations rely on log and event data from different sources such as applications, firewalls and identity systems running in the cloud and on premises, each using a unique and often incompatible data format. To uncover security-related insights, such as spotting unauthorised external data transfers for sensitive information or identifying the installation of malware across employee devices, organisations must first aggregate and normalise all these data into a consistent format.

Once the data are formatted consistently, users can analyse them and understand the current level of vulnerability, and then correlate and monitor threats for improved observability. They typically use different security options to address specific use cases, such as incident response and security analytics, which often means they duplicate and process the same data multiple times because each option has its own data stores and format.

This is time consuming and costly, slowing down security teams’ ability to detect and respond to issues.

As users, tools and data sources are added, security teams must also spend time managing a complex set of data-access rules and security policies to track how data are used and ensure people can get the information they need. Some security teams create a central repository for all their security data in a data lake, but these systems require specialised skills and can take months to build due to the large amount of log data from different sources, which can run into petabyte scale.

Security Lake is a purpose-built security data lake that can be created in just a few clicks and lets users aggregate, normalise and store data so they can respond to security events faster using their preferred tools. After setup and connections to selected data sources, Security Lake automatically builds a security data lake in a user-selected region, which can help meet regional data compliance requirements.

After they choose their data sources, Security Lake automatically aggregates and normalises data from AWS, combines them with third-party sources that support OCSF and optimises them into a format that is easy to store and query. Security Lake automatically orchestrates the end-to-end process from data lake creation and data aggregation to normalisation and integration.

The service builds the security data lake using Amazon Simple Storage Service (S3) and AWS Lake Formation to set up security data lake infrastructure automatically in a user’s AWS account, providing full control and ownership over security data. Once ingested and normalised, users can use their preferred security and analytics tools, including Amazon Athena, OpenSearch, and SageMaker, along with third-party offerings such as IBM, Splunk or Sumo Logic to make it faster and easier to capture broader and deeper analytics from AWS and more than 50 third-party (such as Cisco, CrowdStrike and Palo Alto Networks) and user data sources.

As a result, Security Lake helps users improve their overall security posture, provide greater visibility for security teams to identify and understand events, and reduce the time to resolve security issues.

“Customers must be able to quickly detect and respond to security risks so they can take swift action to secure data and networks, but the data they need for analysis are often spread across multiple sources and stored in a variety of formats,” said Jon Ramsey, vice president at AWS. “Customers tell us they want to take action on these data faster to improve their security posture, but the process of collecting, normalising, storing and managing these data is complex and time consuming. Security Lake lets customers of all sizes securely set up a security data lake with just a few clicks to aggregate logs and event data from dozens of sources, normalise them to conform with the OCSF standard, and make them more broadly usable so customers can take action quickly using their security tools of choice. With Security Lake, customers get superior visibility and control, with help from the largest ecosystem of security partners.”