It’s a common question: Can I use cloud logging services for PCI DSS? And if so, which cloud providers should I use?
The short answer: since you can use cloud providers like AWS and Azure to host your core cardholder data environment, the answer is ultimately yes. You can oursource log management to cloud providers. Information about some of the more popular cloud logging services is below.
A better question is probably should I use cloud log management for PCI compliance?
Contents
Pros and Cons of Cloud-based Logging
As with many areas of PCI compliance, how exactly you meet a specific requirement involves an element of risk management. These are risk-based decisions both for you, as the company deciding how to meet the requirement, and for your QSA, as they decide whether they’ll accept your solution. This is particularly true for the logging solution you use to meet PCI DSS Requirement 10.
Is log management part of the PCI Cardholder Data Environment (CDE)?
The fact that this is even an open question shows how complicated things can quickly get. In my view, logging is always a part of the environment because it’s needed to meet various PCI DSS requirements.
I’ve met some people who argue that the logging system is not part of the PCI CDE because it doesn’t, and isn’t, allowed to store PCI Cardholder Data. That is certainly true, but it’s still a necessary part of your PCI compliance operation.
In the event of a breach, the stored logs are one of the things the forensic investigators will rely on to trace back and determine what happened. In a sense, they act as the “black box” flight recorder for your environment. You need to have confidence that these logs are retained for the appropriate time, and resistant to tampering.
Automated scanning, correlation, and alerting on those same logs is also likely to be how you meet the requirement to review logs on a daily basis. If that process breaks down, one of your key lines of defence goes with it.
Some cloud logging providers actually undergo QSA audits as Level 1 Service Providers. Because of this, they can provide their PCI compliant clients and QSA with an AoC, which shows their compliance. Bravo for them!
Others do not, but list out on their marketing pages how they meet the various PCI DSS logging requirements. That’s great, but the lack of an AoC means that your QSA will now have to include the cloud provider in the scope of your audit.
Without that AoC, we’re back into risk-based decision making, since it’s a cloud service provider without any kind of PCI Responsibility Matrix to fall back on. Will the QSA be OK with looking at the cloud service management interface, seeing the retention is set and that you have alerting rules in place? Or will they want to understand the physical security, backup mechanisms, etc. in play to safeguard your log data?
In most cases, the cloud providers don’t have the bandwidth to deal with client auditors on an individual basis. If your QSA wants to dig into the solution, they may not get very far. That creates issues for you.
If your cloud provider isn’t PCI compliant but otherwise appears to have the necessary controls, one thing that could be useful for your QSA would be a SOC 2 Type II. It’s at least a formal description of the control environment and its operation, and may allow them to feel comfortable with your use of the service.
Advantages of a cloud logging solution
There are some advantages of using a cloud-based logging solution. The obvious advantage of any cloud-based solution is the transfer of the infrastructure management onto someone else. But there are other benefits.
By using an external provider for logging, you make it as close to impossible as is feasible to modify historical log records. Even with stolen administrator credentials to the management interface, the most an attacker is likely to be able to do in terms of tampering is to delete old logs ahead of the retention policy, which should get noticed.
Attackers don’t have any way to inject or delete individual records to cover their tracks. New log records are submitted through APIs which record their own unique timestamps, making insertion in the past impossible. There typically won’t be any API available to update or delete at the individual log record level.
Potential disadvantages of cloud logging
As discussed above, using a cloud provider that is not themselves PCI compliant creates some additonal work at the least for your QSA.
But assuming you’ve chosen to use a PCI compliant provider, there are still some potential issues to be aware of. PCI states that logs must not contain CHD, but remains silent on the issue of logging any personal data. If there is any possibility of personal data being logged, then you need to carefully consider whether there is any issue with transferring that data to the cloud logging provider.
Since the GDPR defines personal data quite broadly, to include things like IP address information, this is actually a real concern. It’s not enough to simply avoid transferring things like a person’s first and last name. If infrastructure logs contain the IP address of anyone receiving the protection of the GDPR, then appropriate arrangements need to be in place with the provider to safeguard that data.
When using any cloud provider, you need to look beyond the PCI implications when doing your due diligence. Understand what customer data may be transferred, where it will be stored, and how access will be controlled.
Cloud Logging Provider Selection Criteria
Before looking at some of the cloud providers currently in market, let’s consider some appropriate selection criteria.
Log Types
What do you need to be able to log? Consider at least 3 different types: (1) your application logs; (2) traditional infrastructure components such as server OS and firewalls; and (3) cloud infrastructure components such as AWS IAM activity. Each of these has their own logging mechanism that your chosen solution needs to support.
Locally developed applications can typically be updated to use whatever API is provided by your chosen cloud logging services, but it’s still worth confirming a logging SDK is available for your environment. But any off-the-shelf software you use that generates audit trails may not integrate directly with the cloud solution. In the latter case you may be able to use an agent on your servers to transform the logs then send them to the cloud, but carefully consider if you’re using the right solution at that point.
Infrastructure components often generate logs through the traditional syslog protocol. Does your cloud service have the ability to ingest this directly (how?) or do you need to use some kind of collector or relay agent deployed on site?
Cloud infrastructure components are a whole different issue. Since the cloud providers don’t want to introduce dependencies on your systems to their infrastructure, these typically log to a specified location, which you can then access through APIs. A common example of this is logging to an S3 bucket. How does your chosen cloud logging provider handle these? And does it provide coverage for all your cloud infrastructure?
Data Retention
Section 10.7 of the current PCI DSS (3.2.1) has some very specific requirements for log retention:
“Retain audit trail history for at least one year, with a minimum of three months immediately available for analysis (for example, online, archived, or restorable from backup).”
On this basis, you need to be able to go back 90 days without difficulty. Since you’re allowed to quickly restore logs within the 90 days from backup, it’s fairly clear that you don’t need 90 days of history be kept in an online index in your logging system. However, if you don’t have the full 90 days online, then you’re creating a situation where within the 90 day window, you use different approaches to access online data versus the archived data.
Most of these cloud providers do not offer 90 days of retention by default, although some suggest you call them to discuss custom plans with higher retention. You should understand your options, and determine whether you’ll need a combination of archived logs plus the online indexes in order to comply.
Popular Cloud Services
This is not a complete list of all providers offering cloud-based logging services, but it includes the ones people keep bringing up in business discussions. One of the great things about cloud services is that a new one could appear tomorrow that’s such a compelling offering, it will sweep all of these away. But until then, please consider the following list and do reach out if there’s anything you think should be added to the list.
Splunk Cloud
Splunk is arguably the best known of these SIEM platforms, and has been available as deployable software since the mid 2000’s. In 2013, Splunk expanded its offering with Splunk Cloud, providing a cloud hosted version of its popular platform. Splunk Cloud itself is hosted within AWS and offered as a managed service. As such, it’s less of a fully cloud-native solution, and more like paying an MSP to operate Splunk for you – it’s just that the MSP in this case are Splunk themselves.
Marketing Site: https://www.splunk.com/en_us/software/splunk-cloud.html
PCI Status: Compliant as a Level 1 Service Provider sec. 19 if you are deployed in their PCI-DSS certified environment (Premium Environment)
Other Security/Regulatory Certifications: FedRAMP (U.S. Federal Government), ISO 27001, HIPAA, SOC 2 Type II Report available
Operating Locations: Various AWS regions, including US (Oregon, Virginia, GovCloud), EU (Dublin, Frankfurt, London), Asia Pacific (Singapore, Sydney, Tokyo), Canada (Central)
Pros: Splunk provides for strong separation of data between customers through using per-customer dedicated VMs to run instances of Splunk. Everyone you encounter will know of Splunk; many will have actually used it at some point over the last decade too. Lots of really useful information between the Cloud Service Description and the FAQ.
Cons: Because it’s a hosted enterprise application rather than cloud native, dedicated infrastructure resources are used per customer. This creates a lead time for provisioning requests: Splunk advertise that you can be up and running in as little as 2 days; competitors are more like 5 minutes to pilot. PCI DSS compliance requires deployment in a Premium Environment.
Thoughts: Splunk is a good product, they know it’s a good product, so they can charge like it’s a good product. Having a PCI offering at all makes it stand out from the croud.
Solarwinds Papertrail
Solarwinds Papertrail is explicitly sold as a hosted log management system, rather than a more general purpose data management system. It selling points are a searchable event viewer, configurable alert notifications, and even a command-line tail feature like on Unix!
Marketing Site: https://www.solarwinds.com/papertrail
PCI Status: Not compliant, per Papertrail Security and compliance page
Other Security/Regulatory Certifications: N/A
Operating Locations: All data is stored within the United States
Pros: Designed by a company with a strong networking focus primarily to process log events. It does that very well, for a very reasonable price.
Cons: The Security and compliance page has a paragraph on how you can use Papertrail features to support PCI DSS requirements. That’s all well and good, but if Papertrail maintains the audit trails for your PCI CDE, then it becomes a part of that environment. Either you need their ROC, or you need to include it in your own audit; the latter isn’t really feasible with a cloud service that doesn’t even have a SOC report to fall back on.
Thoughts: I really want to like Papertrail because of features and price point, but they’re not formally PCI DSS compliant. At least they’re up-front about that, so you can make your own risk-based decision whether to include their services as part of your solution.
Solarwinds Loggly
Solarwinds Loggly is the other cloud-based log management solution from Solarwinds. Loggly was acquired by Solarwinds, rather than developed internally. Compared to Solarwinds Papertrail, Loggly is more fully featured in terms of ability to analyze and visualize the data it’s ingesting.
Marketing Site: https://www.loggly.com/
PCI Status: Unclear; Loggly state “… can be used in environments regulated by PCI, … You can control what information is sent to Loggly, and filter or obfuscate regulated information like … or credit card numbers … our billing provider Braintree, which is a Level 1 PCI DSS Compliant Service Provider.” This stronly suggests that Loggly itself isn’t certified, but their billing provider is.
Other Security/Regulatory Certifications: In one place, Loggly state “we keep your data safe in our facilities, which are ISO 27001, SSAE16 SOC-1 Type II, and SOC-2 Type II certified.” In another “We store your data in data centers that are ISO & SOC2 certified”, so it sounds like Loggly use certified facilities, but the Loggly service is not certified.
Operating Locations: Amazon Web Services (AWS) and Equinix, but it’s not clear which regions / facilities
Pros: Loggly can be thought of a value-added log management system. It does the usual log ingestion, filtering, alerting, and searching. But it also allows you to extract additional value from the data through visualizations and the ability to create dashboards of any KPIs that can be derived from the log data.
Cons: The ambiguity around PCI DSS and other compliance frameworks is a concern. Hosting in ISO 27001-certified facilities is certainly better than not, but it doesn’t say anything about how Loggly manage their services.
Thoughts: Loggly have a good reputation, but I’d like to see them provide more, clearer information about the compliance position and operating jurisdictions of their solution.
Sumo Logic
Sumo Logic has been around for about a decade, and is very popular in the DevOps world. Going beyond the standard log ingestion and rule-based alerting, Sumo Logic uses machine-learning to analyze the logs being processed and generate insights. They liken this to Google News scanning every article published, then extracting those likely to be of interest.
Marketing Site: https://www.sumologic.com/
PCI Status: Yes, Level 1 Service Provider
Other Security/Regulatory Certifications: EU-US Privacy Shield, HIPAA, ISO 27001, SOC 2 Type II
Operating Locations: Selected at account creation time from various AWS regions: Sydney, Canada Central, Ireland, Frankfurt, Mumbai, Tokyo, Northern Virginia, Oregon
Pros: Sumo Logic clearly understand the need for PCI DSS solutions, and offer a couple of targetted PCI DSS compliance options. First is a set of Apps bundled with the standard subscription which are intended to extract PCI-relevant insights from Linux, Windows, AWS CloudTrail, and Amazon VPC flow logs. Second is the PCI Compliance Application Professional Services, which is a combination of existing technology and Professional services customization. The solution analyzes your data and feeds it into dashboards covering 11 of the 12 PCI DSS requirements. This is only offered to Enterprise accounts, requires Professional Services (additional $) to install and configure, and certainly isn’t a one-stop-shop for PCI DSS compliance.
Cons: As a platform with significant value add, it’s not one of the cheaper options. Default storage available for retention is in line with their competitors, meaning that you can’t store 90 days of data online and have to use an archival solution for PCI DSS compliance or pay more.
Thoughts: With their various PCI Apps, and the machine learning technology, Sumo Logic have a very interesting offering. Their solution is probably overkill for small IT shops, but larger enterprises prepared to make an investment in Sumo Logic should be able to realize significant value. I like their transparency about where data is stored.
AlienVault USM Anywhere
AlienVault USM (Unified Security Management) is a suite of SIEM products built from a common core. USM Anywhere is the cloud-based offering; USM Appliance, per the name, is appliance-based for deployment on-prem. They also offer managed instances of USM through MSSP relationships. Beyond log monitoring and management, it offers asset discovery & inventory, IDS, and compliance reporting.
Marketing Site: https://cybersecurity.att.com/products/usm-anywhere
PCI Status: Yes, Level 1 Service Provider with Coalfire as QSA
Other Security/Regulatory Certifications: HIPAA, SOC 2 Type II
Operating Locations: Certainly U.S., elsewhere unclear
Pros: They outright sell their most expensive ‘Premium’ package as having 90 day retention online, and as being “Ideal … to meet PCI DSS audit requirements”. While it’s far from the cheapest option, you are getting additional tools necessary in a PCI environment including vulnerability assessment and host intrusion detection.
Cons: Not designed as cloud native software, so scalability and provisioning designed for more traditional deployment models.
Thoughts: If you are looking to deploy a suite of security technologies at a greenfield site, or are looking to consolidate, AlienVault USM Anywhere may offer you value because it’s a package of different techologies. If all you’re going to use is log monitoring and management, it’s hard to recommend because you’re paying a lot of money for features you’re not using.
LogDNA
LogDNA is a relatively recent addition to the market, with the company being founded in 2015. LogDNA is designed to ingest logs from applications and system infrastructure components, through both its REST API, and also directly through syslog.
Marketing Site: https://logdna.com/
PCI Status: Yes, Level 1 Service Provider
Other Security/Regulatory Certifications: HIPAA, EU-US Privacy Shield, SOC 2 (presumably Type II)
Operating Locations: “mostly on Azure US-East“
Pros: Sensible pricing, and a free option with no retention so you can get started and evaluate the platform for your needs. The free option can ingest “unlimited” amounts of data – they can offer it free by providing zero retention on this tier. On the per-GB plans, there are monthly minimums, but those are very reasonable for this type of service. Ability to ingest from syslog as well as its own API.
Cons: Their standard pricing tiers only offer up to 30 days of retention, so to meet the 90 day PCI requirement, you need to use a combination of online data plus recently archived logs.
Thoughts: I like LogDNA, and am considering it for my next implementation. Just watch out for retention requirements.
Datadog
Datadog offers a suite of product features in the infrastructure and application monitoring space. One of which is targetted at log management, and offers both event correlation and alerting, and also indexing for historical analysis.
Marketing Site: https://www.datadoghq.com/
PCI Status: No comments regarding PCI found
Other Security/Regulatory Certifications: SOC 2 Type II
Operating Locations: Customer data currently resides in the U.S., primarily in Virginia
Pros: Pricing for the logging solution is fairly straightforward. However, unlike some competitors pricing is done by number of log records rather than per GB – this can make direct comparisons somewhat tricky if you don’t know the average size of the log records you’re planning to store. The ability to expand into Application Analytics, and to visualize network utilization, using the same tool is useful for sites looking to maximize visibility through one pane of glass.
Cons: The maximum online retention offered through a standard plan is 60-days. Extended retention plans are available, but you have to contact them to request those. There is an ability to ‘rehydrate’ archived logs for audits and historical analysis, but that raises friction for PCI purposes vs. just having 90-day retention by default.
Thoughts: Datadog has a very interesting offer, which would be much easier to recommend if they were clearly PCI DSS compliant. I’m speculating, but the default 60-day maximum retention is probably designed to trigger a conversation with prospects working under the 90-day PCI requirement.
Amazon CloudWatch
Amazon CloudWatch is a suite of services available in the Amazon AWS cloud environment. CloudWatch Logs is the service component which allows you to aggregate all of your logs and store them in one place. CloudWatch Alarms is a parallel service which can monitor various metrics, including searches of CloudWatch Logs, and send alerts if certain criteria are met. Between the two you have the core of a SEIM platform.
Marketing Site: https://aws.amazon.com/cloudwatch/
PCI Status: Included in scope for AWS PCI DSS
Other Security/Regulatory Certifications: Multiple, for details see https://aws.amazon.com/compliance/programs/
Pros: When hosting at AWS, events from other services are available for use within CloudWatch. So typically the only custom configuration required will be to ensure CloudWatch ingests any application logs you are generating.
Cons: Like all AWS services, CloudWatch starts out cheap at low volumes, but can become expensive as the volumes ramp up. Keep a close eye on the number of events you are watching for, and ensure that the retention policy you configure meets your regulatory and business requirements.
Thoughts: If you are already hosting with AWS, then using AWS CloudWatch Logs for your PCI DSS logging and alerting requirements is almost a no-brainer. If you’re not already hosting with AWS, then you should probably look elsewhere.
Final Thoughts
It’s an unfortunate but almost universal truth. In general, the more you pay for a service, the better the level of public disclosure of compliance features. In turn, that’s often because the underlying features are more comprehensive, and therefore worth bragging about. That’s clearly illustrated here, with Splunk Cloud and Sumo Logic both being very good in this regard. AlienVault is a little hit and miss, despite its high price point: very good for disclosing who the QSA was for their PCI audit, but they make it very hard to find out where they will store your data.
If I was just starting out and on a tight budget, I would give serious consideration to LogDNA. It does what it says, for a sensible price, and they went to the trouble of going through a Level 1 Service Provider PCI DSS audit. If the purse strings were a bit looser, and I was looking to get a bit more capability from the additional spend, I’d consider how Sumo Logic and AlienVault could fit with my plans. They are very different solutions, but both provide more than just basic log management.