The last 5 posts to the Eiara Blog
“Data is the New Oil” as an idea really took off in 2016, touching on the modern value of a business revolving around data collection, of the value of their users or the demographics of their customers. We have framed it as the hidden value we didn’t know we had, waiting for us to unlock it.
However, in the light of the reveal of Facebook allowing Cambridge Analytica deep access to their treasure trove of data, explicitly to achieve ends that are deeply concerning to modern democracy and the health of the Internet at large, “data is the new oil” requires considerable re-examination.
For Facebook, this was their business model. It was always their business model. Allowing access to their data was undertaken with the full support of their stated goals, of what they set out to do.
This was their intent, and the consequences belong solely to themselves.
So when we pursue “data is the new oil”, when we seek to unlock our own hidden value, how will we be different from Facebook? How can we learn from their example?
Data is powerful, offering deep meaning and insight never before accessible, but it comes with new concerns and requirements. An oil spill is a catastrophic consequence, a necessary consideration of its use.
But to call data the new oil, we must also understand that data is as dangerous as oil and consider ideas such as data leaks or data spills, and describe the catastrophic events that are as much a consequence and consideration of the use of data as of oil.
When we then pursue the value of data, we must build organisations and policies with this in mind, that assume adversarial usage and intent, that believe that the behaviours of Cambridge Analytica are not aberrations or outliers, that this is the norm. To understand that Cambridge Analytica is the organisation that got caught, not the only organisation behaving this way.
We must believe that, without a robust organisation, without that consideration for adversarial intent, that data spills are inevitable.
We must ask, what results do we want? Do we value the privacy of our customers and users? We must ask, if we truly value them and their privacy, will an agreement not to misbehave ever be enough?
We must ask these questions because there is no way to unleak the data, no way to clean up a data spill. Consequences, regardless of our intent or stated values and goals, will remain consequences.
To use data as our new oil, we must realise the value of data without creating the conditions for privacy breaches. We must consider the data spill and work tirelessly to prevent it. We must analyse our data ourselves, and offer only our own interpretations. We must build our organisations such that our people care about safety and privacy, that the organisation promotes safety and privacy, that it makes violating safety and privacy difficult.
We must do more than say we care, we must build a system that cares, that will care when we are not watching.
And we may think that we are too small to notice, that we will not be that target, but there is no such thing as a target too small to notice. In the new world, the value of data grows when it is combined with other data, our data included.
Cambridge Analytica targeted a giant, and succeeded. We will be targeted as well. Will our data adversaries succeed against us?
Cambridge Analytica has shown us the future of “data is the new oil,” and the public concern over that future has shown us the questions that we must ask, the culture that we must strive for.
This is our future.
Are we ready for it?
AWS’s Lambda, and the serverless architecture in general has been a huge boon not just to our ability to build experimental web software, but also a new level of interesting and valuable workflow and automation tooling.
However, one of the things that’s been regularly difficult is easily integrating the build process into our Terraform cloud automation, in a manner that is consistent with the goals and philosophies of Terraform.
To that end, we’ve open-sourced the Terraform module we use to automatically build Python packages for use on AWS Lambda. It’s intended to easily integrate into your existing Terraform infrastructure and only rebuild the payload archive on any change to your Python Lambda function.
You can get the module here.
One of the core aspects of modern DevOps process is the CI/CD pipeline, where newly built deployment artefacts can be easily pushed into our various environments, automating the tedium of deployment and going live.
In terms of convenience, this is great. Complicated deployment procedures are turned into turn-key automation, enabling us to get new software into testing or production, and getting feedback on our work has never been easier.
However, when we’re building these systems, security is often a secondary thought, or not considered at all, and it’s all too easy to use highly-privileged credentials to set up these deployment roles.
We recently built out a deployment system for AWS Lambda, using Terraform, and it takes a lot of thought to consider what, exactly, our deployment role should be able to do, but also what we’re trading when we limit things like that.
AWS Lambda is easy to deploy and easy to work with, and there’s a myriad of opinions on how to deploy code to it.
We wanted to be able to use our standard TravisCI-based build process for pushing new versions of a codebase to AWS Lambda, by using the industry-standard tool Terraform to manage the deployment process.
As Terraform requires AWS API access, the fastest way to achieving this goal is just to make standard AWS access keys and insert them into the TravisCI UI. While this is largely safe, in that TravisCI is well-defended and manages their own breach detection, including highly-privileged access keys here is still a risk if someone breaches your source code repository or one of your developers’ workstations.
Because TravisCI has to be trusted, those access keys are available in the clear, and so anyone with access to your source code repository must have access to those keys. This would be the same with any CI provider, because any CI provider would be in the same position of high trust.
So what should we be doing?
In this, and every case, we should be thinking about how little access we need to get the job done.
When we built our system to deploy into AWS Lambda, we asked what we needed and how we should be thinking about deploying functions.
We decided on using a versioned S3 bucket to provide a historic record of function payloads, and a limited execution role to be passed to the Lambda itself.
From those decisions, the smallest set of permissions needed to deploy a function would be: - Write to an S3 bucket - List, create and delete functions - Pass in an IAM role for execution
It’s important to note that our deployment isn’t going to be creating the S3 bucket, or the IAM role needed for execution.
The S3 bucket permission is fairly straightforward, we need to be able to upload payload files, and know where to upload them. This part of our role won’t need to delete or modify existing files, as we’re focussing solely on letting S3 manage the historic record.
The second piece is being able to modify functions. Because the CI system is authoritative when it comes to the deployment of functions, it needs to have the permissions to make these modifications.
However, the CI system should only be authoritative over its own functions. In our design, we implemented this by requiring a CI-specific prefix for the function names, ensuring that functions created through other means couldn’t be touched.
Finally, a newly-created function needs to have an execution role associated. This role determines what a function is capable of doing, and this is probably the most important aspect of ensuring a consistent security profile when building this sort of CI service.
In order to create a function, the deployment role needs to be able to pass a role. In general, this could be any role, from the most basic AWSLambdaBasicExectionRole to the core admin role.
In order to ensure we’re not able to assign an admin role to our function, we set up our deployment role to only be able to assign a single, pre-determined execution role, and only that role.
By doing this we can be assured that our functions can’t leak elevated access, and that functions can never do anything more than we expect.
Limiting our deployment role to this extent does come with a major potential downside, in that it introduces a gatekeeping stage every time we need to deploy functions which require different levels of access to our AWS accounts.
This kind of gatekeeping can be one of the major drivers behind creating a shadow IT environment, because it interferes in rapid testing and iteration processes.
This level of deployment role lockdown may not be appropriate for your environment, but it is necessary to consider and have the conversation in your team as to its necessity and impact, as well as the impact of not implementing these ideas.
So what does the setup for this deployment role look like in practise? Let’s look at some Terraform code to set it up:
This first block of code defines the IAM policies required to create AWS Lambda functions. These policies are the core that enables what our CI role can do.
data "aws_iam_policy_document" "lambda_create" {
# So that CI can bind the Lambda to the execution role
statement {
actions = [
"iam:PassRole",
"iam:GetRole",
]
# Allows us to only provide one role to our lambda function
resources = [
"arn:aws:iam::account-id:role/role-name",
]
}
# So that we can create and modify all the Lambdas
statement {
actions = [
"lambda:CreateAlias",
"lambda:CreateFunction",
"lambda:GetPolicy",
"lambda:DeleteFunction",
"lambda:GetFunction*",
"lambda:ListFunctions",
"lambda:ListVersionsByFunction",
"lambda:PublishVersion",
"lambda:UpdateAlias",
"lambda:UpdateFunctionCode",
"lambda:UpdateFunctionConfiguration",
]
# But only the ones with the prefix
resources = [
"arn:aws:lambda:region:account-id:function:cicd_prefix_*",
]
}
}
data "aws_iam_policy_document" "s3" {
# Allows our CI provider to list the payloads bucket
statement {
actions = [
"s3:ListBucket",
"s3:GetBucketLocation",
]
resources = [
"arn:aws:s3:::example-deployment-bucket",
]
}
statement {
# So we can update files in our own bucket, but not get or delete them.
actions = [
"s3:PutObject",
"s3:PutObjectAcl",
]
resources = [
"arn:aws:s3:::example-deployment-bucket/*",
]
}
}
The second part is the user itself, and the group membership. By creating the user in this way and isolating it from the existing roles in AWS, we’re able to strongly control our role capabilities, and providing a skeleton for creating new deployment roles in the future.
# Creates an AWS user to hold the CI deployment role
resource "aws_iam_user" "ci" {
name = "ci_user"
}
resource "aws_iam_group" "ci_group" {
name = "ci"
path = "/ci/"
}
resource "aws_iam_group_membership" "ci_group_membership" {
name = "ci-group-membership"
users = [
"${aws_iam_user.ci.name}",
]
group = "${aws_iam_group.ci_group.name}"
}
# Create the policy that permits creating Lambda functions
resource "aws_iam_policy" "allow_create_lambda" {
name = "allow_create_lambda"
path = "/ci/"
description = "allows limited lambda source creation and modification"
policy = "${data.aws_iam_policy_document.lambda_create.json}"
}
# Create the policy that permits uploading Lambda functions to S3
resource "aws_iam_policy" "allow_ci_S3" {
name = "allow_ci_S3"
path = "/ci/"
description = "allows limited S3 access for CI"
policy = "${data.aws_iam_policy_document.s3.json}"
}
# Connect the policies to the group that our CI user is a part of
resource "aws_iam_group_policy_attachment" "lambda_attach" {
group = "${aws_iam_group.ci_group.name}"
policy_arn = "${aws_iam_policy.allow_create_lambda.arn}"
}
resource "aws_iam_group_policy_attachment" "s3_allow" {
group = "${aws_iam_group.ci_group.name}"
policy_arn = "${aws_iam_policy.allow_ci_S3.arn}"
}
# Creates credentials that can be used in your CI provider of choice
resource "aws_iam_access_key" "ci_credentials" {
user = "${aws_iam_user.ci.name}"
}
Pulling the credentials as we are in aws_iam_access_key does have the implication that these credentials are being written into the Terraform state file, which may be inappropriate for your threat model.
If it is inappropriate, generating the access keys from the AWS console will be a better option, and should be explored instead.
With the rise of DevOps, we've seen a shift from the old world of a delivery handoff and disconnect of developer resources from operational concerns to delivery teams with a full gradient of skills, ranging from development best practices to appropriate SRE1 policies.
Coinciding with the rise of infrastructure-as-code has been the awareness of how some practices are harmful, the most primary example being "don't check your secrets into source control."
We hear numerous stories of this having happened, of AWS keys compromised or SSL certificates leaked because they were inadvertently checked into Github, to trusted self-signed certificates being used to mask malfeasance. From this example, one of the patterns we've picked up is to encrypt our secrets, to ensure that they are never stored unencrypted. This is an important step, and we can be assured to some extent that our secrets will remain secret, remain safe.
Security requirements and recommendations often touch on this necessity, point to a lack of encryption as a flaw in your information security architecture.
So we encrypt our files. We feel safe. We feel secure. We are secure. Right?
In this case, encryption has been added because our secrets must be encrypted. But, why? Because our security requirements said we must.
But, why? So that the secrets cannot be read by parties unknown.
So how does encryption prevent that?
If an attacker gains access to the ciphertext, say through a Github account breach, we feel as though we have security, that they will not be able to access the file or our secrets. But we won't know if they ever succeed in gaining access except through secondary means, such as detecting unauthorised queries.
If an attacker gains access to our deployed servers, servers which must have the secrets in the clear in order to use them, all our encryption is for nothing because anyone on that server can access those secrets, and use them as they see fit.
As a core axiom of security planning is the awareness that we will, one day, lose, we must treat our secrets as though they could be breached and misused at any moment, we start to see that just encrypting our secrets in our Github accounts or configuration management data stores is not and can not be enough.
Instead, we need to think about access.
Let's consider a standard three-tier web application deployment, with our loadbalancers, web application tier and database. Each layer of this design has access to a set of secrets, from the loadbalancers with SSL for encryption to our clients, to the web servers holding passwords to log in to the database.
At each tier, the machines are trusted users of those passwords, not just expected to have them but required. The web server can't do its job if it can't connect to the databases, after all!
From that, a user with access to that machine is also trusted, to the same degree.
This tells us that we must consider a world where our secrets may be accessed at any time, by people outside of our control, and since the location of access is trusted, we cannot rely on encryption to enforce access control.
So, what if we accepted that? That anyone or anything on that server could reasonably access those secrets?
At that point, we start thinking not of restricting access but of awareness of access. If I know that my web application restarted and decrypted the database password, then I have external awareness of that fact. If my secrets are accessed and there is no correlated application restart, I can now ask if it was a legitimate access, and react appropriately if it wasn't.
By changing our focus from performing an action to considering a result, we move from encrypting secret material to being aware of illegitimate access of the secret material by using encryption technology.
The approach of treating access as the fundamental point of instrumentation is one of the underlying ideas behind Vault and Eiara's own Ratchet proposal, where encryption is used to provide a gateway, and accesses are always logged. We are given new ways to frame "is this a valid access?" and "should I continue to trust this server?", questions that are difficult to answer without this consideration.
Site Reliability Engineering ↩
The DevOps movement has been running for close to a decade, and has produced some of most powerful tools for managing and deploying software.
We talk about how automation improves our lives, but we don’t talk about the underlying goals that drive why we focus on automation. We talk about moving from specialist servers to classes of machine, with only an implicit idea of the problems we are solving.
As part of our journey, we’ve identified three core principles that underpin, on a technical level, everything that DevOps is trying to achieve and improve, the principles of isolation, repeatability and determinism. These principles are the guiding forces behind the technical choices that our tools are making, and frame the way we think about reliability and resilience.
In DevOps terms, the isolation principle covers clean lines of authority within our technical components, where each component of our infrastructure must be separate from each other piece by the greatest extent possible.
In a deployment sense, this tells us that our product is distinct and must be considered distinct from the operating system, and isolated from changes.
In a microservices sense, we’re taught that each service must have a strong, defined interface between other components, one that is static and unlikely to change.
The next property speaks of our processes, insisting that repeating the process is possible. In order to achieve that possibility, we are required to fully capture every detail of the process, in a form that can be consumed by others.
A process isn’t repeatable if it relies on steps in a single persons’ head, after all, and so that specialist knowledge must be documented, each step fully explored, and each stage thoroughly captured.
In a server sense, this tells us that a new server can be created from scratch in a known and knowable way, that we can give that task to anyone on our team and be reasonably assured that the process will be completed.
This principle is the first time we begin to capture the necessity for automation.
The final property tells us that the process we’ve defined must always produce the same result. If I set up a server, then it must be the same as the server you set up. If I install the product, it must be the same as if you install the product.
This final tenet demands automation due to the difficulty inherent in achieving consistent results without automated testing and build processes.
Determinism seems like it should be a part of Repeatability. After all, doesn’t a repeatable process imply that the results will be the same? Rather, Determinism posits that people performing a process will produce similar, not identical results, and the goals of Determinism is to fully control for that variation.
Using these core principles we are able to see the very powerful properties that DevOps practises offer us, enabling new forms of deployment and maintenance of our infrastructure.
For instance, the DevOps principles insist we invest in both automation and consistent and decoupled interfaces between our components, which allows us to begin to treat our infrastructure as disposable. Why would we modify a system in an attempt to repair it when rebuilding it from scratch is an automated, repeatable, and deterministic act?
But a pure replacement does not help us diagnose the failure. Fortunately, instead of removing and replacing failed systems, we can disconnect a failed machine and isolate it for forensics and analysis, all without impacting the core of our infrastructure.
Being able to discard and recreate our infrastructure in this manner is only possible because we can guarantee the result of our process, because we know that there is no tight coupling that will break when we remove critical infrastructure, and that possibility only exists by adherence to the DevOps principles.
Strongly isolated components in our infrastructure also allows for access to the true power of the cloud, scale. By providing standard interfaces and decoupled properties, any component that communicates with another component shouldn’t need to communicate with just one, it could communicate with one of many.
With determinism and repeatability we are able to reach the promise of the elastic cloud, because we are never in doubt of our ability to guarantee the consistency of our processes.
We can start to talk about the reliability and resilience properties that DevOps unlocks. When we guarantee that what has failed will be replaced with an identical result, we can treat failures as a known quantity. Test environments to reproduce the failure can be constructed without complicated work because our automation ensures the process will be identical.
This allows transient failures themselves to be managed automatically, reducing the workload of operational firefighting and freeing up resources to work on improving resilience in other areas of the infrastructure.
Finally, adherence to DevOps principles allows for development and quality assurance to happen with access to an environment as close to production as possible. Changes can be tested, tried, and retested without complicated work creating environments or long change requests, increasing the turnaround from development to testing.
Complete automation also allows for tested environments to be vetted before introduction into production, with full, visible and automated visibility into the differences now being within reach.
These are the properties that the DevOps principles unlocks, by viewing our deployments as integral to our software development and by adherence to the principles, we are no longer limited to “I think”, but are much more able to say “I know.”