Getting past the ‘muck’ to make a success of secrets management
The slow, hidden build-up of operational and security issues must be addressed before it can cause real issues. Dennis Mastin, DevOps Solution Engineering, CyberArk discusses the importance of secrets management, risks presented by security management sprawl, and how to manage this vulnerability.
2006 might feel a long time ago to you, but by then one of today’s foremost technology entrepreneurs, Jeff Bezos, was already a highly influential technology leader.
Bezos became very well known in developer circles that year for comments he made at MIT Tech Review’s Emerging Technologies Conference, where he coined the term ‘undifferentiated heavy lifting’ in a keynote address. The phrase was intended to encompass some of the obstacles that web developers have to navigate, and has since come to define the ‘muck’ which takes up about 70% of developers’ time – such as backend work like bandwidth management and server hosting – that was essential to running a service successfully, but adds no extra value to the end product.
Today our IT environments are much more focused on DevOps, agile, and hybrid, but it’s a phenomenon that’s still prevalent, manifesting as a serious (although not unique) issue among the islands of security that we have created among our use of multi-cloud developer tools and applications. These extra tools and applications add value to the company’s mission, but the backend work that comes with maintaining their credentials does not. Frequently, we find that the credentials needed to use these apps, platforms, and tools, are stored in a mess of different places.
The crux of it is that the way those credentials are managed and accessed is a form of undifferentiated heavy lifting, one which has resulted in a slow, hidden build-up of operational and security issues for organisations, which will have to be addressed before they cause real damage. While it’s not the same threat as a world recession, it does present a very real and growing risk to organisations that ignore the signs.
Keeping DevOps credentials under control
Is it truly a problem to have millions of credentials or secrets? Not necessarily. Most tools have a way to manage secrets for you, whether that’s Red Hat, Kubernetes, or cloud providers such as Azure. Using more than one or two of these systems is the problem, though. As so many organisations have hybrid environments and use multiple container management and automation tools, amongst others, it becomes very easy to end up with secrets management sprawl.
Keeping credentials and secrets management in order is critical. The Verizon 2021 Data Breach Investigations Report found that stolen credentials were used in over 20% of breaches, while earlier this year SentinelOne found an infected code library in the wild that attempts to instal malware on Macs used by software developers, which if successful, downloads software that monitors what the developer does via their device’s microphone, camera, and keyboard. Computerworld acknowledged that this attack signifies that cybercriminals understand that software developers may be the best attack vector into the Apple ecosystem. Equally, in 2019, Glasswall reported that software developers were the role most targeted by hackers going after the technology sector. The reason why is simple: developers build the software and usually have administrator privileges. A compromise there is a high value entry point to the rest of the organisation.
Scoping out susceptible secrets
Secrets management sprawl thus becomes a point of vulnerability, as those same teams are doing the aforementioned undifferentiated heavy lifting. It is arguably impossible to administer credentials and security for all of these environments, tools, and platforms while still knowing where they all are, without making any errors. The first way this sprawl usually manifests is while managing operations, when the security team recognises they are not able to administer secrets management tools to the required level of service which, in the confidentiality, integrity, and availability (CIA) triad, would show up as an availability problem. The issue also presents a confidentiality problem when a CISO or equivalent recognises that secrets and credentials are at risk of compromise due to the high potential for errors being made in this administration-heavy discipline.
Solving secrets management sprawl is not easy, and it involves some of the same automation skills used in DevOps tools being applied to the management of the secrets they contain. Automation of DevOps security not only minimises human interaction and manual intervention, but also reduces administrative costs and the possibility of errors. For example, automation can proactively rotate common secrets – such as passwords, keys, and certificates – which can help organisations prevent attackers from gaining access to DevOps tools and keys. Equally, automated tools can be used reactively if a breach is detected – for example by immediately rotating secrets to move them out of harm’s way.
Ultimately, using a centralised and automated approach to secrets management is critical because it alleviates the load of undifferentiated heavy lifting from developers. These individuals are key creators in the new digital-first economy. Giving them time back to focus on other, more important, tasks within DevOps pipelines means they are able to better focus on driving business value.