“To protect your software supply chain you need proactive security”
Eran Orzel is Senior Director of Argon Customers and Sales, at Argon Security, a recent acquisition by Aqua Security. He speaks to Sarah Schlothauer at JAXenter.com on the rise in software supply chain attacks, how to keep open source code free from vulnerabilities, and more.
JAXenter: What did we learn about supply chain attacks in 2021? What are hackers primarily targeting and why?
Eran Orzel: In 2021, attackers focused heavily on three areas: open-source through exploiting vulnerabilities and package poisoning, compromising CI/CD tools and code integrity, thus, exploiting the software supply chain process and supplier trust to distribute malware or backdoors. Our recent research found that software supply chain attacks more than tripled between 2020 and 2021, indicating cybercriminals have clearly discovered some low-hanging fruit to target organisations’ networks and hold them to ransom – or even worse. Following the SolarWinds attack, these attacks have rapidly increased in number and sophistication and enterprise security teams are scrambling to protect what’s theirs.
Cybercriminals have learned that going through the back door of the supply chain they can access tens of thousands of customers. The three large high profile open-source poisoning attacks in November of 2021, educated attackers through successful attempts to exploit and access applications that relied on them.
The software supply chain process is a core component of the modern application development lifecycle. Leaving this wide attack vector open threatens to severely reduce a company’s application security posture, potentially exposing sensitive data and creating additional entry points into the application in runtime. Most of the time, security teams have no visibility into this process until it’s too late, lacking vital built-in preventative capabilities within the CI/CD tools and processes.
Knowing how much these attacks can paralyse an organisation with sometimes months of dwell time, and how long the attacks can go undetected makes them especially attractive target points for attackers.
By virtue of the fact that open source is open – everyone can use it, so the magnitude of this usage proves a key security issue.
JAXenter: What can we do to keep our open-source code secure from vulnerabilities?
Eran Orzel: Today, almost all commercial software relies on open-source components. By virtue of the fact that open source is open – everyone can use it, so the magnitude of this usage proves a key security issue. Sometimes, even when a vulnerability is discovered, it isn’t solved immediately but it can still impact users. It’s different if it’s an active attack, as in this case it will be immediately patched. But weaknesses can often be pushed back for developers to fix in a later upgrade. Developers want to innovate and are hard to tie down. But, it’s this cavalier approach to security that introduces immediate risk and it’s one of the fastest-growing methods for an attack.
Two common attack types leverage open source packages. The first is exploiting packages’ existing vulnerabilities to obtain access to the application and execute the attack (for instance, the recent Log4j cyberattacks). The second one is package poisoning, which involves planting malicious code in popular open-source packages, and private packages to trick developers or automated pipeline tools into incorporating them as part of the application build process.
There are three key security measures teams can take:
- Always ensure there is a security element to scan all open-source packages and determine their risk factor, this will help you to know if you’re under attack, or if there are indications that suggest you need to prevent developers from using this package. Due to the complexity of this application, it is recommended to have a process to monitor what new open-source packages are getting on to your application.
- It’s vital to have a mechanism to review issues regularly (perhaps monthly or weekly) and to have a cadence for how to limit the exposure. You need to have a fix process that is fast enough so that if there is a critical vulnerability, you can identify it immediately, notify customers, issue an update and minimise the damage.
- An incident response process helps teams to prevent attacks or, to identify the attack when it happens, and start the patch process to move users to a safe version. An appropriate response requires the right process and the right tools to enable the quick response required to protect your application and customers.
JAXenter: Do you think the average company is doing enough on the security side to protect itself from harm? If not, what should companies be focusing on?
Eran Orzel: A disconnect remains between security and developer teams in the software supply chain and the average company lacks knowledge and awareness to protect this vector. and to be fair, until fairly recently, there were no effective tools to combat this security issue.
There is typically a wide array of critical misconfigurations and vulnerabilities in the pipeline tools, which reduces the overall security posture of an organisation’s environment. Teams don’t have visibility into the supply chain risk, and the adoption of security tools designed to protect the build process in the CI/CD pipeline is fairly limited in most organisations. In addition, established agile DevOps practice prioritise rapid commits and deployments, putting security controls in place too late to prevent malicious activity.
Companies should be focusing on securing the software supply chains themselves, as well as improving the quality of the component of the code they are using. This involves having access to the tools and deploying security solutions. Automation is key and end to end visibility is critical. They then need to secure their open-source packages, scanning for the code itself and making sure there are no vulnerabilities. The cost of fixing is much more bearable if fixing is part of the development process.
Many organisations believe they lack the resources, budget, and knowledge to deal with supply chain attacks but scrimping on security isn’t an option. In addition, AppSec teams typically lack cooperation from development and DevOps teams, therefore organisations must make a culture change to promote security as part of development if they want to safeguard their infrastructure and AP.
Real security is a marathon, a long-term culture shift within an organisation; not a bandaid for a few errors.
We’re all aware of outside malicious attackers, but what about accidental vulnerabilities from inside the company? How can we train employees to keep sensitive data safe?
Eran Orzel: A robust software supply chain solution will help protect from two internal risks – mistakes, or human error, and malicious activity by employees or insider threats. It’s critical to make sure that people who are building the software are aware of what alerts to look out for and understand the importance of security. You need a blend of technology and human expertise for an effective solution.
Workers often take security training, but subsequently, forget what they have learned. Various issues can arise around misconfigurations or errors in writing code. But whether it is accidental or intentional, organisations must learn a way to fix this.
Argon has created a solution that helps developers fix their own mistakes as part of the development process without any additional training. With hundreds of pre-set rules to ensure that any breaches, vulnerabilities, and suspicious activity are detected, it’s a smart process that can correct a mistake or rule violation. Developers are aware that there is a security process that is safeguarding them and they must comply with, but also that the system will recommend the fix and help them secure their code. In this way, mistakes can be stopped immediately, and minor mistakes are also flagged for developers to fix less urgently.
Real security is a marathon, a long-term culture shift within an organisation; not a bandaid for a few errors. As developers get security alerts and suggested fixes, this approach establishes the process of writing code the right way. Getting the buy-in from developers on security approaches is vital to stopping potential incidents before they happen, as systems alone cannot prevent them. DevSecOps is a collaboration between the three teams and the only way to protect large environments.
JAXenter: How do attackers use CI/CD pipelines to gain access and how do we remedy that vulnerability?
Eran Orzel: Attackers can take advantage of privileged access, misconfigurations, and vulnerabilities in the CI/CD pipeline infrastructure (e.g. source code management system, build agent, package registries and service dependencies), which give access to critical IT infrastructure, development processes, source code and the applications. A compromised CI/CD pipeline can expose an application’s source code, which is the blueprint of the application, the development infrastructure and processes. Attackers can then change code or inject malicious code during the build process and tamper with the application (e.g., SolarWinds).
This type of breach is hard to identify and can cause significant damage before it is detected and resolved. Attackers also use compromised package registries to upload compromised artifacts instead of legitimate ones. In addition, there are dozens of external dependencies connected to the pipeline that can be used to access it and launch attacks (e.g., Codecov).
Arguably, the weakest element of the pipelines is the way they are constructed. Collaborative tools focus on automation and speed, not security. There are many places where vulnerabilities can creep in, particularly with workers connecting from home and everything is run by script. e. by accessing the pipeline, attackers can manipulate the code and the application or the infrastructure.
The first way to remedy this is to secure the tools as part of the project, identifying the vulnerability and remediating them immediately. Blocking access through vulnerability and misconfigurations, making sure nothing has been left open for the attackers, is key. It’s important to limit the access privileges of users, for instance, if a user has left the organisation but their credentials are still on the system, this allows them access to the whole environment.
To protect your software supply chain you need proactive security that provides 24/7 checks for abnormalities and unusual behaviour. This solution will look at the tools, process, code and components from the beginning, checking the security and quality on each step of the process letting developers know if there are issues in real-time ensuring good code quality and that they can release the software safely.
JAXenter: According to your customer study results, there was triple the amount of software supply chain attacks in 2021 compared to 2020. Why do you think this is?
Eran Orzel: Last year was a breakthrough year in hacker intelligence. In 2020, the high-profile source code leaks showed those processes weren’t protected and that the software supply chain was fair game. It’s not a well-kept secret and hackers are looking all the time for new ways to penetrate attacks. Targeting the components of the software supply chain allows bad actors to compromise more victims in one hit and achieve widespread distribution of malicious code. If they notice the software supply chain has no security, why would they try to access cloud infrastructure when they can get the access code easily and gain control of a network? It’s a case of ‘less effort, more gain’.
All the companies surveyed by Argon didn’t have security built-in in their software supply chain, which created the big gap attackers were exploiting in 2021. This may be in part due to current security solutions such as code scanners or firewalls cannot stop such attacks, Because of the speed and complexity and amount of people involved in this process, securing the software supply chain is no easy task. A security solution is needed that’s integrated into the process, apply security practices on it and works at machine speed to ensure a holistic view of the software supply chain and the ability to prevent attacks.
If security teams can bolster their collaboration with DevOps teams and implement automation of security within development processes, they can protect their software supply chains.
JAXenter: Do you think this year will get better or worse in terms of security and why?
Eran Orzel: As organisations move to more cloud-based applications and services, the software supply chain is becoming a critical part of the application and being their weak link for attacks. Just as in 2021, attackers will put more effort in finding more poisoning attacks for open source and leveraging vulnerabilities, scan the tools for openings and take advantage of user mistakes. Until the gap is closed, they feel compelled to continue, as they’re ‘in the money’. While organisations catch up with the problem, attackers are identifying what will be next.
The knock-on effects of these attacks are increasingly costly – not just financially, but also in terms of long-term brand damage. Many of us wouldn’t be a customer of a bank which has a two-week downtime every few months. Customer loyalty is everything in our fast-moving digital world.
The biggest issue for organisations is the lack of collaboration between security and development, which means huge exposure to risk. If security teams can bolster their collaboration with DevOps teams and implement automation of security within development processes, they can protect their software supply chains.
On a positive note, increased industry awareness should see an uptick in software supply chain security investment, while the steep learning curve of organisations will continue from 2022 through to 2023. Attackers will retain the upper hand, being three steps ahead, and the larger enterprises that can afford security will be the best at protecting themselves. Those organisations which take steps to understand their weaknesses, improve their DevSecOps team collaboration, and adopt built-in security solutions will protect their critical assets and empower developers to own security and their customers to trust their software releases.