npm operations incident autopsy: Use two-factor authentication to protect yourself
Last Thursday, someone gained access to an npm publisher’s account and used it to publish an unauthorized update of a popular package which contained malicious code. “What the code did was attempt to steal additional login tokens for npm from these other users, by sending them to a remote server,” Laurie Voss explained. “This, if successful, would have allowed them to hijack other packages and compromise more users.” Learn more about the incident here.
JAXenter: The incident of July 12th affected ESLint and was caused not by a breach elsewhere, as ESLint also confirms. Can you give us a short explanation as to what happened and who was affected?
Laurie Voss: ESLint is an organization in which multiple volunteers collaborate to create open-source software. There’s about a dozen people who have access to publish the ESLint’s packages to npm. Per ESLint’s blog post, one of ESLint’s members was using a password they had re-used on other sites, and had not enabled two-factor authentication (2FA) provided by npm. ESLint believes the beginning of the attack was somebody discovering this user’s password from some other source and then trying it against their npm account.
Once the attacker had a username and password that worked, they logged in to npm and published a new version (3.7.2) of the package “eslint-scope”. This new version had malicious code in it that would execute when a user downloaded and installed eslint-scope. What the code did was attempt to steal additional login tokens for npm from these other users, by sending them to a remote server. This, if successful, would have allowed them to hijack other packages and compromise more users.
The malicious package was posted at 3.40am PST. By 5am PST npm users had spotted the issue and the remote server was taken down by 5.30am, preventing further collection of credentials. By 7.30am npm had measures in place to prevent any compromised packages being published. We then took the important step of invalidating all login tokens for npm created before 7.30am; this means that any tokens stolen by the attacker between 3.40am and 7.30am would be useless, and the hole was closed.
To the best of our knowledge, no other packages were successfully hijacked during the window or since. Part of the reason for this is that npm 2FA features include a “2fa on publish” feature that means that even if you successfully steal a login token, you would still need physical possession of the victim’s phone or other device to be able to successfully publish a hijacked package. Authors of the most popular packages on npm have enabled this feature, preventing the attack from spreading. ESLint’s case was a rare weak link.
Given the window for a successful attack was about 2 hours, and the number of logged-in users who would have downloaded the package in that time, the total potential exposure was ~500 users, and the actual exposure so far appears to have been zero people other than the original victim. We are still investigating to confirm this is the case.
JAXenter: How do incidents like this get identified?
npm 2FA features include a “2fa on publish” feature that means that even if you successfully steal a login token, you would still need physical possession of the victim’s phone or other device to be able to successfully publish a hijacked package.
Laurie Voss: npm’s enormous user base of 10 million developers creates an extremely deep pool of experts capable of identifying malicious or unexpected behavior. While npm has numerous automatic mechanisms for detecting security flaws, humans are often the first people to report a problem. That was the case this time; our first user notification of an issue arrived at 4.56am. Had a larger chain reaction started npm’s automated systems would have caught it, but humans caught it on the very first link in the chain.
JAXenter: How long does that usually take?
Laurie Voss: Security incidents like this are pretty rare, so it’s hard to say “usually”, but previous incidents have been cleaned up in a similar time frame: 2-3 hours after the initial problem for emergency measures, and then a 6-7 hour period of cleanup and system hardening after that.
JAXenter: npm worked quite a bit on security for the last releases. Do the latest additions to npm security have an effect on this kind of incidents?
Laurie Voss: Absolutely. By 7.30am a security advisory had been inserted into npm’s database about the package, so anyone using npm version 6 or later was notified that there was a critical security issue. But npm has a lot more work in the pipeline to further harden our systems and prevent this kind of attack from happening in the first place, not just rapidly notifying people.
JAXenter: Blog posts on the incident both by npm and ESLint show that there has been a data breach elsewhere, exposing login credentials that were also used for npm by that user. Another theory of how this incident happened is that there might be other npm packages infected by the same kind of malware, causing access tokens for ESLInt to be exposed in the first place. What is your take on that theory?
Laurie Voss: We agree with ESLint that the ESLint organization member was the “patient zero” in this attack. We’re not relying on that conclusion, though: we and our partners have been conducting deep scans of every one of the 750,000 packages in the registry to ensure this pattern of malicious code is not present anywhere else. So far there’s been no sign of it but scans are ongoing.
JAXenter: Is there any npm feature users can utilize now if they want to make sure this kind of malware isn’t hidden in some dependency of their projects?
Laurie Voss: The easiest and most obvious step is to upgrade to npm 6 if they haven’t already done so; this will immediately notify users of any known vulnerabilities. The “npm audit” command does a full scan of your dependency tree to make sure there are no issues. There are additional npm configuration steps that can be taken to protect more paranoid users; the “ignore-scripts” configuration option disables the behavior that allowed the malicious code to run on install. However, that comes with trade-offs: lots of useful configuration work is often done by packages at install time, so disabling it could lead to a lot of inconvenience. Developers are free to make their own trade-offs.
SEE ALSO: How well do you know your npm trivia?
JAXenter: Your blog post recommended that developers should use two-factor-authentication to protect themselves from this kind of incident. How do you set that up with npm and is there any other advice you can give users to protect themselves from this kind of incidents?
Laurie Voss: The easiest way is to log in to npmjs.com and follow these quick instructions. The process takes about 30 seconds, and if you’ve ever used two-factor-auth on some other website the steps will be very familiar. We recommend the highest security setting here, requiring a one-time password both to log in and every time you publish a package. In terms of other steps, there’s not a lot that normal npm users need to do; this attack targets package authors rather than people who install packages.