Hey Pixelers,

Oh dear LastPass,

It is not my goal to focus on sensualist or dramatic topics, but this is something we have to talk about.
The implications of the LastPass breach could, and should, represent a learning lesson for all devops and software engineers.
I think, therefore, it’s good to discuss what actually happened, how it could have been avoided and what the future should look like.
Perhaps we have overindulged in and trusted software as a service offerings too greatly.

I want to emphasise that the purpose of this video is not to mock the actions of the LastPass DevOps engineer, as while some blatant lapses of judgement were made, we are all vulnerable to making mistakes.

This is an example of a “Persistent Attack”, which means the threat-actors increased their foothold in stages without rushing the process.
Therefore, it is important to look at this attack as a long series of events, one by one.

# What Is Believed To Have Happened

I have strung together a series of events, which aligns with the current theory of how this breach occurred, but expect there to be alternative explanations.
I have tried to compile sources for all the information, however I am unable to confirm all the specific details of the attack.
There is a lot of agencies reporting facts, which are being reported to them personally though a source in or close to LastPass. Without any official statement from LastPass confirming them.

# May 7th, 2020

The Plex Security Team discovered a major security flaw, this allowed an actor with access to a servers Plex account to upload malicious files to the Camera Upload feature.



# June 21st, 2022

Interestingly, the attack has not occurred yet, however, a LastPass spokesman provided the following statement:

“Any LastPass user that created an account after Sept. 16 or had deleted their account prior to June 21, would not have had their vault data taken”

Source: LastPass via CyberSecurityDive.com

I assume this is because LastPass keeps user data for 30 days, in-case of accidental deletion.
However, if you deleted your account on June 21, then it should be wiped from LastPass by July 21.
This is 18 days before the apparent attack took place.



# August 8th, 2022

This is the beginning of the first attack.

A threat actor compromised a LastPass software engineer’s corporate laptop to gain access to a cloud-based development environment. The adversary stole source code, proprietary technical documentation and some of the company’s internal system secrets.
The threat actor used technical documentation and source code to exfiltrate 14 of approximately 200 source-code repositories related to components of the LastPass service.
The source-code repositories included cleartext embedded credentials, stored digital certificates for the company’s development infrastructure and encrypted credentials used for production.

Source: LastPass via CyberSecurityDive.com

The software engineer was one of LastPass’ most senior DevOps engineers.

This was accomplished by targeting the DevOps engineer’s home computer and exploiting a vulnerable third-party media software package, which enabled remote code execution capability and allowed the threat actor to implant keylogger malware

Source: LastPass via Arstechnica.com

ArsTechnica reports that the “third-party media software package” was disclosed to them as Plex.
It is unknown if the aforementioned Plex vulnerability was the cause of the attack, however, a comment by LastPass states:

The threat actor exploited a vulnerability in an earlier, unpatched version of Plex Media Server on a LastPass DevOps engineer’s home computer. We have reached out to Plex Media Server to inform them.

Source: LastPass via PCMag.com

Presumably, the DevOps engineer had exposed his Plex Media Server publicly on the internet, punching a hole through his firewall.
Utilising the exploit, a keylogger was installed on this machine, to scrape the engineers master password and multi-factor authentication codes for the companies LastPass corporate vault.

Even though LastPass mentions a corporate laptop being compromised, several times, it is implied that the machine that ran Plex, was the same machine that the keylogger was installed on.
This means that the attacker was capable of moving laterally onto the work laptop, the DevOps engineer used the same personal machine for work purposes or that Plex was installed on the work machine.



# August 12th, 2022

The LastPass security team was alerted to the malicious activity. The company refers to this as the “first incident,” which was immediately followed by a “second incident” the company says began Aug. 12.
In the follow-on compromise, the threat actor used information exfiltrated from the initial breach to initiate a more widespread and damaging attack.

Source: LastPass via CyberSecurityDive.com


# August 13th, 2022

LastPass engaged with incident response firm Mandiant.

Source: LastPass via CyberSecurityDive.com


# August 14th, 2022

The threat actor copied a backup of LastPass’ customer database containing unencrypted account information, related metadata and application configuration options such as multifactor authentication.

Source: LastPass via CyberSecurityDive.com


# August 24th, 2022

Now, this may be a coincidence but, on August 24th 2022, Plex reported a major network intrusion.
In the email by Plex, it is reported that peoples data was leaked, this included: emails, usernames and encrypted passwords.

Yesterday, we discovered suspicious activity on one of our databases.
We immediately began an investigation, and it does appear that a third-party was able to access a limited subset of data that includes emails, usernames and encrypted passwords.

Source: Email From Plex

The Email later clarifies that the passwords were hashed with bcrypt and I can only assume properly salted.
As far as I can understand, Plex did not report that any other information was leaked, however, in a 2015 compromise of Plex, it is noted that IP addresses were also leaked.

I am therefore, going to assume the full set of data leaked was: emails, usernames, bcrypt hashed passwords and ip addresses.
It could be theorised, that this attack started a few weeks earlier, and might explain how the DevOps engineers machine may have initially been compromised.
You can imagine that the following information would be sufficient for an attack to happen:

  1. A username from Plex
  2. An IP from Plex
  3. Password from another unknown source (Maybe another credential dump)
  4. A possibly exposed Plex installation running on the DevOps engineer’s machine
  5. A Plex installation with a critical vulnerability for logged in administrators


# August 25th, 2022

The first public announcement is made by LastPass, implying the whole thing is under-control and that there is nothing to be concerned about.

To All LastPass Customers,
Two weeks ago, we detected some unusual activity within portions of the LastPass development environment. After initiating an immediate investigation, we have seen no evidence that this incident involved any access to customer data or encrypted password vaults.
We have determined that an unauthorized party gained access to portions of the LastPass development environment through a single compromised developer account and took portions of source code and some proprietary LastPass technical information. Our products and services are operating normally.

Source: LastPass

Has any data within my vault or my users’ vaults been compromised?
No. This incident occurred in our development environment. Our investigation has shown no evidence of any unauthorized access to encrypted vault data. Our zero knowledge model ensures that only the customer has access to decrypt vault data.

Source: LastPass

Has any of my personal information or the personal information of my users been compromised?
No. Our investigation has shown no evidence of any unauthorized access to customer data in our production environment.

Source: LastPass


# September 8th, 2022

The threat actor started to copy five binary large objects database shards.
The backups were dated: Aug. 20, Aug. 30, Aug. 31, Sept. 8 and Sept. 16. The exfiltration of database backups occurred between Sept. 8 and Sept. 22.

Source: LastPass via CyberSecurityDive.com


# September 15th, 2022

On this date LastPass assured their customers that the situation was under-control and that no further concerns were found.

To All LastPass Customers,
We have completed the investigation and forensics process in partnership with Mandiant. Our investigation revealed that the threat actor’s activity was limited to a four-day period in August 2022. During this timeframe, the LastPass security team detected the threat actor’s activity and then contained the incident. There is no evidence of any threat actor activity beyond the established timeline. We can also confirm that there is no evidence that this incident involved any access to customer data or encrypted password vaults.
Our investigation determined that the threat actor gained access to the Development environment using a developer’s compromised endpoint. While the method used for the initial endpoint compromise is inconclusive, the threat actor utilized their persistent access to impersonate the developer once the developer had successfully authenticated using multi-factor authentication.

the LastPass Development environment is physically separated from, and has no direct connectivity to, our Production environment. Secondly the Development environment does not contain any customer data or encrypted vaults. Thirdly, LastPass does not have any access to the master passwords of our customers’ vaults - without the master password, it is not possible for anyone other than the owner of a vault to decrypt vault data as part of our Zero Knowledge security model.

Source: LastPass

The information that LastPass could gather though from the attacker was limited:

Third-party VPN services allowed the threat actor to obscure their location, impersonate the software engineer and access and maintain a dedicated connection to the cloud-based development environment via corporate VPN.
LastPass describes this as a “tailgate” approach that relied on the software engineer’s successful authentication with domain credentials and MFA.
“No privilege escalation was identified or required,” the company said in its incident report.
The threat actor also performed anti-forensic activity, and an operating system upgrade on the software engineer’s corporate laptop scheduled during the four-day period overwrote logs and system artifacts.
The initial threat vector that the adversary used to gain access to the software engineer’s machine remains unknown, according to LastPass.

Source: LastPass via CyberSecurityDive.com


# October 26th, 2022

This is supposedly the last date that we know the threat actor was operating inside the system.
I would hesitate taking this as fact, though, as the timeline and scope of this breach keeps growing in scale.

The threat actor, still active in LastPass systems, “engaged in a new series of reconnaissance, enumeration and exfiltration activities” involving the company’s AWS S3 storage buckets, a subsequent investigation found.
The threat actor operated undetected by LastPass for almost three months as part of the second incident, which LastPass said spanned from Aug. 12 to Oct. 26.
“We cannot confirm with certainty that it was one or more threat actors,” a LastPass spokesperson told Cybersecurity Dive.
“There were no further exfiltration activities after Sept. 22, 2022. Since Oct. 26, 2022, we have not seen any threat actor activity.”

Source: LastPass via CyberSecurityDive.com


# November 30th, 2022

The realisation of the scope of the breach is now fully realised, and an announcement is made:

To All LastPass Customers,
We recently detected unusual activity within a third-party cloud storage service, which is currently shared by both LastPass and its affiliate, GoTo. We immediately launched an investigation, engaged Mandiant, a leading security firm, and alerted law enforcement.
We have determined that an unauthorized party, using information obtained in the August 2022 incident, was able to gain access to certain elements of our customers’ information. Our customers’ passwords remain safely encrypted due to LastPass’s Zero Knowledge architecture.

Source: LastPass

The parent company GoTo then realised their first statement:

To All GoTo Customers,
I am writing to inform you that GoTo is investigating a security incident. While we are currently working to better understand the scope of the issue, we wanted to let you know about the situation and how we are responding.
Upon learning of the incident, we immediately launched an investigation, engaged Mandiant, a leading security firm, and alerted law enforcement. Based on the investigation to date, we have detected unusual activity within our development environment and third-party cloud storage service. The third-party cloud storage service is currently shared by both GoTo and its affiliate, LastPass.

Source: GoTo

The statement by goto was initially met with criticism, as they had flagged their post with ‘noindex’, so browsers would not show it in their search results.
According to TechCrunch, the noindex flag was removed, however this likely occurred over 2 weeks afterwards.



# December 22nd, 2022

Now the full bomb-shell is dropped:

To Our LastPass Community,
Based on our investigation to date, we have learned that an unknown threat actor accessed a cloud-based storage environment leveraging information obtained from the incident we previously disclosed in August of 2022. While no customer data was accessed during the August 2022 incident, some source code and technical information were stolen from our development environment and used to target another employee, obtaining credentials and keys which were used to access and decrypt some storage volumes within the cloud-based storage service.
LastPass production services currently operate from on-premises data centers with cloud-based storage used for various purposes such as storing backups and regional data residency requirements. The cloud storage service accessed by the threat actor is physically separate from our production environment.
To date, we have determined that once the cloud storage access key and dual storage container decryption keys were obtained, the threat actor copied information from backup that contained basic customer account information and related metadata including company names, end-user names, billing addresses, email addresses, telephone numbers, and the IP addresses from which customers were accessing the LastPass service.
The threat actor was also able to copy a backup of customer vault data from the encrypted storage container which is stored in a proprietary binary format that contains both unencrypted data, such as website URLs, as well as fully-encrypted sensitive fields such as website usernames and passwords, secure notes, and form-filled data.

What Does This Mean? Is My Data at Risk?
The threat actor may attempt to use brute force to guess your master password and decrypt the copies of vault data they took.

Source: LastPass


# January 3rd, 2023

In Massachusetts, a class-action lawsuit has been brought up against LastPass due to the alleged involvement of the data-leak in a theft of $53,000 bitcoins.

At this point we know very little about the profile of the threat actor.
It seems to me that if the culprit was a state-actor, then they might be careful to only attack the individuals that are of-interest to them.
For example politicians and business relations that are connected to the events in Ukraine.
Targeting more individuals than necessary would risk triggering alarms, which could make your actual targets retreat from your radar.

If the threat actor was not a state actor, then you can imagine that it could be more profitable to sell huge segments of the data to other parties, than to crack it.
Breaking into 33 million accounts will take an enormous amount of time.
Likely instead of blind bruteforce, they will use their local-dump as a way of avoiding any sort of soft-lockout or 2fa, and attempt each emails known leaked credentials and common password combinations.
However, do not feel too comfortable as it is not impossible to brute-force a vault.

a current NVIDIA GeForce RTX 4090 graphics card could test more than 88,000 guesses per second!

we are already down to on average ‘merely’ 860 thousand years to guess a baseline LastPass password.

that’s merely a single graphics card …, it will be faster if someone is willing to throw more/better hardware at the problem.

Source: https://palant.info/2022/12/23/lastpass-has-been-breached-what-now/

The reason why I say this, is I am sceptical whether this bitcoin heist is related to the LastPass, as we have not heard of any other breaches that could be related yet.
At the moment though, it is not even known if the attack was conducted by a single actor, since the system was vulnerable for around 3 months.



# January 23rd, 2023

The compromise of LastPass is reported to have spread to the rest of the GoTo ecosystem.

To All GoTo Customers,
Our investigation to date has determined that a threat actor exfiltrated encrypted backups from a third-party cloud storage service related to the following products: Central, Pro, join.me, Hamachi, and RemotelyAnywhere.
We also have evidence that a threat actor exfiltrated an encryption key for a portion of the encrypted backups.
The affected information, which varies by product, may include account usernames, salted and hashed passwords, a portion of Multi-Factor Authentication (MFA) settings, as well as some product settings and licensing information.
In addition, while Rescue and GoToMyPC encrypted databases were not exfiltrated, MFA settings of a small subset of their customers were impacted.

Source: GoTo

It is reported that this happened because these GoTo services shared the same backup storage vault as LastPass.



# February 27th, 2023

The source of the second breach is finally linked to the initial breach.

This attack targeted LastPass infrastructure, resources, and an employee in a campaign of overlapping activity. The observed tactics, techniques, and procedures (TTPs), as well as the indicators of compromise (IOCs) of the second incident were not consistent with those of the first. While proximal in terms of timeline, it was not initially obvious that the two incidents were directly related.
Our investigation has revealed that the threat actor pivoted from the first incident, which ended on August 12, 2022, but was actively engaged in a new series of reconnaissance, enumeration, and exfiltration activities aligned to the cloud storage environment spanning from August 12, 2022 to October 26, 2022.
The second incident saw the threat actor quickly make use of information exfiltrated during the first incident, prior to the reset completed by our teams, to enumerate and ultimately exfiltrate data from the cloud storage resources.
Alerting and logging was enabled during these events, but did not immediately indicate the anomalous behavior that became clearer in retrospect during the investigation. Specifically, the threat actor was able to leverage valid credentials stolen from a senior DevOps engineer to access a shared cloud-storage environment, which initially made it difficult for investigators to differentiate between threat actor activity and ongoing legitimate activity. Ultimately AWS GuardDuty Alerts informed us of anomalous behavior as the threat actor attempted to use Cloud Identity and Access Management (IAM) roles to perform unauthorized activity.
To access the cloud-based storage resources - notably S3 buckets which are protected with either AWS S3-SSE encryption, AWS S3-KMS encryption, or AWS S3-SSE-C encryption - the threat actor needed to obtain AWS Access Keys and the LastPass-generated decryption keys. The encrypted cloud-based storage services house backups of LastPass customer and encrypted vault data.
As mentioned in the first incident summary, certain LastPass credentials stolen during the first attack were encrypted and the threat actor did not have access to the decryption keys, which could only be retrieved from two locations:
1: A segregated and secured implementation of an orchestration platform and key-value store used to coordinate backups of LastPass development and production environments with various cloud-based storage resources, or
2: A highly restricted set of shared folders in a LastPass password manager vault that are used by DevOps engineers to perform administrative duties in these environments.
Due to the security controls protecting and securing the on-premises data center installations of LastPass production, the threat actor targeted one of the four DevOps engineers who had access to the decryption keys needed to access the cloud storage service.
This was accomplished by targeting the DevOps engineer’s home computer and exploiting a vulnerable third-party media software package, which enabled remote code execution capability and allowed the threat actor to implant keylogger malware. The threat actor was able to capture the employee’s master password as it was entered, after the employee authenticated with MFA, and gain access to the DevOps engineer’s LastPass corporate vault.
The threat actor then exported the native corporate vault entries and content of shared folders, which contained encrypted secure notes with access and decryption keys needed to access the AWS S3 LastPass production backups, other cloud-based storage resources, and some related critical database backups.

Source: LastPass

It seems like LastPass is claiming that this was a targeted attack, the threat actors knew exactly who they needed to compromise, and they managed to find a way in.
Whether they knew this before they broke into this DevOps engineers machine, or afterwards, is unknown.
However, the implication is that this is a sophisticated attack, conducted by a party with a specific goal in mind.
Can we accept that it is a coincidence, that the targeted DevOps engineer was only one of four with the specific access required for this attack to be conducted?



# March 1st, 2023

The curtain is pulled back, as we can now see the full scope of the breach.

As detailed in the incident summaries, the threat actor stole both LastPass proprietary data and customer data, including the following:
Summary of data accessed in Incident 1:
1: On-demand, cloud-based development and source code repositories - this included 14 of 200 software repositories.
2: Internal scripts from the repositories - these contained LastPass secrets and certificates.
3: Internal documentation - technical information that described how the development environment operated.
Summary of data accessed in Incident 2:
DevOps Secrets - restricted secrets that were used to gain access to our cloud-based backup storage.
Cloud-based backup storage - contained configuration data, API secrets, third-party integration secrets, customer metadata, and backups of all customer vault data. All sensitive customer vault data, other than URLs, file paths to installed LastPass Windows or macOS software, and certain use cases involving email addresses, were encrypted using our Zero knowledge model and can only be decrypted with a unique encryption key derived from each user’s master password. As a reminder, end user master passwords are never known to LastPass and are not stored or maintained by LastPass - therefore, they were not included in the exfiltrated data.
Backup of LastPass MFA/Federation Database - contained copies of LastPass Authenticator seeds, telephone numbers used for the MFA backup option (if enabled), as well as a split knowledge component (the K2 ‘key’) used for LastPass federation (if enabled). This database was encrypted, but the separately-stored decryption key was included in the secrets stolen by the threat actor during the second incident.

Source: LastPass

An important note is the following:

We have shared technical information, Indicators of Compromise (IOCs), and threat actor tactics, techniques, and procedures (TTPs) with law enforcement and our threat intelligence and forensic partners. To date, however, the identity of the threat actor and their motivation remains unknown. There has been no contact or demands made, and there has been no detected credible underground activity indicating that the threat actor is actively engaged in marketing or selling any information obtained during either incident.

Source: LastPass


# How Could This Have Been Avoided

There is a lot of unknown details, and this attack does seem quite sophisticated.
If this threat-actor is well-funded and connected, it is not unreasonable to think that if they couldn’t have broken-through with through other-means.

From the perspective of the clients of password managers, it does bring into question on whether such a tool should be a cloud service at all.
If the mechanism to upload code changes to production was infiltrated as-well, then it would have been additionally possible to have injected a hook into the web-form which extracts peoples master-passwords too.

With password managers, such as KeePass, Bitwarden or Vaultwarden the individuals credentials can be kept within their own ecosystem, without being exposed on the public internet.
Centralising all of our credential management into a few entities, is only going to make those few entities very high risk.
It brings into question whether we have become too comfortable offloading concerns to third-party software-as-a-service providers.

# What Are The Issues

There are two major issues I have with the report from LastPass:

  1. How did the attacker jump from the DevOps personal machine, to the corporate laptop.
  2. How did the attacker jump from the dev environment, to the production backup vault.

I am not going to blame the DevOps engineer for having an insecure Plex instance on his personal machine. It is dumb.
However, it is hard to ensure that every network that someone connects their machine to is guaranteed to be safe.
According to the report they seem to have no idea what happened on the corporate laptop because a system update wiped the logs.
Personally, I get the impression that the DevOps engineer was somehow able to connect to the VPN and dev environment from his personal machine, and this was the machine that compromised the system.

For the second issue, I can only assume that the production backup vault, was a shared storage vault for all environments, for LastPass and multiple services by GoTo.
Meaning the dev environment had access to the production backup vault.

There are two big assumptions, but they present two major security shortcomings of LastPass’ system.
It is worth nothing though, that it is entirely possible that exploit methods were used that we are still not fully aware of, which may also explain these events.

# What Could Be Done

# Proper Company Education

It is said that after the breach that LastPass helped the engineer secure his home environment.
Perhaps, especially as a security company, this is something that should be done before the individual gains privileged access?

# Never Open Ports To Your Home Network

I would only ever open a port if it was for a secure VPN connection. Nothing else needs to be available on the public internet.
Some modern VPN’s even utilise NAT traversal in order to connect nodes together without an open port.
If Plex was not available on the open internet, then the attackers would not have had any attack vector to exploit.

# Segment Your Home Network

As an engineer with privileged access, it is important to have a proper work environment, even at home.
By segmenting your home network, you can ensure that compromised machines cannot easily hop to other systems.
I would usually segment the network into 3 sections:

  1. Untrusted, for IOT’s and guests.
  2. Personal Machines
  3. Work Machines

# Mandatory Locked-Down Corporate Laptops

I know corporate laptops often suck to work on, but as a company with sensitive data, I do not think you have a choice.
This means things like:

  1. Full Disk Encryption
  2. Restricted Administrator Control
  3. Firewall that blocks everything
  4. Mandated Anti-Virus software with strict policies
  5. Restrictive Group Policies, such as limiting the usage of removal media, limiting the ability to create hotspots
  6. Enforced software updates
  7. Strong Password Policies
  8. Restricted VPN access which ensures the machine can only connect if all the above comply with the system

Even if the engineer did some work-related programming on his personal machine, it should not have been possible to have used that as a launchpad to somehow jump onto the network.

# Hard Division Between Dev and Production

LastPass claims that there already was a strong division between dev and production. As far as I can see, the production environment itself was never breached.
However, there seems to be no division between dev and the production backup vault. This makes no sense to me.

# Break-Glass Accounts

Part of the issue was that the LastPass DevOps engineers needed access to decryption keys to perform maintenance tasks.
I am skeptical on whether that really is necessary. However, it is not hard to imagine a system where accessing such keys requires an audit process which also causes a system alarm that notifies engineers immediately.

I would be interested to hearing what recommendations you guys would make or if there are any updates on what exactly caused this incident in the first place.