Early on Sunday, March 24th, hackers activated ransomware to attack a securities brokerage company, causing the collapse of the system serving investors. I was in Hanoi. Being connected, I arrived at the scene out of curiosity, only to find myself staying for over a week. Ransomware is a type of computer malware that, when activated, encrypts all data.
The encryption algorithm makes it possible to decrypt only with the secret key. Such an incident requires addressing two main tasks: urgent restoration of business operations and ensuring security to prevent further attacks. Like wings, these two tasks require balance; the system cannot be restored if it's not secure, but it also needs to be reopened as soon as possible. We joked, saying we had to be "urgently patient." As I was about to leave Vietnam, my team didn't have enough time to handle the entire incident. I proposed hiring another domestic cybersecurity company to investigate and review security. I called upon two energetic engineers from Saigon to fly over and collaborate with the team in Hanoi. Immediately, all of us started working on restoring the system. The process consisted of three steps: Isolation, elimination, and restoration. Isolating the suspected compromised systems by controlling network traffic, ensuring no external communication.
Next, reviewing to eliminate, minimize risks, ensuring a clean and secure system. Finally, rebuilding the application software system for recovery. Having been a cryptography engineer at Google, decryption is my favorite task, but this time it was unexpectedly challenging. The large amount of data made decryption time-consuming and error-prone. We found some small improvements, but it wasn't as fast as expected. Moreover, our initial enthusiasm was hindered by lack of sleep; there were moments of confusion. Instead of sleeping, the team tried to come up with solutions in a state of being "half awake, half dreaming." As a result, the decryption process was slow, with many errors, some cases being inexplicable. Decryption wasn't something to be taken lightly, but the lack of explanation worried me tenfold. The "engineer" in me felt very uneasy; even if we couldn't do it, we had to understand why. The solution I chose was to return to the basics. If decryption was sometimes successful and sometimes not, we first had to understand how the malware operates. The team had started analyzing the malware from day one but got distracted and couldn't focus. This time, we gathered the most experienced engineers, determined to finish the job. From Hanoi, connecting with Saigon, Houston, and with malware expert Chương Đồng in Atlanta, the Vietnamese-American engineer team finally understood how the malware operates, when encryption fails, and how to fix it.
