Hilong Ecuador's production Zimbra mail server was owned. Cryptominer eating CPU, reverse shell in a persistence crontab, unauthorized SSH keys in root's authorized_keys, outbound traffic to command-and-control servers on obscure ports. The company found out because email throughput had degraded — the initial call was "our email is slow", the reality was much worse.

## Triage — the first two hours

Rule zero of IR: do not reboot. A reboot loses volatile evidence (running processes, network connections, mapped memory).

The first two hours were pure observation:

- `ps auxf` to enumerate every running process, with parent-child chains.
- `netstat -tunapl` to list every listening port and every established connection.
- `ss -tunp` cross-check.
- `crontab -l` for every user with a shell, `ls -la /etc/cron.*`.
- `find / -name authorized_keys 2>/dev/null` — audit every SSH trust.
- `last -30` and `wtmp` review for unexpected logins.
- `ausearch --start today` for auditd events.

Findings, in the order they surfaced:

1. A binary at `/tmp/.cache/kdevtmpfsi` pegging 380% CPU. Textbook Monero cryptominer.
2. A reverse shell attached to a listener on `91.x.x.x:4444` originating from a process named `kinsing`.
3. A crontab entry `* * * * * /tmp/.cache/kdevtmpfsi` under root, re-launching the miner if killed.
4. A root SSH key in `/root/.ssh/authorized_keys` matching no known engineer.
5. A modified `/etc/rc.local` re-adding the SSH key on every boot.

## The entry point

CVE-2024-45519, a critical Zimbra postjournal RCE disclosed in September 2024. The compromise timeline matched public exploit availability. The attacker had root within an hour of Zimbra being scanned.

## Eradication

Order of operations:

1. **Isolate the network at the switch level** — not at the server. Cutting the switch port prevents the attacker from noticing we are cleaning up and reacting.
2. **Snapshot the disk** — full raw image to an external drive for forensic preservation. This is the evidence bundle that later goes to the client's lawyer if needed.
3. **Kill the miner and reverse shell** — `kill -9` on the identified PIDs.
4. **Remove the persistence** — delete the malicious crontab entries, the modified `rc.local`, the unauthorized SSH keys.
5. **Rotate every credential** — root password, every service account, every SSH key, every API token stored on the server. Assume the attacker has all of them.
6. **Patch Zimbra to the CVE-fixed version** — but Zimbra's official patch was not yet available for the client's OS version.

## The custom Perl patch

Zimbra's postjournal handler is a Perl script that accepts a message body over a local Unix socket. The CVE was an input sanitization flaw that let an attacker inject shell metacharacters into a downstream `system()` call.

The official fix was pending. The client's mail server needed to be back online today, not next week.

I wrote a 40-line Perl patch that intercepts the postjournal input, strips shell metacharacters, rejects any input matching the CVE's exploit signature, and logs the attempt. It is not a proper fix — a proper fix goes into the codepath that does the `system()` call. But it closes the exploitable surface and it works.

The patch was applied, Zimbra was restarted, the network isolation was lifted, and mail flow resumed. Total downtime after eradication: 12 minutes.

The official Zimbra patch was applied 4 days later when it shipped. The custom Perl patch was removed at that point.

## Hardening

The server was owned once. The goal now is to make owning it a second time expensive.

- **fail2ban** on SSH, Zimbra web mail, IMAP, POP3. Any client that fails auth 5 times in 10 minutes gets a 24-hour ban.
- **firewalld** replacing the previous default iptables. Explicit allow rules for mail ports, deny everything else.
- **SSH key-only auth** — password login disabled entirely.
- **Root SSH disabled** — an attacker who steals a key still has to escalate.
- **auditd** rules for filesystem writes to `/etc`, `/var/spool/cron`, `/root/.ssh`.
- **Weekly SOC review** — a scheduled log review of auth failures, unusual processes, outbound connections. The client is on a small retainer for this.

## What the client got

- Mail server back online in the same day.
- Malware fully eradicated with a preserved forensic image.
- Every credential rotated.
- Weekly SOC monitoring for early detection of re-compromise.
- A written incident report describing the compromise, the actions taken, and the residual risks.

## What the client learned

Zimbra is a target. Any mail server with public IMAP/webmail exposure is a target. The right posture is: patch aggressively, monitor auth failures continuously, keep backups off-server, and assume the box will be owned eventually — plan for that.

The engagement continues as a monthly retainer.

Case study of the Hilong Ecuador incident: a production mail server actively compromised, malware eradicated (cryptominer + reverse shell + persistence crontab), CVE-2024-45519 mitigated with a custom Perl patch, fail2ban + firewalld hardening, weekly SOC monitoring.

Security

CVE-2024-45519 incident response — patching a live Zimbra mail server mid-compromise with a custom Perl fix