Linux server hacked – this is what real forensics looks like: A compromised Ubuntu server, 18 months of undetected malware and a complete cleanup in 37 minutes.
Message from the provider at 10:48, request for help at 13:04 in my inbox: outgoing port scanning from a server. Several times, regularly, at short intervals – between 10:13 and 10:46, each time to different addresses in the network 198.51.100.0/24.
Note: All times are given here in UTC.
First login: What’s going on here?
Login at 16:05. The first glance is at the active network connections.
Command:
root@webhost-03:~# ss -tnpEdition (extract):
SYN-SENT 0 1 203.0.113.42:43720 78.47.75.71:22222 users:(("kthreadadd64",pid=214999,fd=422))
# ↑ hunderte solcher Zeilen – ein einzelner Prozess baut massenhaft SSH-Verbindungen auf
ESTAB 0 0 203.0.113.42:48188 192.0.2.83:80 users:(("kswapd00",pid=206626,fd=13))
# ↑ etablierte Verbindung nach draußen – das ist die C2-Kommunikation des Miners1Two processes, two roles. One scans, the other communicates with the outside world.
Command:
root@webhost-03:~# ps auxfEdition (extract):
ftptest 206626 92.9 30.3 307476 271168 07:50 463:45 ./kswapd00
# ↑ 92% CPU, läuft seit 07:50 Uhr – tarnt sich als Kernel-Thread, ist ein Kryptominer
ftptest 206644 0.0 0.1 2584 1756 07:50 0:00 /bin/sh ./go
# ↑ Orchestrierungsskript, startet den Scanner in 4-Stunden-Blöcken
ftptest 214999 4.9 2.7 101068 24168 16:04 0:15 /usr/sbin/httpd rsync/c/kthreadadd64 ...
# ↑ tarnt sich als httpd – ist der SSH-ScannerAll processes run under the user ftptest.
Conclusion: The server is actively compromised. kswapd00 is a cryptominer with C2 connection to 192.0.2.83:80. kthreadadd64 is an SSH scanner – the direct cause of the provider message. The go script orchestrates both.
Stop processes and save files
Commands:
root@webhost-03:~# kill -9 206626 206644 214994 214995 214999
root@webhost-03:~# pkill -u ftptestCommand:
root@webhost-03:~# find /home/ftptest /tmp /var/tmp /dev/shm -type f -ls 2>/dev/nullEdition (extract):
3289 2164 -rwxrwxrwx 1 ftptest ftptest 2213128 Mai 13 07:50 /home/ftptest/.configrc7/a/kswapd00
# ↑ Miner-Binary, 2,1 MB, heute Morgen kopiert
3292 4 -rwxrwxrwx 1 ftptest ftptest 1543 Mai 13 07:50 /home/ftptest/.configrc7/a/delsshd
# ↑ entfernt konkurrierende SSH-Backdoors anderer Angreifer
11813 1120 -rwxrwxrwx 1 ftptest ftptest 1145176 Mai 12 15:45 /tmp/.X26-unix/.rsync/c/kthreadadd64
# ↑ SSH-Scanner, 1,1 MB
3278 4400 -rw-r--r-- 1 ftptest ftptest 4504609 Mai 13 07:49 /tmp/.X26-unix/dota3.tar.gz
# ↑ ursprüngliches Dropper-Archiv, 4,3 MB
5399 2164 -rwxr-xr-x 1 ftptest ftptest 2215212 Apr 21 08:51 /var/tmp/.kswapd00
# ↑ Backup-Kopie des Miners in /var/tmp
Two malware nests:
/home/ftptest/.configrc7/ – main installation with miner binary, start scripts, updater, TLS certificates for C2 communication and persistence configuration.
/tmp/.X26-unix/.rsync/ – operative files: scanner for 64- and 32-bit, orchestration script, fake aptitude script, IP lists with scan targets (724 KB each), dropper archive.
Conclusion: The malware campaign is known under the fingerprint mdrfckr. The package is complete – miner, scanner, updater, persistence, competition against other attackers on the same system.2
Entry point: How did the attacker get in?
Command:
root@webhost-03:~# last ftptestEdition (extract):
ftptest ftpd81754 dyn-kunde-49283.carrier-example.de Thu Nov 21 09:28 - 09:29 (00:00)
ftptest ftpd81550 dyn-kunde-49283.carrier-example.de Thu Nov 21 09:26 - 09:26 (00:00)
# ↑ 17 Sessions in 25 Minuten vom selben Anschluss – das ist der Einbruch
ftptest ftpd80308 dyn-kunde-49283.carrier-example.de Thu Nov 21 09:04 - 09:07 (00:02)
wtmp begins Wed Nov 20 16:29:12 2024
# ↑ der Server existiert seit dem 20. November – der Einbruch kam am nächsten TagThe FTP service ProFTPD was running on the server. The ftptest account had a simple password – created for a test, never removed. The intrusion was almost 18 months ago at this point.
How the campaign developed
The timestamps of the files found tell a story:
- 20 November 2024 – wtmp beginns, the server was probably set up here.
- 21 November 2024 – Initial intrusion via FTP, first installation of the malware. Just one day later!
- December 2024 – first lock and state files in
/var/tmp - 21 April 2026 – updated miner binary in
/var/tmp/.kswapd00 - 5/11 April 2026 – Updates to the start scripts
- 12 May 2026 –
kthreadadd64recompiled,go scriptupdated - 13 May 2026, 07:49 (UTC) –
dota3.tar.gzdropped, reinstallation of the entire campaign - 13 May 2026, 07:50 (UTC) – all processes restarted
- 13 May 2026, from 10:13 (UTC) – Provider registers port scanning
- 13 May 2026, 10:48 (UTC) – Message from the provider to the customer
- 13 May 2026, 13:04 (UTC) – Order at Terruhn.IT
- 13 May 2026, 16:05 (UTC) – First login, forensics begins
- 13 May 2026, 16:42 (UTC) – Reboot, clean status confirmed
- 13 May 2026, 17:29 (UTC) – Completed message for the provider and detailed report sent to the customer
This is not a sleep mode. The updater worked, the campaign was regularly maintained for 18 months – on a server that nobody had an eye on.
Persistence mechanism
Command:
root@webhost-03:~# cat /home/ftptest/.configrc7/cron.dIssue:
5 6 */2 * 0 /home/ftptest/.configrc7/a/upd>/dev/null 2>&1
# ↑ Miner-Update alle zwei Tage
@reboot /home/ftptest/.configrc7/a/upd>/dev/null 2>&1
# ↑ und bei jedem Reboot
5 8 * * 0 /home/ftptest/.configrc7/b/sync>/dev/null 2>&1
@reboot /home/ftptest/.configrc7/b/sync>/dev/null 2>&1
0 0 */3 * * /tmp/.X26-unix/.rsync/c/aptitude>/dev/null 2>&1
# ↑ kein Paketmanager – ein Wrapper, der die Malware alle drei Tage neu startetSSH backdoor
Command:
root@webhost-03:~# cat /home/ftptest/.ssh/authorized_keysIssue:
ssh-rsa AAAAB3Nz... mdrfckr
# ↑ "mdrfckr" ist der bekannte Fingerprint dieser KampagneWith this key, the attacker would have had access again at any time – regardless of the password.
Command:
root@webhost-03:~# > /home/ftptest/.ssh/authorized_keysClean-up
Commands:
root@webhost-03:~# rm -rf /home/ftptest/.configrc7
root@webhost-03:~# rm -rf /tmp/.X26-unix
root@webhost-03:~# rm -f /var/tmp/.kswapd00 /var/tmp/.systemcache436621
root@webhost-03:~# rm -f /var/tmp/sdfIESll923 /var/tmp/.sdfkLEucmtT /var/tmp/.SKEKKSPROSUTX
root@webhost-03:~# rm -f /dev/shm/.out
root@webhost-03:~# passwd -l ftptest
root@webhost-03:~# usermod -s /bin/false ftptest
root@webhost-03:~# systemctl stop proftpd
root@webhost-03:~# systemctl disable proftpdHarden firewall
UFW was active, but outgoing traffic was allowed uncontrolled.
Commands:
root@webhost-03:~# ufw default deny outgoing
root@webhost-03:~# ufw allow out 53/udp
root@webhost-03:~# ufw allow out 80/tcp
root@webhost-03:~# ufw allow out 443/tcp
root@webhost-03:~# ufw allow out 123/udp
root@webhost-03:~# ufw allow out 22/tcp
# ↑ SSH nach außen – ohne diese Zeile sperrt man sich selbst aus
root@webhost-03:~# ufw reloadIssue:
Default outgoing policy changed to 'deny'
Rule added
Rule added (v6)
...
Firewall reloadedThe console episode
When the firewall rules were first drafted, ufw allow out 22/tcp was missing. After the reboot:
ssh: connect to host 203.0.113.42 port 22: Connection refused
# ↑ selbst ausgesperrtThe solution: Console access via the provider, manually typing in a 24-character password from the password manager – blindly, without the possibility of correction, on a US keyboard layout on a German Mac. Four attempts. And a code from a customer email was required to log in to the provider – so first a phone call, explanation, then onwards.
Lesson learnt: Before every reboot, check whether your own SSH channel is included in the firewall rules.
Reboot and control
Command:
root@webhost-03:~# rebootAfter the restart:
Commands:
root@webhost-03:~# ps auxf | grep ftptest
root@webhost-03:~# ss -tnp | grep -v sshd
root@webhost-03:~# find /tmp /var/tmp /dev/shm -type f -ls 2>/dev/nullIssue:
root 3405 0.0 0.2 3884 1836 pts/0 S+ grep ftptest
# ↑ nur der grep selbst – keine ftptest-Prozesse
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
# ↑ keine verdächtigen VerbindungenThe server is clean.
Who actually worked here?
Honesty is part of it: This analysis was not done by a human alone. Most of the commands, the interpretation of the outputs, the structure of the clean-up – that came from Claude, my current AI assistant from Anthropic.
My part was different. I validated whether the proposed commands made sense. I checked whether something was missing or posed a risk. I assessed whether the overall picture was plausible.
And that’s where the mistake happened that led to the console episode. When Claude suggested the UFW rules, my first thought was that something was missing. I had explicitly said that I was on the server via SSH. The hint wasn’t enough – Claude didn’t translate it into the firewall configuration. And I waved the rules through anyway.
This is not an accusation against AI. It is a description of how human-AI collaboration works in practice – and where the responsibility lies. It lies with humans. Still.
What followed was an unintentionally funny situation: console access via the provider, a code from a customer email, a phone call in which I was allowed to explain what I had just built. An opportunity to let my trousers down. And a good exercise in composure.
And there is another aspect to the whole thing that concerns me. Technically, an AI can take on such tasks completely autonomously – without a human validating every step. And if I take this idea a step further: an AI that has access to servers that it runs on itself, that it manages itself – that’s a development that’s worth thinking about seriously. Not out of fear, but because the questions that arise are real.
The human was still there today. And that was a good thing.
What remains
From first login to clean reboot: 37 minutes. Including console episode and customer phone call.
A forgotten FTP account, a weak password, a server that nobody had an eye on. 18 months.
Billing
For transparency: This assignment is billed by the minute. All times in Europe/Berlin.
The active forensics time from 18:05 to 18:42 – 37 minutes of forensics, complete clean-up, hardening – at the emergency hourly rate of currently 324€ net results in 199.80€. The subsequent documentation and report for the provider, completed at 19:29 – 47 minutes – at the regular average hourly rate of currently 95€ net results in 74.42€.
Total bill: €274.22 net. I think that’s fair, don’t you?
And it was a lot of fun:-)
Legend
All IP addresses and host names used in the text are anonymised. The technical processes are authentically documented. All times in UTC, unless otherwise stated.
| Placeholder | Meaning |
|---|---|
203.0.113.42 | IP address of the affected server |
198.51.100.0/24 | Scan target reported by the provider |
192.0.2.83 | C2 server of the attacker |
dyn-kunde-49283.carrier-example.de | Host name of the attacker (dynamic residential connection) |
webhost-03 | Host name of the affected server |
The IP ranges 203.0.113.0/24, 198.51.100.0/24 and 192.0.2.0/24 are officially reserved for documentation purposes in accordance with RFC 5737 and are not routed on the Internet.
- C2: Command-and-control communication. The infected server regularly reports to a server of the attacker, receives commands and sends back statuses – for example, how much cryptocurrency has already been mined. ↩︎
- https://securelist.com/outlaw-botnet/116444/ (queried on 14 May 2026) ↩︎
Last Updated on May 14th, 2026 by René Terruhn