Daniel's got a whole thing this week about malware identification, and he opens with a strange observation. One of the most powerful forensic tools on the internet is free, and almost nobody uses it properly. He wants the landscape. The checksum-level databases, the sandbox forensics that services like VirusTotal will run on your behalf, and why he thinks they're wildly underused. Then he wants the lineage. How these databases evolved, how the open-source definition libraries actually feed into the protection and remediation tools people pay for, and what the closed, invite-only professional communities look like by comparison.
The phrase that sticks with me is "random APK downloaded from GitHub." That's not a hypothetical. That's Tuesday.
It really is. I had someone in my inbox last week asking whether a release binary from a repository with three stars and a single commit was safe to run. And my answer was, well, let's walk through what you can actually do to find out. Which is what Daniel's really asking about.
And his question is really about the gap between knowing a file is bad and knowing what a file does. Those are two different jobs, and most people only ever touch the first one.
Right. Identification versus remediation. VirusTotal and MalwareBazaar live in the first layer. Your endpoint protection, your EDR, your SIEM rules, those live in the second. And Daniel's instinct is correct that the first layer is criminally underused, because most people think of VirusTotal as a place where you paste a URL and get a green checkmark or a red X.
Which is like describing a hospital as a place where they take your temperature.
The thermometer is in the lobby. The MRI is down the hall and nobody goes down the hall.
And then they complain that the hospital didn't tell them anything useful. Well, you never left the waiting room. You looked at the vending machine and walked out.
So let's start with the thing everyone has heard of and almost nobody actually uses properly. VirusTotal.
The checksum model is the oldest trick in the book, and it's still the backbone of most detection. You take a file, you run it through a hash function. MD5, SHA one, SHA two fifty six. You get back a fixed-length string that is a fingerprint of that file's exact bytes. If that fingerprint matches a hash in a known-bad database, you have identification without ever executing the file. No sandbox, no behavioral analysis. Just math.
And the elegance is that it's fast. Hashing a file takes milliseconds. Running it in a sandbox takes minutes, and someone has to pay for the virtual machine.
The vulnerability is the exact-match problem. A single byte changes, the hash changes completely. So a malware author recompiles an APK with a new timestamp, or repacks an installer with a different compression setting, and suddenly it's a brand new file as far as the hash database is concerned. Clean slate.
This is why the hash database feels like a game of whack-a-mole if you're the one maintaining it. The malware author can iterate faster than you can collect.
And they do. There are malware families that release new builds daily, sometimes multiple times a day. Each build is a new hash. If you're only checking hashes, you're always behind.
Which is precisely why the sandbox layer exists. The hash tells you if you've seen this exact file before. The sandbox tells you what the file would do if you ran it, regardless of whether anyone has ever seen it.
And that's the tension Daniel's poking at. Hashes are trivially defeated, but they remain the fastest and cheapest first check. Most blocklists, most endpoint agents, most threat intel feeds, they're all hash-indexed at the bottom. The question is what you do when the hash comes back empty.
So walk me through the actual workflow. I have a random APK from a GitHub release page. I don't trust it. What do I do?
First step, you hash it locally. You don't need to upload anything. Just compute the SHA two fifty six on your own machine. Then you paste that hash into MalwareBazaar first.
Why MalwareBazaar before VirusTotal?
Because MalwareBazaar is a queryable repository keyed by hash, and querying by hash doesn't require uploading the file. You're just asking, has anyone else already submitted this exact file and what did they learn about it. If the hash is known, you get back the malware family, the first-seen date, tags, sometimes a YARA rule you can run locally. You've identified the file without ever sending it anywhere.
That's a meaningful privacy distinction. You're not giving them the file. You're giving them a fingerprint and asking if it's in the book.
And for a lot of people, that's the entire barrier. They don't want to upload a file they're not sure about. But pasting a hash, that feels safe. And it is safe, because a hash doesn't contain the file. It's a one-way function. You can't reconstruct the APK from the SHA two fifty six.
So the first step is essentially free, both computationally and in terms of privacy. And yet most people skip it and go straight to uploading, or they skip the whole thing entirely and just run the file and hope for the best.
The hope-for-the-best approach is shockingly common. People will run a random binary because the GitHub page had a nice README. A nice README is not a security control.
A nice README is marketing. And malware authors write nice READMEs too.
Some of the best documentation I've ever seen was attached to a credential stealer. The author had a whole wiki. Screenshots. A FAQ.
That's terrifying and also completely believable.
It was a professional operation. The malware was the product. The README was the sales page.
Okay, so I've hashed my file locally. I've queried MalwareBazaar. The hash comes back empty. Now what?
Then you escalate. You upload to VirusTotal, and now you're getting the full forensic treatment. Multi-engine verdict from sixty or seventy antivirus engines. A first-submission timestamp. Community comments. But the real value is not the aggregate score. The aggregate score is the least useful thing VirusTotal gives you.
Say more about that.
The Details tab is where the actual forensics live. You get the PE headers if it's a Windows executable. The embedded strings. Packer signatures. The import table. And then there's the relationships graph, which shows what domains the file contacted, what other files it dropped, what URLs it was downloaded from. You can pivot from a single file into an entire campaign.
So you're not just getting a verdict. You're getting a map of everything that file has ever touched, or would touch, in the instrumented environment.
And the sandbox itself. VirusTotal runs the file in instrumented virtual machines and reports back the process tree, the network calls, the files it dropped, the registry writes. That's behavioral forensics on someone else's infrastructure, for free.
Let's pause on that for a second, because I think people don't appreciate how much infrastructure is behind that free report. You're getting a virtual machine that someone provisioned, instrumented, and tore down, just so you could see what your sketchy APK does.
And you're getting it in minutes. Contrast that with standing up your own Cuckoo or CAPE sandbox. You need a host machine, you need guest VMs, you need to configure the instrumentation, you need to keep it patched, and you need to feed it samples. That's a weekend project if everything goes well, and a month if it doesn't. VirusTotal gives you the same thing in a browser tab.
And if you're a single person who just downloaded an APK, you're never going to stand up a sandbox. You're not going to configure a VM. You're going to run the file and hope.
Or you're going to ask a friend who knows computers, and the friend is going to say, upload it to VirusTotal, and then you're done. Which is the right answer, but it's only the first half of the right answer.
So why does nobody use it? Daniel's right that it's underused, but what's the actual reason?
Three reasons. First, fear. Uploading a file feels like leaking it, and sometimes it is. A public VirusTotal upload becomes part of the shared corpus. Other people can see the hash, see the file, download the sample. That's the point, but it's also the problem.
There's a no distribution toggle.
There is, and it matters. If you check it, the file stays private and only the scanners see it. But most people don't know the toggle exists, and enterprises have policies against uploading anything to third parties regardless. So the corporate world largely stays away, which means the public corpus is biased toward the people who don't have anything to lose.
Which is a real structural problem. If the only people contributing to the public corpus are individuals and small teams, the corpus never gets the samples that the enterprise world is seeing.
And the enterprise world is seeing the most interesting stuff. The targeted attacks, the supply-chain compromises, the things that matter. Those samples stay in the private feeds.
Second reason?
Ignorance. People think it's just an antivirus scanner. They paste a URL, see a score, and leave. They never open the Details tab, never look at the sandbox report, never touch the relationship graph.
And the third?
The UI is intimidating. It's built for analysts, not for people who just downloaded a suspicious APK. There are a hundred fields and no obvious starting point. The learning curve is real.
I remember the first time I opened VirusTotal and saw the full report. I had no idea what I was looking at. It was just a wall of jargon. PE headers, import tables, entropy scores. I closed the tab and went back to my life.
And that's the tragedy. The information is there, but the on-ramp is steep enough that most people never get past the aggregate score. They see the green checkmark and they're done.
So the free tool is powerful but the on-ramp is steep, and the privacy tradeoff is real but manageable. That's the identification layer. Where does MalwareBazaar fit in the broader picture?
MalwareBazaar is run by abuse dot ch, and it's the open-feed counterpart to VirusTotal. It's a free, queryable repository of malware samples keyed by hash. Anyone can submit a sample, anyone can query by hash, and the whole thing is designed to be machine-readable. You can pull the feed into your own tools.
And this is where the open-source definition library story actually lives.
A MalwareBazaar entry will often include a YARA rule. YARA is a pattern-matching language for malware. You write a rule that says, if a file contains this byte sequence at this offset, or if it imports these functions and contains this string, it's this family. And because MalwareBazaar publishes the rules alongside the samples, those rules get pulled into open-source scanners.
So the same hash that identifies a file in a database becomes a blocklist entry in an endpoint agent. And the same YARA rule that identifies a family becomes a detection signature in ClamAV.
That's the pipeline. Identification and remediation are the same data at different points in the flow. A hash in MalwareBazaar is a blocklist entry in a firewall. A YARA rule in MalwareBazaar is a detection rule in a SIEM. The open-source definition libraries are the connective tissue.
It's worth dwelling on that for a moment, because it's easy to miss. The free service you use to check a file is also the free service that feeds the paid tools. The line between free and paid is not where people think it is.
The paid tools are often just the free data with better packaging and a support contract. Not always, but often. The underlying signatures, the hash lists, the YARA rules, they're coming from the same place.
So when someone pays for an endpoint agent, they're partly paying for the convenience of having the free data already integrated and updated. The data itself is not the expensive part.
The expensive part is the freshness, and we'll get to that. But the base layer, the identification layer, that's a public good. It's maintained by volunteers and funded by donations and it's free to use.
That's the how. Now let's talk about where all of this came from, and who gets the good stuff.
The lineage starts in the early nineties, with the Computer Antivirus Research Organization. CARO. Before there were centralized databases, antivirus vendors were all independently discovering the same malware and giving it different names. The same virus would be called one thing by McAfee, another thing by Norton, another by F-Prot.
Which made it impossible to share intelligence.
Right. If you're an analyst at one vendor and you want to ask another vendor whether they've seen this thing, you can't even agree on what to call it. So CARO established a naming convention. Family name, variant, suffix. The convention that modern family attribution still descends from.
So the first problem wasn't sharing samples. It was sharing vocabulary.
And once you have a shared vocabulary, you can start sharing signatures. The early nineties were vendor-to-vendor. There were mailing lists, private FTP servers, floppy disks sent through the mail. The key shift came in the two thousands, with the rise of centralized multi-scanner services.
VirusTotal being the obvious one.
VirusTotal launched in two thousand four. The idea was, instead of each vendor maintaining their own scanner and their own corpus, you upload a file once and get back sixty vendor verdicts. That was a genuine innovation. It turned malware identification from a fragmented, vendor-specific process into a centralized, queryable one.
And then the next shift was from vendor-to-vendor sharing to public, hash-indexed corpora. MalwareBazaar, the abuse dot ch feeds, the open-source threat intel platforms.
The key shift is that the data became public and queryable. You no longer need to be a vendor with a sharing agreement. You can be a random person with a suspicious APK and a browser.
That's a genuine democratization. The same query that used to require a vendor relationship now just requires an internet connection.
And the same data that used to live on a private FTP server now lives on a public web page. That's a real shift in who gets to participate.
But that openness has a cost, and this is where the two-tier economy comes in.
The closed, invite-only layer. Vetted membership, NDAs, sample-sharing agreements. The most valuable samples never appear in public corpora.
Why not?
Because publishing a sample burns the detection. If you find a new malware variant and you upload it to a public repository, the malware author can see that you caught it. They know the hash is now in the public corpus. So they recompile, they repack, they change a byte, and the public hash is useless against the new variant.
So the act of publishing the sample tells the adversary exactly what you know.
And exactly when you knew it. The first-seen timestamp on a public upload is a signal to the malware author. It says, here's the moment your detection died.
That's a really uncomfortable dynamic. The thing that makes the public corpus useful is also the thing that makes it self-limiting. Every public sample is a sample the adversary knows about.
So the best intel stays private. The invite-only communities share samples under NDA, and the samples never hit the public feeds.
That creates a two-tier detection economy. Public feeds get you yesterday's malware. Private feeds get you today's. The gap between them is measured in hours to days.
That gap is the entire product of the commercial threat-intel industry.
That's the knock-on effect. The commercial vendors are not selling better analysis. They're selling freshness. The free tools are powerful for one-off forensics. The professional layer's advantage is not capability, it's latency.
Which is a strange thing to wrap your head around. The free tool is just as good at telling you what a file does. The paid tool just tells you sooner.
For a lot of use cases, sooner is the whole game. If you're defending a network and a new variant hits at nine in the morning, knowing about it at nine thirty versus noon is the difference between containing it and cleaning up after it.
There's a fun fact buried in here that I think about a lot. The average time between a malware sample being first seen in the wild and its hash appearing in a public feed is not zero. It's often six to twelve hours, sometimes longer. That's the lag. And that lag is the commercial industry's entire value proposition.
Six to twelve hours doesn't sound like much, but if you're the one who got hit at nine in the morning, those hours are an eternity.
If the malware author is actively iterating, the sample you saw at nine is already obsolete by noon anyway. The private feeds are chasing a moving target, and the public feeds are picking up the spent casings.
But Daniel's question was about examining a single file. A random APK from GitHub. For that use case, the free tools are not just adequate. They're the right tool.
Completely. If you're doing one-off forensics, you don't need today's malware. You need to know what this file is and what it does. The public feeds will tell you that, as long as the file is more than a few hours old.
If it's brand new, the sandbox report will still show you the behavior. The hash won't match anything, but the process tree and the network calls will tell you what it's trying to do.
That's the thing people miss. The hash layer and the sandbox layer are complementary. The hash is the fast path. The sandbox is the thorough path. And the free services give you both.
Let me make sure I have the pipeline straight. A YARA rule written from a MalwareBazaar sample gets pulled into ClamAV. The same hash becomes a blocklist entry in an endpoint agent. The identification data becomes remediation data without changing form.
The lineage goes CARO naming convention, to vendor signature sharing, to centralized multi-scanner services, to public hash-indexed corpora. Each step made the data more accessible. And each step made the malware authors more careful about what they let leak.
Which brings us back to the private layer. The invite-only communities are not a new thing. They're the old vendor sharing model, still running, now with better NDAs.
The early vendor mailing lists and FTP servers never went away. They just became more exclusive. The same principle applies. You share your best samples with people you trust, because the value of the intel is in its secrecy.
The public feeds are the exhaust. The stuff that's already burned, already detected, already stale. Still useful, but structurally a lagging indicator.
Understanding that lag is more useful than memorizing which service to use. The free tools are not toys. They're just pointed at the past.
Hilbert: Iomega Zip disk.
Say again?
Hilbert: The abuse desk at the regional ISP, we kept the shared hash folder on an Iomega Zip disk. Hundred meg. Everyone else was still on floppies, but we had a Zip drive and we thought we were the future.
When was this?
Hilbert: Ninety eight, maybe ninety nine. The whole malware identification toolkit was that disk, a bootable floppy with a scanner on it, and a shared email folder where the other ISPs sent us hash lists. That was it.
The Zip disk was the shared folder?
Hilbert: We'd get a hash list in the morning, update the disk, and then when a customer called in with a suspicious file, we'd hash it against the disk. Took about ten seconds. The scanner on the floppy took twenty minutes and usually found nothing.
The hash check was the fast path, even then.
Hilbert: It was the only path that worked. The scanner was for when the customer insisted we run something. The hash list was what actually told you whether the file was bad.
You still have the floppy.
Hilbert: It's in a box. Still boots, last I checked. Won't say what's on it.
What I take from that is the underuse problem isn't new. The free tools are underused now, but the abuse desk was underfunded then. Same reason.
Hilbert: Nobody wants to pay for the thing that only matters after something has already gone wrong. The ISP paid me for forty hours a month to do abuse work, and I spent most of it arguing with accounting about whether the Zip disks were a reimbursable expense.
Were they?
Hilbert: No. I bought my own. Forty dollars in ninety four on a shareware antivirus license that turned out to be a rebranded free scanner. I've been suspicious of the whole industry ever since.
That rebranded free scanner is the open-source feed problem in miniature. Same code, different label, forty dollars.
Hilbert: The hash list was more useful than the scanner. Faster to check a hash than to run a scan. You could look up a hundred files in the time it took the scanner to finish one.
That's the checksum layer in a sentence. It's not obsolete. It's just the first thing you do, because it's cheap and it eliminates most of the work.
Hilbert: We had a folder of text files. One per family. The hash, the name, the date we first saw it. When a new one came in from another ISP, we'd add it to the disk. That was the whole database.
The lineage from that to MalwareBazaar is almost direct. The data model is the same. Hash, family, first-seen date. The only thing that changed is the scale and the access.
Hilbert: The price. The Zip disk cost forty dollars. MalwareBazaar is free. The expensive thing was never the data.
It was knowing who to share it with.
Hilbert: That's the part that never got cheap.
If the best intel is structurally private, where does that leave the rest of us?
The individual researcher or the small team that will never get an invite. They're stuck with the public feeds, which means they're always a few hours to a few days behind.
As malware authors get better at recompiling and repacking, the checksum layer becomes less useful. The hash changes every time, so the fast path stops working.
Which pushes more of the value into the behavioral sandbox layer. And that layer is expensive to run at scale. The services that can afford to run sandboxes on every upload are the ones that end up with the freshest behavioral data.
The two-tier economy gets worse. The public feeds get slower, the private feeds get more valuable, and the gap widens.
The open question is whether that gap ever closes. Maybe the public feeds get better automation. Maybe the sandbox technology gets cheap enough that anyone can run one. I'm not sure about that part.
I want to sit with that open question for a second, because I think there's a version of this where the gap does close, and it's not through better technology. It's through better defaults. If the public feeds got better at ingesting samples automatically, if the sandbox layer got cheaper to run, if the publishing lag dropped from hours to minutes, the freshness gap would shrink without anyone having to change the structure.
But the malware authors are not standing still either. They're watching the feeds. They're checking whether their samples have been caught. The lag is not just a technical problem. It's an adversarial one. The adversary is actively working to keep the gap open.
The gap is structural, but not because the technology can't improve. It's structural because the adversary has a say.
That's the part people forget. This is not a static problem. The malware author is a participant in the system. Every public upload is a move in a game they're also playing.
The cutting-room floor detail I liked, Daniel. MalwareBazaar lets you query by hash without uploading the file at all. You just paste the SHA two fifty six and ask. That's the privacy-preserving first step that most people skip because they don't know it exists.
It's the step that costs you nothing. No upload, no corpus, no leak. Just a question.
The open question we're left with is whether the individual researcher can ever close the freshness gap, or whether the two-tier economy is just the permanent structure of the field.
I think the honest answer is that the gap is structural. The free tools are pointed at the past, and the past is where most of us live.
Maybe that's okay. If you're checking a random APK from GitHub, the past is probably where you need to be. The file you're holding is almost certainly not the first of its kind. It's a repack of a repack, and someone has already seen it.
The past is underrated. Most malware is not new. Most of it is a recompiled version of something that was first seen months ago. The public feeds will catch that.
The practical advice is: hash it first, query the public feeds, and if the hash comes back empty, upload to VirusTotal and read the sandbox report. That's the full workflow, and it's free.
It's more than most people do. The baseline is so low that just running the hash through MalwareBazaar puts you ahead of the majority of people who download random files from the internet.
Thanks to Hilbert Flumingtop for producing, and for the Zip disk story we will never fully get out of him.
This has been My Weird Prompts. The human AI collaboration podcast.
If you want to send us a prompt, email us at show at my weird prompts dot com. We'll be back soon.
